DMARC-Filter:	OpenDMARC Filter v1.4.2 delorie.com 4AJ423Y62948369
Authentication-Results:	delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results:	delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter:	OpenDKIM Filter v2.11.0 delorie.com 4AJ423Y62948369
Authentication-Results:	delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=QZhspQz8
X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 22C7D3858D21
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1731988921;
	bh=oY9xSAEMpf4yedmixSuYRxNJMNwMile3sXNPxMYlTNs=;
	h=Date:Subject:To:References:In-Reply-To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	From;
	b=QZhspQz8/x0rhpd8lDer8Dyn8f0S3jAPV+GOqNmqLAmbh7oCIc7Wy5QuqpXuSd14h
	ei9gL6F331V4e3lt6ARSoi3gFaY6EcifhS17kIM2/T8hZE+dEEVquw17hz4rL5WSJI
	cArTTU4FqtrsFKBRYhkNmvvyt3ZwuoHzfx4j8K6g=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.2 sourceware.org 4680B3858D21
ARC-Filter:	OpenARC Filter v1.0.0 sourceware.org 4680B3858D21
ARC-Seal:	i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1731988856; cv=none;
	b=rOxNPaGGTENBkSA4UOsyEdNFN3jJ/Epqr3JeO5cSMVAw9NEstM++EO6bdDrnvzSCmHvsqrE/pl3uQC6hCkOkHBVDZgh7UaRg+WbipG2+C/EE7JhGYEuN8IjowoZyw0fMCfZ3jZdBIDwRoKuphoQBkiv315vZ70hw+319ZUvLj70=
ARC-Message-Signature:	i=1; a=rsa-sha256; d=sourceware.org; s=key;
	t=1731988856; c=relaxed/simple;
	bh=tk1rLhQ/J0M3AK6Q7KMnuBKaIakJdychxKMXV0vquck=;
	h=Message-ID:Date:MIME-Version:Subject:To:From;
	b=KxUmzuFxZ2tZmn6Leqa0WJMQtThR/6XIX4qYfD+N/Zuy91ENbJC48R2LkvlOXfoLq+UqFYea7ROrzieLByGIkn8rdeP1evzonvo1gehrVYlTijGD9xvlYX0s18xMxAf6yTLW70S3qr+fSupbZ2Y4I/iDAjIrkyzpKSdq6ZDAp4Q=
ARC-Authentication-Results:	i=1; server2.sourceware.org
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 4680B3858D21
Message-ID:	<0bec82b7-503d-4cfd-b940-cb66914b61f1@maxrnd.com>
Date:	Mon, 18 Nov 2024 20:01:00 -0800
MIME-Version:	1.0
User-Agent:	Mozilla Thunderbird
Subject:	Re: Thread memory allocation issue
To:	"cygwin AT cygwin DOT com" <cygwin AT cygwin DOT com>
References:	<791865992 DOT 3601338 DOT 1731871978376 DOT ref AT mail DOT yahoo DOT com>
	<791865992 DOT 3601338 DOT 1731871978376 AT mail DOT yahoo DOT com>
In-Reply-To:	<791865992.3601338.1731871978376@mail.yahoo.com>
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.30
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe:	<https://cygwin.com/mailman/options/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	Mark Geisert via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	Mark Geisert <mark AT maxrnd DOT com>
Errors-To:	cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender:	"Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
X-MIME-Autoconverted:	from base64 to 8bit by delorie.com id 4AJ423Y62948369

Hello Teepean,

On 11/17/2024 11:32 AM, Teepean via Cygwin wrote:
> I raised this issue couple of years ago on cygwin-developer but now when the problem has manifested again with recent versions of Cygwin I decided to post this to general discussion list.

This (main Cygwin) list is the correct place for reports like this. 
There is no need to contact me (or other maintainers/devs) off-list.

Given that the result of the investigation a couple years ago was, 
essentially, no change to Cygwin's malloc*, why has the problem 
manifested again recently?  Have you been benchmarking/testing all 
along?  Can you be more specific about which recent Cygwin versions?

*My own benchmark, building the Cygwin tree, showed that there wasn't 
much difference between the half-dozen malloc implementations I tried 
and they were all spending more time in Windows' ntdll.dll than the 
current Cygwin malloc (==dlmalloc), though a little less time in Cygwin 
itself.

> Steps to Reproduce
> 
> 1. Compile BWA normally
> 
> https://github.com/lh3/bwa/

What's involved with that? Clone the repo, ./configure, make? Anything else?

> 2. Compile BWA with rpmalloc and the following patch:
> 
> 
> // In thread worker function:
> #ifdef __CYGWIN__
> rpmalloc_thread_initialize();
> #endif
> 
> 
> // ... thread work ...
> #ifdef __CYGWIN__
> rpmalloc_thread_finalize(1);
> #endif

Where does that patch go? Assume I know nothing about BWA.

> 3. Run both versions with the following command:
> time ./bwa mem -t 11 chr19_KI270866v1_alt.fasta test_1.fastq test_2.fastq > testorigsingle.sam
> 
> 
> Without Patch (Default malloc):
> 
> 
> [M::mem_process_seqs] Processed 120000 reads in 30.296 CPU sec, 3.743 real sec
> [main] Real time: 3.883 sec; CPU: 30.436 sec
> real    0m3.907s
> user    0m19.186s
> sys     0m11.265s
> 
> 
> With Patch (rpmalloc):
> 
> 
> [M::mem_process_seqs] Processed 120000 reads in 7.530 CPU sec, 0.702 real sec
> [main] Real time: 0.830 sec; CPU: 7.640 sec
> real    0m0.868s
> user    0m7.343s
> sys     0m0.327s

Are these examples of runs one would do "in production"? Or are you 
running much longer-lasting processing in the usual case?

> Analysis
> 
> 1. The default malloc implementation shows extremely high system time (11.265s) compared to the rpmalloc version (0.327s)
> 2. Total real time is about 4.5x slower with default malloc
> 3. The dramatic difference in system time suggests heavy contention in the memory allocation subsystem
> 4. The issue only manifests on Cygwin with bwa; the same code performs normally on native Linux and MacOS

Are you saying there is non-bwa code that runs on Cygwin comparably to 
Linux and Mac?

> 5. The issue manifests with recent versions of Cygwin but does work with older versions

Again, it would really help if you could give Cygwin versions or at 
least dates here...

> The issue becomes more pronounced with higher thread counts

That I believe; dlmalloc as it is currently set up for Cygwin is not the 
greatest performer for heavy thread usage.

> The patched code is located here in branch Cygwin:
> 
> 
> https://github.com/WGSExtract/bwa.git
> 
> 
> Simple testsuite. Run bash testsuite.sh. The testsuite includes a version compiled with an older version of Cygwin called bwa_working.exe
> 
> 
> https://drive.google.com/file/d/1jtbQVUAcCmpJM-8Exi0C6pzDXcEo4cV6/view?usp=drive_link

I'll glance at this stuff when I can but I hope to have some answers to 
my questions above from you to save me some time.

..mark


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -