X-Recipient: archive-cygwin AT delorie DOT com
DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:content-type; q=dns; s=default; b=HP
	wvzs0DQGvXrMvfgaAeLOtApPrZWiOmCSzbA+4EUcuyFI59dRKuCuvscZ3FaYID84
	wP/pHFc7KV9CZyIHQiotgpfn8Rv6G++yHgr+MVwEkRshoZl1vHd+tykKIdzfPAp8
	WPdZv77nwqrMqoqsxdLilpXZEV6/1ZGo4Lrdmvl5E=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:content-type; s=default; bh=eKEeP/Vo
	GgwaNKMlpxSpXtozB+I=; b=Upja4g3CgBsbH5Z/Zu/kN7BbCwBsnw0BotZACRpi
	E7zWfRXLk0ZUMnQ0llSH8R+SXh8c4UmKfRJRaIuLcAbjq0Xk3OdxDIso+2A/d/Zi
	S0oGZnpor5JUoYM8rhUiK2kphmTjlQ6dSkFcIxxSMXlTXj5FDZvhJ5eBlEQ25K2T
	jG8=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id: <cygwin.cygwin.com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
Authentication-Results: sourceware.org; auth=none
X-Virus-Found: No
X-Spam-SWARE-Status: No, score=0.3 required=5.0 tests=BAYES_50,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2
X-HELO: mail-wm0-f42.google.com
MIME-Version: 1.0
X-Received: by 10.194.175.194 with SMTP id cc2mr23145866wjc.121.1449320874700; Sat, 05 Dec 2015 05:07:54 -0800 (PST)
In-Reply-To: <5662C199.7040906@maxrnd.com>
References: <CABPLASTtRK4mNxh0M_AnZgjJQ15kWPx+L=U=VCU3Wwi7jV_57A AT mail DOT gmail DOT com>	<564E3017 DOT 90205 AT maxrnd DOT com>	<CABPLASTLrH_udLuu2F-m5P6dkENW1Z4YHEudp4NG0-FGLJgPMg AT mail DOT gmail DOT com>	<5650379B DOT 4030405 AT maxrnd DOT com>	<20151121105301 DOT GE2755 AT calimero DOT vinschen DOT de>	<5652C402 DOT 7040006 AT maxrnd DOT com>	<24780-1448274431-7444 AT sneakemail DOT com>	<5653B52B DOT 5000804 AT maxrnd DOT com>	<20151126093427 DOT GJ2755 AT calimero DOT vinschen DOT de>	<5656DDEF DOT 9070603 AT maxrnd DOT com>	<5662C199 DOT 7040906 AT maxrnd DOT com>
Date: Sat, 5 Dec 2015 14:07:54 +0100
Message-ID: <CABPLAST5EnifrAQ2xKZmohKhyxQHh=K3x3DeCL+BTdHN8nN98w@mail.gmail.com>
Subject: Re: Cygwin multithreading performance
From: Kacper Michajlow <kasper93 AT gmail DOT com>
To: cygwin AT cygwin DOT com
Content-Type: text/plain; charset=UTF-8
X-IsSubscribed: yes

2015-12-05 11:51 GMT+01:00 Mark Geisert <mark AT maxrnd DOT com>:
> Mark Geisert wrote:
>>
>> Corinna Vinschen wrote:
>>>
>>> On Nov 23 16:54, Mark Geisert wrote:
>>>>
>>>> John Hein wrote:
>>>>>
>>>>> Mark Geisert wrote at 23:45 -0800 on Nov 22, 2015:
>>>>>   > Corinna Vinschen wrote:
>>>>>   > > On Nov 21 01:21, Mark Geisert wrote:
>>>>>   > [...] so I wonder if there's
>>>>>   > >> some unintentional serialization going on somewhere, but I
>>>>> don't know yet
>>>>>   > >> how I could verify that theory.
>>>>>   > >
>>>>>   > > If I'm allowed to make an educated guess, the big serializer
>>>>> in Cygwin
>>>>>   > > are probably the calls to malloc, calloc, realloc, free.  We
>>>>> desperately
>>>>>   > > need a new malloc implementation better suited to
>>>>> multi-threading.
>>
>> [...]
>>>>>
>>>>>
>>>>> Someone recently mentioned on this list they were working on porting
>>>>> jemalloc.  That would be a good choice.
>>>>
>>>>
>>>> Indeed; thanks for the reminder.  Somehow I hadn't followed that thread.
>>>
>>>
>>> Indeed^2.  Did you look into the locking any further to see if there's
>>> more than one culprit?  I guess we've a rather long way to a "lock-less
>>> kernel"...
>
> [...]
>>
>> But that is just groundwork to identifying which locks are suffering the
>> most contention.  To identify them at source level I think I'll also
>> need to record the caller's RIP when they are being locked.
>
>
> In the OP's very good testcase the most heavily contended locks, by far, are
> those internal to git's builtin/pack-objects.c.  I plan to show actual stats
> after some more cleanup, but I did notice something in that git source file
> that might explain the difference between Cygwin and MinGW when running this
> testcase...
>
> #ifndef NO_PTHREADS
>
> static pthread_mutex_t read_mutex;
> #define read_lock()             pthread_mutex_lock(&read_mutex)
> #define read_unlock()           pthread_mutex_unlock(&read_mutex)
>
> static pthread_mutex_t cache_mutex;
> #define cache_lock()            pthread_mutex_lock(&cache_mutex)
> #define cache_unlock()          pthread_mutex_unlock(&cache_mutex)
>
> static pthread_mutex_t progress_mutex;
> #define progress_lock()         pthread_mutex_lock(&progress_mutex)
> #define progress_unlock()       pthread_mutex_unlock(&progress_mutex)
>
> #else
>
> #define read_lock()             (void)0
> #define read_unlock()           (void)0
> #define cache_lock()            (void)0
> #define cache_unlock()          (void)0
> #define progress_lock()         (void)0
> #define progress_unlock()       (void)0
>
> #endif
>
> Is it possible the MinGW version of git is compiled with NO_PTHREADS
> #defined?  If so, it would mean there's no locking being done at all and
> would explain the faster execution and near 100% CPU utilization when
> running under MinGW.

Nah, there is no threading enabled when there is no pthreads. How
would that work? :D See thread-utils.h

#ifndef NO_PTHREADS
#include <pthread.h>

extern int online_cpus(void);
extern int init_recursive_mutex(pthread_mutex_t*);

#else

#define online_cpus() 1

#endif


Looks like there is indeed a bug in git code when passing "--threads"
explicitly to "git pack-objects", because they show warning about
threads being unsupported, but doesn't overwrite delta_search_threads
value. I will go to git's ML about it. This is completely not related
to our issue.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple