X-Recipient: archive-cygwin AT delorie DOT com DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; q=dns; s=default; b=pz OMTpnFdc4YLL6achbuYr8VV4DCL0zFLaejZjxn2JEkEJdyApNm9YCAt8QnFol4Qu efk9zrMbgpkJgGtF/TE4pYMPjMZQggazVz6/VIsSqWzWcJJ5sA/2PuYICOvQQkH0 MH3i9H9G81GPX7485fdnPHuazphAoPxggKpT0G4uo= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; s=default; bh=v5s7+UJ+ EylPIIOSwOZkENmu3a8=; b=kl/G+70nbO/uv4SWQI/2FgqpFyAZCqJzXafIOfQl CaziKmwSre+EG1NvxM+XHWH6uQyYZDUY1uMDblhhIdSkFfQHkOIWfpySfQ3/UZk5 nbIDBMTmSmPr/LknaipquGKS9Fu35AB8iHHpVUanREDGFNh6Vuq9z0uUIU82MdfI M/g= Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner AT cygwin DOT com Mail-Followup-To: cygwin AT cygwin DOT com Delivered-To: mailing list cygwin AT cygwin DOT com Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.0 required=5.0 tests=AWL,BAYES_00,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-wm0-f43.google.com MIME-Version: 1.0 X-Received: by 10.194.175.194 with SMTP id cc2mr23351672wjc.121.1449323984089; Sat, 05 Dec 2015 05:59:44 -0800 (PST) In-Reply-To: References: <564E3017 DOT 90205 AT maxrnd DOT com> <5650379B DOT 4030405 AT maxrnd DOT com> <20151121105301 DOT GE2755 AT calimero DOT vinschen DOT de> <5652C402 DOT 7040006 AT maxrnd DOT com> <24780-1448274431-7444 AT sneakemail DOT com> <5653B52B DOT 5000804 AT maxrnd DOT com> <20151126093427 DOT GJ2755 AT calimero DOT vinschen DOT de> <5656DDEF DOT 9070603 AT maxrnd DOT com> <5662C199 DOT 7040906 AT maxrnd DOT com> Date: Sat, 5 Dec 2015 14:59:44 +0100 Message-ID: Subject: Re: Cygwin multithreading performance From: Kacper Michajlow To: cygwin AT cygwin DOT com Content-Type: text/plain; charset=UTF-8 X-IsSubscribed: yes 2015-12-05 14:07 GMT+01:00 Kacper Michajlow : > 2015-12-05 11:51 GMT+01:00 Mark Geisert : >> Mark Geisert wrote: >>> >>> Corinna Vinschen wrote: >>>> >>>> On Nov 23 16:54, Mark Geisert wrote: >>>>> >>>>> John Hein wrote: >>>>>> >>>>>> Mark Geisert wrote at 23:45 -0800 on Nov 22, 2015: >>>>>> > Corinna Vinschen wrote: >>>>>> > > On Nov 21 01:21, Mark Geisert wrote: >>>>>> > [...] so I wonder if there's >>>>>> > >> some unintentional serialization going on somewhere, but I >>>>>> don't know yet >>>>>> > >> how I could verify that theory. >>>>>> > > >>>>>> > > If I'm allowed to make an educated guess, the big serializer >>>>>> in Cygwin >>>>>> > > are probably the calls to malloc, calloc, realloc, free. We >>>>>> desperately >>>>>> > > need a new malloc implementation better suited to >>>>>> multi-threading. >>> >>> [...] >>>>>> >>>>>> >>>>>> Someone recently mentioned on this list they were working on porting >>>>>> jemalloc. That would be a good choice. >>>>> >>>>> >>>>> Indeed; thanks for the reminder. Somehow I hadn't followed that thread. >>>> >>>> >>>> Indeed^2. Did you look into the locking any further to see if there's >>>> more than one culprit? I guess we've a rather long way to a "lock-less >>>> kernel"... >> >> [...] >>> >>> But that is just groundwork to identifying which locks are suffering the >>> most contention. To identify them at source level I think I'll also >>> need to record the caller's RIP when they are being locked. >> >> >> In the OP's very good testcase the most heavily contended locks, by far, are >> those internal to git's builtin/pack-objects.c. I plan to show actual stats >> after some more cleanup, but I did notice something in that git source file >> that might explain the difference between Cygwin and MinGW when running this >> testcase... >> >> #ifndef NO_PTHREADS >> >> static pthread_mutex_t read_mutex; >> #define read_lock() pthread_mutex_lock(&read_mutex) >> #define read_unlock() pthread_mutex_unlock(&read_mutex) >> >> static pthread_mutex_t cache_mutex; >> #define cache_lock() pthread_mutex_lock(&cache_mutex) >> #define cache_unlock() pthread_mutex_unlock(&cache_mutex) >> >> static pthread_mutex_t progress_mutex; >> #define progress_lock() pthread_mutex_lock(&progress_mutex) >> #define progress_unlock() pthread_mutex_unlock(&progress_mutex) >> >> #else >> >> #define read_lock() (void)0 >> #define read_unlock() (void)0 >> #define cache_lock() (void)0 >> #define cache_unlock() (void)0 >> #define progress_lock() (void)0 >> #define progress_unlock() (void)0 >> >> #endif >> >> Is it possible the MinGW version of git is compiled with NO_PTHREADS >> #defined? If so, it would mean there's no locking being done at all and >> would explain the faster execution and near 100% CPU utilization when >> running under MinGW. > > Nah, there is no threading enabled when there is no pthreads. How > would that work? :D See thread-utils.h > > #ifndef NO_PTHREADS > #include > > extern int online_cpus(void); > extern int init_recursive_mutex(pthread_mutex_t*); > > #else > > #define online_cpus() 1 > > #endif > > > Looks like there is indeed a bug in git code when passing "--threads" > explicitly to "git pack-objects", because they show warning about > threads being unsupported, but doesn't overwrite delta_search_threads > value. I will go to git's ML about it. This is completely not related > to our issue. Obviously I was wrong. There is #define ll_find_deltas(l, s, w, d, p) find_deltas(l, &s, w, d, p) So 'delta_search_threads' value is never used. Still not related to cygwin issue tho ;) -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple