X-Recipient: archive-cygwin AT delorie DOT com
DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:from:to:subject:message-id:references
	:mime-version:content-type:in-reply-to; q=dns; s=default; b=wfUF
	hXc8u8j+1La/cZVt9SeTocU2s26DWhFyQrH24+LPs+fLPntOriN2swCDQKiRbCS2
	mCJY0AAH/tCPbwAXJZb1KRohbpE2HwNNGkMppyHBZ1G9fP544Jmcbj2wathQY49M
	Y9wMUXY/3MHuIVkvKGAbJl+ohx7MDeoLmMr5WL4=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:from:to:subject:message-id:references
	:mime-version:content-type:in-reply-to; s=default; bh=ZqGrHgmemB
	b2fmWJ3jDRppwZ4IY=; b=snsQQ5Qlw64LEbFnbUbiQ3+Xc8fo+U4pYx1bly8Kb/
	W0ImITMUuHIUiwrZi/lXTq3KjKXEiz1LVyxCGsPyaO6nFfHt7qFh2wXGH2t3UAbD
	gHNb25LIAw+rBJpGyl3md5Po/ot1xHWPi9ok8usiyP0u/3x4nQPCTtJgvW/2f5Zh
	U=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id: <cygwin.cygwin.com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
Authentication-Results: sourceware.org; auth=none
X-Virus-Found: No
X-Spam-SWARE-Status: No, score=-1.9 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.2 spammy=concluding, installations, symptoms, freezes
X-HELO: mail-pg0-f41.google.com
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;        d=1e100.net; s=20161025;        h=x-gm-message-state:date:from:to:subject:message-id:references         :mime-version:content-disposition:in-reply-to:user-agent;        bh=rSPnyQaYo4MpZVix8lcBxL7qnbFsIso2T59HDkTpXEg=;        b=jFDYRtWlrUbh8tfDiAHJMnKRZ7+2rnd4k+1NI8uUjmcPfCl/uFTbiH/6TFDJvzT7oY         AdJb1FgvvcGpwmPgKEgnq+p4bwnbx/4IVDwzpG9zHAAEI7vsHyOMA2D9KNtyDWDwctT9         On/5JvvScDiSdJdZ9e/fxesoDa6OSWJwOssu4G6JnVLBtKYjpSR1t+p4GGgIXgSU5g0+         J4yTUHJNiL+WyZhkmQg9pWNnawivwVBIRyA5qEB5U2bEC3vNTS8LWK/sZQmvTgBxGGiB         7SU9Cna2nXzyYJtvLP1OFnIx38kRPgRMHJwgBSiV7rwKXazaNkfTIy303giuG/A6Bw7Z         6wwg==
X-Gm-Message-State: AFeK/H2bVqthAAABQwqIsoVI+wjiYetVffoG0HWlb3a19GE4uFsonHT98aSaz18RSSADoQ==
X-Received: by 10.84.238.22 with SMTP id u22mr34002497plk.137.1490678816268;        Mon, 27 Mar 2017 22:26:56 -0700 (PDT)
Date: Tue, 28 Mar 2017 01:26:52 -0400
From: Noah Misch <noah AT leadboat DOT com>
To: cygwin AT cygwin DOT com
Subject: Re: cygserver - Postgres Multiple connection Load Testing - Inifinte Loop
Message-ID: <20170328052652.GA2351961@tornado.leadboat.com>
References: <200408030333 DOT i733XEXn023894 AT mx3 DOT redhat DOT com> <20040803100612 DOT GP31522 AT cygbert DOT vinschen DOT de> <20170321025614 DOT GA2100214 AT tornado DOT leadboat DOT com> <20170324171101 DOT GI29995 AT calimero DOT vinschen DOT de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20170324171101.GI29995@calimero.vinschen.de>
User-Agent: Mutt/1.5.24 (2015-08-30)

On Fri, Mar 24, 2017 at 06:11:01PM +0100, Corinna Vinschen wrote:
> - cygserver is using a defined number of threads in a thread pool for
>   application requests.  Every request is added to a request submission
>   queue and handled by the next free thread in the pool.
> 
>   The default number of threads in the pool is 10.  Each wait for a
>   semaphore is blocking one thread.  If more than the number of threads
>   in the pool are supposed to wait on a semaphore the pool starves.

Interesting.  I can confirm that, without updating software, "cygserver -r 40"
fixes both my self-contained test and my PostgreSQL test case.  Folks can use
that workaround in released-version installations.

>   So what I did now is to allow cygserver to raise the number of worker
>   threads on demand.  That is, if a request is in the queue and all
>   worker threads are busy, just create a new one.
> 
>   There's no way yet to drop threads again, but this should be a minor
>   problem in scenarions which really have a lot of contention.

Agreed.  This is nicer.

> I pushed a patchset now, and uploaded new developer snapshots for
> testing to https://cygwin.com/snapshots/

> Please give it a try

Self-contained test case results look good:

cygwin-20170321.tar.xz "cygserver -r40": ok
cygwin-20170324.tar.xz "cygserver -r40": ok
cygwin-20170321.tar.xz "cygserver -r10": freezes (expected)
cygwin-20170324.tar.xz "cygserver -r10": ok; cygserver output concludes with
  "cygserver: All threads busy, added one (now 21)".

I then tried my PostgreSQL test case ("pgbench -i -s 50" once to setup, then
"pgbench -S -j2 -c16 -T900 -P5" to test):

cygwin-20170321.tar.xz "cygserver -r40": ok for >3600s
cygwin-20170324.tar.xz "cygserver -r40": limited freeze in <1000s; no
  cygserver output
cygwin-20170321.tar.xz "cygserver -r10": classic freeze in <1000s (expected)
cygwin-20170324.tar.xz "cygserver -r10": limited freeze in <1000s; no
  cygserver output for most of the run, then output concluding with
  "cygserver: All threads busy, added one (now 15)" just before the freeze

I call the cygwin-20170324 freezes "limited" because the symptoms differ from
the classic freeze I described upthread.  "strace /bin/true" and "cat
/proc/sysvipc/sem" do not hang, but every PostgreSQL backend process is stuck
waiting on a synchronization primitive.

I can distill another self-contained test case for the limited freeze seen in
cygwin-20170324, but that make take awhile.  I'm sending this early report so
you're aware of the possible regression in cygwin-20170324.

Thanks,
nm

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple