X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org F0E19393C862
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1598621808;
	bh=aLjqI+tmW6ElPC/RYk4RlnQ6HQWiWb1m0fPqCfNTPBM=;
	h=Date:From:To:Subject:References:In-Reply-To:List-Id:
	 List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe:
	 Reply-To:From;
	b=qBt3iDFYsD/ZMzb6ON3ec6c/1CAmS2Ie6YuHBTmPNfkQpsN/INRhuXPGCerfIZ7I4
	 E6ZYA521Dp4feXajy7fFA6v2Ss6R0572rGdb9Jj7MuRxLJqO7YaWlqI+ZbqPWBVzhH
	 qsjEmprUgWB1GdMXNhpnmDATMEsISZmI4sFA0lTs=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 606303861002
Authentication-Results: sourceware.org;
 dmarc=none (p=none dis=none) header.from=cygwin.com
Authentication-Results: sourceware.org;
 spf=fail smtp.mailfrom=corinna-cygwin AT cygwin DOT com
Date: Fri, 28 Aug 2020 15:36:43 +0200
From: Corinna Vinschen <corinna-cygwin AT cygwin DOT com>
To: cygwin AT cygwin DOT com
Subject: Re: Sv: Sv: Limit for number of child processes
Message-ID: <20200828133643.GK3272@calimero.vinschen.de>
Mail-Followup-To: cygwin AT cygwin DOT com
References: <000401d67ba0$8b1f33b0$a15d9b10$@gmail.com>
 <20200826175724 DOT GQ3272 AT calimero DOT vinschen DOT de>
 <000701d67c6c$10bcf720$3236e560$@gmail.com>
 <6d698a32-06bb-a47b-58e6-ceeecca722c9 AT cornell DOT edu>
 <001101d67d16$aa5db9f0$ff192dd0$@gmail.com>
 <6242991f-e5ec-150b-bd6c-15a8c348c7cf AT cornell DOT edu>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <6242991f-e5ec-150b-bd6c-15a8c348c7cf@cornell.edu>
X-Provags-ID: V03:K1:GqPf6K7fq5JP8vLBDur1p3QQPhwP7d7DmvUuHdDLE2HofvAUkpx
 PcuRzcJnrZmUVyooCXGSxOU9Ufkqw7aZjsZoNF8K901Px6wgnfnluJN8b99zhhPRArT9LCX
 c5cmNU6jBI3+7Jf+ey1ogFSlACdZlGZmIfESHqbHeMvwN06IAjswdZeGSKW9Dok3UqxmVqx
 0zpuKgYnhVwsyu0359phA==
X-UI-Out-Filterresults: notjunk:1;V03:K0:vGIwFQ8VcpA=:51g84ujsYyTFuw5Iy7NL2M
 MW0j3KH5OEPcc02a36igTQKWUPqo6eEgiKZCxzYzZfFemiljeUiwys+ERRmEmGt1igK0sCuUg
 7co15i37dY0zC+jRMZdX1hHBsxxpUhSS3UvGPtQvlbpDh3MYqtflr263nQeiWVnHbyWYHl0sA
 KiX/RoBcBmpANnaA1fCfVGuxnJCvtf9n7vnnCX8T8/yeYb2VwjUE578xQVxGYejOsfRA9+1be
 gBKTMDKoHl3CTiBOkNEt/v9gSdZV9RTte8KrncAxlR9wUFdO/iToIU6i8wcDCfKK7PI10XR/s
 SeFXJx+fGcn82hAq7H0hfsaLuu5JVCh2ycPXoK/cqxTg/2oYpH9XKw1+xGRQWaKmXsaFHd59t
 BuawoJjtiTa9We7ZKUACnZi8CZ+Jx+n03B0rWYgiMoHV/99EEja2yEs78agXL4N8l/tURjQpr
 Mn/UaKQur2VIxwwEhNwmQAru9gIASZRW8AqX6XCHFtCYyqOUZ8FWZbtIA5IT5zYg1GEPw2QLO
 wqmzDwkvn1Yu6cGK1rTa/zxmQKzXdBscemJsx1e4v3FKq6vcbGlb5WZDEOQIQKWJgc5GpMpuF
 ZBE9AFrQnaQzClUeGKe92MfnPY/czeZyrW6Tb//yuvcLGCmec2huDlqQ/5H0unvGIUuh7Aw6K
 tBSZuR6OAU1oglacGcktJa156maM4V4+lJuSXCxlgSymNOnxh+VvLU6VnT+mh3V/ReAQa2wIz
 9GzQi7IjKxHaAsyec9m+JUtk8C0LeuozH/SV9BgQWdSXJWxrbfaFIheXhP1q6f94PClUPF4At
 gM6f0ldl1jVcBCOjQzThmGb7SZMPo/uW++zbGDPw2yACv1up1WqQbR0ABfbzYE4FaQ3mzBjsg
 3TSTS99FxCpOzEpjAJPw==
X-Spam-Status: No, score=-99.9 required=5.0 tests=BAYES_00,
 GOOD_FROM_CORINNA_CYGWIN, JMQ_SPF_NEUTRAL, KAM_DMARC_STATUS,
 RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NEUTRAL,
 TXREP autolearn=ham autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Reply-To: cygwin AT cygwin DOT com
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: cygwin-bounces AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces AT cygwin DOT com>

On Aug 28 08:29, Ken Brown via Cygwin wrote:
> On 8/28/2020 4:38 AM, sten DOT kristian DOT ivarsson AT gmail DOT com wrote:
> > > > > > It seems like there's a limit of the number of possible child
> > > > > > processes defined to 256 with 'NPROCS' in
> > > > > > //winsup/cygwin/child_info.h used in 'cprocs' in
> > > > > > //winsup/cygwin/sigproc.cc
> > > > > > 
> > > > > > 256 is quite few possible children in an enterprise environment and
> > > > > > perhaps the limit should be limited by the physical resources or
> > > > > possibly Windows ?
> > > > > 
> > > > > The info has to be kept available in the process itself so we need
> > > > > this array of NPROCS * sizeof (pinfo).
> > > > > 
> > > > > Of course, there's no reason to use a static array, the code could
> > > > > just as well use a dynamically allocated array or a linked list.
> > > > > It's just not the way it is right now and would need a patch or
> > > rewrite.
> > > > > [...]
> > > > A linked list could be used if you wanna optimize (dynamic) memory
> > > > usage but an (amortized) array would probably provide faster linear
> > > > search but I guess simplicity of the code and external functionality
> > > > is the most important demands for this choice
> > > 
> > > Any change here (aside from just increasing NPROCS) would have to be done
> > > with care to avoid a performance hit.  I looked at the history of changes
> > > to sigproc.cc, and I found commit 4ce15a49 in 2001 in which a static array
> > > something like cprocs was replaced by a dynamically allocated buffer in
> > > order to save DLL space.  This was reverted 3 days later (commit e2ea684e)
> > > because of performance issues.
> > 
> > I wonder what kind of performance issue ? 
> > [...]
> I don't know for sure, but I doubt if it had anything to do with memory
> access. My guess is that the performance hit came from the need to free the
> allocated memory after every fork call (see sigproc_fixup_after_fork).

Either way, I rewrote this partially so we now have a default array size
for 255 child processes on 32 bit and 1023 child processes on 64 bit.

The new code is mainly a minor update in that it convertes the code
directly accessing stuff into using a class, encapsulating the mechanism
used under the hood behind a class barrier and access methods.

As POC, I added a bit of code to maintain a second array, which is only
allocated (using HeapAlloc so as not to spill into the child processes)
if the default array overflows.  This second array adds room for another
1023 (32 bit) or 4095 (64 bit) child processes, raising the number of
max child processes per process to 1278 on 32 bit and 5118 on 64 bit.

My STC just forking like crazy overflowed my 4 Gigs RAM + 2.5 Gigs
pagefile after roughly 1450 child processes.  I'm pretty confident that
this POC implementation is sufficient for a while, even in enterprise
scenarios.

And if not, we can now easily tweak the numbers without having to 
tweak much of the code.

For testing purposes I uploaded a developer snapshot to
https://cygwin.com/snapshots/


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple