X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org EA7E4393C874
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1598623749;
	bh=9uPz0XUGIOxg0cviKqiWXRfZIGRR1WgIrZRGCXdkuDg=;
	h=To:References:In-Reply-To:Subject:Date:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	 From;
	b=bR7DZTPWfhvlmPx4YiIIXLSd1chwQyGUwbF+1NF2CI264k6j8wDqujQmFXm3YXuF1
	 RDPlSRGuEinfdtu+9avSoVyH2QzCbFzHL2+ttUhaYQqwdnCY8skcswtq7nDo79MREO
	 37y3LnbgiSTAoYAdTdLHpwaLpI7LNPmb3K8HtVO0=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 13310384A40A
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:references:in-reply-to:subject:date
 :message-id:mime-version:content-transfer-encoding:thread-index
 :content-language;
 bh=xaaFQzFacexG0VK5xOde5ID8vJR+7wvffdXuAA1ug4w=;
 b=bcgnPV5tyycP+iWujTZVA57JUlTW06khH1UQ/aZO16xHAXgEbdkJ2AJ7YwIJF/tURi
 aUqzHHDlVmPtPs0EHLy+aTDp5zvIu8xpeMBjclU/+TRVjVha8oy94kfjev9LCxy2JOx1
 KJmfeB6M4jBXmJRwzILunR3CPRmvKNSaRHbQooyOyxI9KAJwcwEoTvqx/AYF9WoETB89
 XX7OfDQgV6WvODxXjAgsJSyZYuBJmtay5gODHwtoSx9aqDlemCqTrbQ8wXY4DlGUgi7Q
 T4+cfuJ7msDdTKMCzAaPI96sko1Iqq/NL6vSkismD3mOkVp7sk0wrNxorpXjabJ6JcnI
 9SkQ==
X-Gm-Message-State: AOAM532qqV0RdzGMBSPGtSQKl5aAGwIsfCt6feqXAYLIAQTnFiMUuIJj
 +gMxzo2J7LtknbWh3UfOC8HNOjVaCYg=
X-Google-Smtp-Source: ABdhPJz7fTlW+KDIdeQaxoJ+v2nODwl4TMHtES8x7mnSCBI8yBwZV+E4QMPxAEw49UXPwAymX056dA==
X-Received: by 2002:a2e:584:: with SMTP id 126mr912548ljf.413.1598623743869;
 Fri, 28 Aug 2020 07:09:03 -0700 (PDT)
To: <cygwin AT cygwin DOT com>
References: <000401d67ba0$8b1f33b0$a15d9b10$@gmail.com>
 <20200826175724 DOT GQ3272 AT calimero DOT vinschen DOT de>
 <000701d67c6c$10bcf720$3236e560$@gmail.com>
 <6d698a32-06bb-a47b-58e6-ceeecca722c9 AT cornell DOT edu>
 <001101d67d16$aa5db9f0$ff192dd0$@gmail.com>
 <6242991f-e5ec-150b-bd6c-15a8c348c7cf AT cornell DOT edu>
 <20200828133643 DOT GK3272 AT calimero DOT vinschen DOT de>
In-Reply-To: <20200828133643.GK3272@calimero.vinschen.de>
Subject: Sv: Sv: Sv: Limit for number of child processes
Date: Fri, 28 Aug 2020 16:09:02 +0200
Message-ID: <000101d67d44$c8bc4c80$5a34e580$@gmail.com>
MIME-Version: 1.0
X-Mailer: Microsoft Outlook 16.0
Thread-Index: AQI7mQ+hPIKf2raSwP+uxe6Dh/MnugIebBIUAmKifX0CBffj2wJEwYIZAhxUMFkB8sTjDqgcfYXw
Content-Language: en-gb
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, DKIM_SIGNED,
 DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,
 SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Kristian Ivarsson via Cygwin <cygwin AT cygwin DOT com>
Reply-To: sten DOT kristian DOT ivarsson AT gmail DOT com
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: "Cygwin" <cygwin-bounces AT cygwin DOT com>

Hi all

> > > > > > > It seems like there's a limit of the number of possible
> > > > > > > child processes defined to 256 with 'NPROCS' in
> > > > > > > //winsup/cygwin/child_info.h used in 'cprocs' in
> > > > > > > //winsup/cygwin/sigproc.cc
> > > > > > >
> > > > > > > 256 is quite few possible children in an enterprise
> > > > > > > environment and perhaps the limit should be limited by the
> > > > > > > physical resources or
> > > > > > possibly Windows ?
> > > > > >
> > > > > > The info has to be kept available in the process itself so we
> > > > > > need this array of NPROCS * sizeof (pinfo).
> > > > > >
> > > > > > Of course, there's no reason to use a static array, the code
> > > > > > could just as well use a dynamically allocated array or a linked
> list.
> > > > > > It's just not the way it is right now and would need a patch
> > > > > > or
> > > > rewrite.
> > > > > > [...]
> > > > > A linked list could be used if you wanna optimize (dynamic)
> > > > > memory usage but an (amortized) array would probably provide
> > > > > faster linear search but I guess simplicity of the code and
> > > > > external functionality is the most important demands for this
> > > > > choice
> > > >
> > > > Any change here (aside from just increasing NPROCS) would have to
> > > > be done with care to avoid a performance hit.  I looked at the
> > > > history of changes to sigproc.cc, and I found commit 4ce15a49 in
> > > > 2001 in which a static array something like cprocs was replaced by
> > > > a dynamically allocated buffer in order to save DLL space.  This
> > > > was reverted 3 days later (commit e2ea684e) because of performance
> issues.
> > >
> > > I wonder what kind of performance issue ?
> > > [...]
> > I don't know for sure, but I doubt if it had anything to do with
> > memory access. My guess is that the performance hit came from the need
> > to free the allocated memory after every fork call (see
> sigproc_fixup_after_fork).
> 
> Either way, I rewrote this partially so we now have a default array size
> for 255 child processes on 32 bit and 1023 child processes on 64 bit.
> 
> The new code is mainly a minor update in that it convertes the code
> directly accessing stuff into using a class, encapsulating the mechanism
> used under the hood behind a class barrier and access methods.
> 
> As POC, I added a bit of code to maintain a second array, which is only
> allocated (using HeapAlloc so as not to spill into the child processes) if
> the default array overflows.  This second array adds room for another
> 1023 (32 bit) or 4095 (64 bit) child processes, raising the number of max
> child processes per process to 1278 on 32 bit and 5118 on 64 bit.
> 
> My STC just forking like crazy overflowed my 4 Gigs RAM + 2.5 Gigs
> pagefile after roughly 1450 child processes.  I'm pretty confident that
> this POC implementation is sufficient for a while, even in enterprise
> scenarios.
> 
> And if not, we can now easily tweak the numbers without having to tweak
> much of the code.


That is super

Otherwise amortized dynamic arrays (such as C++ std::vector) does provide
deterministic complexity and is proven work well for most use cases fairly
good with no surprise-hick-ups (hence, amortized). I guess the biggest
problem is (if?) how to determine when to shrink allocated memory when
number of children is reduced

Best regards,
Kristian



> For testing purposes I uploaded a developer snapshot to
> https://cygwin.com/snapshots/
> 
> 
> Corinna
> 
> --
> Corinna Vinschen
> Cygwin Maintainer
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple