X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:subject:to:references:from:message-id:date
	:mime-version:in-reply-to:content-type
	:content-transfer-encoding; q=dns; s=default; b=TJSS01WEXtb8x05s
	gilmsH8j0+eHIBcTHw1hnM2my5Fv9wbj5CeXDU3H85C4LUbQ3JUO9SluR0s2EN9K
	igq3oAhB6/7dmTIjJeRkSVEOceqkr0n0r5FLvOyCHnwHF5ohyhYrt5Lx6tWuUIE8
	oA8bJgffxk+WK+o5PK/TZpZ9oAQ=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:subject:to:references:from:message-id:date
	:mime-version:in-reply-to:content-type
	:content-transfer-encoding; s=default; bh=6fxUGKyDFO66gk+VPpjVk2
	sDYvs=; b=s2/2yv9xTG7+O68e2bN8g01V3NRJai5NbUcrLwpT8NgWqnodiiky69
	k7WgQL9MPMLb09zEulEbIL/fDIUYl/yKQlGVeYd5vGzReezF7aG48hSMkgeNij23
	y93GoZxQ6Nz/ZbBRa6eXZRt8GGVfehS3MTP8eCptpS+OWVTBoqnRQ=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Virus-Found:	No
X-Spam-SWARE-Status:	No, score=-2.6 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_LOW,RP_MATCHES_RCVD,SPF_PASS autolearn=ham version=3.3.2 spammy=watching, lightweight, organize, LOT
X-HELO:	sasl.smtp.pobox.com
Subject:	Re: Strange errors running gcc tests on Cygwin
To:	cygwin AT cygwin DOT com
References:	<8fa02a72-e684-2ead-eacb-a5347d7594ae AT pobox DOT com> <82b31abc-7b7f-8f13-fc22-521c9ef84abf AT pobox DOT com> <8bda181f-f0bc-b0dc-2d2d-1bb17031ccee AT gmail DOT com> <b6c0f704-1f04-63ea-8220-ee30d623a9d6 AT pobox DOT com> <583230d9-f45c-aaa0-ed77-5c50863406f5 AT gmail DOT com> <9b872914-d9cf-378e-6eec-96c175a61ffe AT pobox DOT com> <ab35a49e-93e3-4384-bf81-3e8ac99b7194 AT gmail DOT com> <7372df4f-c55d-f9a3-325d-3f8800d67d98 AT pobox DOT com> <aab8c02e-2fe7-8ebf-5f3b-126e0ea0a745 AT gmail DOT com> <a7aca745-bbf2-a356-704a-4956ae6a28b3 AT pobox DOT com> <937197c6-f0cd-a7a0-a12f-d3f943ba2c1d AT SystematicSw DOT ab DOT ca>
From:	Daniel Santos <daniel DOT santos AT pobox DOT com>
Message-ID:	<affe4ca0-4677-6eb9-1d7d-94a129adc30c@pobox.com>
Date:	Thu, 9 Mar 2017 16:53:03 -0600
User-Agent:	Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1
MIME-Version:	1.0
In-Reply-To:	<937197c6-f0cd-a7a0-a12f-d3f943ba2c1d@SystematicSw.ab.ca>
X-Pobox-Relay-ID:	9253CE0C-051A-11E7-BB2A-97B1B46B9B0B-06139138!pb-smtp1.pobox.com
X-IsSubscribed:	yes

First of all, thank you for your response!

On 03/08/2017 02:21 AM, Brian Inglis wrote:
> After any Windows Update, or a lot of package installs, you may want
> look at running
> 	rebase-trigger full[rebase]
> before rebooting to remove all Cygwin and Windows processes, then
> (with no Cygwin services or processes running) run Cygwin
> setup-x86{,_64} to rebase everything and maximize memory available
> to Cygwin processes.

So does rebase organize the Cygwin dlls and reassign them all new base 
loading addresses?  So far I've only done the 64-bit tests using Cygwin. 
I haven't even gotten to 64-bit mingw or 32 bit. Also, I have *not* done 
a Windows Update since I have installed. Could this be an OS issue that 
was addressed in some patch?  I'm running Windows 7 Ultimate SP1.

> This could be critical if you are doing any builds under Cygwin 32
> and have a lot of packages and/or large exes/dlls installed.
>
> If you are running a lot of Cygwin services, cron or Scheduled Tasks,
> and/or background processes, you may want to look at running cygserver
> to cache process info and common system info (including SAM/AD).

I'm only running sshd -- no cron or "at" jobs (except whatever Windows 
installs by its self).  However, gcc's make check spawns a LOT of processes.

> Setup cygserver initially by running cygserver-config with
> elevated/admin rights, and start it at system startup by running
> cygrunsrv with elevated/admin rights.
> If you ever expect to be running more than 62 simultaneous Cygwin
> processes on a system, bump kern.srv.process_cache_size in
> /etc/cygserver.conf created by cygserver-config to one of the higher
> values recommended in the man page.
> I found this seemed to reduce process startup overhead and eliminated
> random broken pipes, hung, and failed processes due to a lot of shell
> forking.
I didn't know about cygserver, so I tried it out and set the 
kern.srv.process_cache_size to 256.  I ran a make -kj4 check and had 
broken pipes within a few short minutes.  I tried it again running make 
-k check, but on two directories at once and got broken pipes somewhere 
along the way (before they completed).  So I should probably try it with 
only a single make running at once.

Even with just the two instances of make (with one job each), there are 
many processes of bash, make, expect, etc.  That's just one of the 
challenges of a Cygwin venture -- Windows was never designed for light 
process creation (or thread creation for that matter).  A lot of the 
*nix software concepts are hinged upon that lightweight, copy-on-write 
fork, whereas Windows-based concepts seem to be centered more around 
pre-spawning and maintaining thread pools and task-specific services.

> Preallocate contiguous paging space at double RAM to reduce the chance
> that any set of Windows processes will reduce Windows free memory too
> low for an instant, and cause something to hang or fail, by giving
> Windows somewhere to page out a lot of LRU pages from inactive
> processes.

I'm not sure how to go about this.  Should I bring the VM down, mount 
the image with ntfs-g3 and dd if=/dev/zero it or is there another way?  
Also, I don't know what Windows expects to see in a pagefile.  Or maybe 
defrag. I'll look at it.

> I found that if Windows (at least up to W7) free memory ever got too
> low, especially when all processors are pegged, it seemed unable to
> dynamically allocate and use more paging space to alleviate the
> crunch.

 From watching the system monitor, I can see that I never get above 35% 
memory usage.  I've assigned 4GiB of ram to the VM and Windows never 
seems to use more than 280MiB for page cache (I guess that's due to the 
license-specific limitation) -- at least there is abundant caching 
happening at the host level (3-ish GiB).

> The situation may have improved on Windows 10, but I've already
> allocated the paging space, and uninstalled some (probably buggy)
> hogs that seemed to only ever allocate memory and never free any.
>
> YMMV

As far as the dll search path issue, I *think* I have a woorking 
patchset in gcc and DejaGnu to resolve the dll load problems.  (I 
haven't yet done the on-off switch yet, so it's basically hard-coded.  I 
found a Tcl "is_Cygwin" procedure in git-gui that I'm going to borrow 
for now.)  So while my make check is still failing with broken pipes, I 
appear to have resolved the issue with the dll search path.

It would probably help if I can figure out where the broken pipe 
actually occurs and try to get that piece to log more verbosly.  I'm 
guessing it's from expect.  This is the exact error message:

FAIL: gcc.c-torture/execute/20000822-1.c   -O2 -flto -fuse-linker-plugin 
-fno-fat-lto-objects  execution test
parent: sync byte write: broken pipe^M
make[3]: Leaving directory 
'/d/builds/head-test-moutline-x86_64-pc-cygwin/gcc'

Don't worry about the FAILed test.  The main thing is that after each 
broken pipe message, I see make "leaving" that directory, so it doesn't 
finish the rest of the tests in that directory.

Daniel

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -