X-Spam-Check-By: sourceware.org
Date: Wed, 15 Nov 2006 11:51:33 -0500
From: Christopher Faylor <cgf-no-personal-reply-please AT cygwin DOT com>
To: cygwin AT cygwin DOT com
Subject: Re: Excessive thrashing when popen()ing after a large malloc()
Message-ID: <20061115165133.GA21115@trixie.casa.cgf.cx>
Reply-To: cygwin AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
References: <Pine DOT LNX DOT 4 DOT 58 DOT 0611141146430 DOT 9574 AT u53026a DOT matapp DOT unimib DOT it> <4559D376 DOT 9030707 AT byu DOT net> <loom DOT 20061115T171109-921 AT post DOT gmane DOT org>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <loom.20061115T171109-921@post.gmane.org>
User-Agent: Mutt/1.5.11
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Precedence: bulk
List-Unsubscribe: <mailto:cygwin-unsubscribe-archive-cygwin=delorie DOT com AT cygwin DOT com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com

On Wed, Nov 15, 2006 at 04:18:10PM +0000, Eric Blake wrote:
>Eric Blake <ebb9 <at> byu.net> writes:
>>>     else
>>>       sprintf(cmd, "sh -c '%s' %d>&-", program, pdes[1])
>>>   }
>>>   pid = spawnl(_P_NOWAIT, _PATH_BSHELL, "sh", "-c", cmd, NULL);
>>
>Why are you going through two levels of sh?  That seems like a waste to
>me; the whole idea of using spawn is to avoid a fork(), but when you
>>invoke "sh" "-c" "sh -c 'cmd'", you are right back to a fork.  True,
>>the new invocation of sh uses less memory than the 1 GB process that
>>invoked popen, so less thrashing will occur, but your whole approach
>>seems fundamentally flawed if you are trying to use spawn to avoid a
>>fork.
>
>You know, maybe a better approach to this would be to first implement
>posix_spawn() in cygwin, since that API seems like it would be possible
>to do the necessary fd manipulation in a spawned child process without
>the penalty of a full fork/exec.  Then you could teach popen how to use
>posix_spawn when it is available, which is probably a patch more likely
>to be accepted in newlib, rather than your current idea of having to go
>through an intermediary sh invocation.  But I do not have a copyright
>on file for cygwin contributions, so I am currently unable to work on
>contributing posix_spawn; you are at the mercy of someone willing and
>able to write such a patch.

Fully implementing posix_spawn is quite a bit of work although I did
tell Linus Torvalds that I'd look into implementing it in Cygwin
eventually (it would make git-on-cygwin faster).

I have a patch in my sandbox which implements a more lightweight
approach for popen using cygwin internals.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/