X-Spam-Check-By: sourceware.org
To: cygwin AT cygwin DOT com
Subject: Re: Excessive thrashing when popen()ing after a large malloc()
In-Reply-To: <loom.20061115T171109-921@post.gmane.org>
Message-Id: <20061115200351.DB4C514425F7@u53026a.matapp.unimib.it>
Date: Wed, 15 Nov 2006 21:03:51 +0100 (CET)
From: grenie AT matapp DOT unimib DOT it (Loic Grenie)
X-IsSubscribed: yes
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Precedence: bulk
List-Unsubscribe: <mailto:cygwin-unsubscribe-archive-cygwin=delorie DOT com AT cygwin DOT com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com

Eric Blake <ebb9 <at> byu.net> writes:
> Eric Blake <ebb9 <at> byu.net> writes:
>
> > >     else
> > >       sprintf(cmd, "sh -c '%s' %d>&-", program, pdes[1])
> > >   }
> > >   pid = spawnl(_P_NOWAIT, _PATH_BSHELL, "sh", "-c", cmd, NULL);
> > 
> > Why are you going through two levels of sh?  That seems like a waste to
> > me; the whole idea of using spawn is to avoid a fork(), but when you
> > invoke "sh" "-c" "sh -c 'cmd'", you are right back to a fork.  True, the
> > new invocation of sh uses less memory than the 1 GB process that invoked
> > popen, so less thrashing will occur, but your whole approach seems
> > fundamentally flawed if you are trying to use spawn to avoid a fork.
>
> You know, maybe a better approach to this would be to first implement 
> posix_spawn() in cygwin, since that API seems like it would be possible
> to do the necessary fd manipulation in a spawned child process without the
> penalty of a full fork/exec.  Then you could teach popen how to use
> posix_spawn when it is available, which is probably a patch more likely to
> be accepted in newlib, rather than your current idea of having to go through
> an intermediary sh invocation.

    That intermediate sh invocation can probably be slightly streamlined
  by doing something similar to

sprintf(cmd, "(%s) >&%d %d>&- %d>&-", program, pdes[1], pdes[0], pdes[1]);
pid = spawnl(_P_NOWAIT, _PATH_BSHELL, "sh", "-c", cmd, NULL);

  but you misunderstand my objective: I'm not "trying to have my patch
  accepted", I'm just signaling a problem and proposing a patch for the
  upstream maintainers: cygwin and/or newlib. I've modified the program
  that uses popen and it is now using a _popen which is a kludge around
  the ideas that I have suggested. If my ideas are useful for anybody else,
  I'm really happy to contribute and help; if nobody cares, neither do I.

> But I do not have a copyright on file for cygwin contributions, so I am
> currently unable to work on contributing posix_spawn;

    That's very nice of you to suggest it.

> you are at the mercy of someone willing and able to write such a patch.

    Actually, as said above, /I/ am not at the mercy of anyone, but anybody
  else who popen()'s with a large memory space is.

> -- 
> Eric Blake

    Loïc Grenié

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/