X-Recipient: archive-cygwin AT delorie DOT com
DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:message-id:date:from:mime-version:to:subject
	:references:in-reply-to:content-type:content-transfer-encoding;
	 q=dns; s=default; b=qaGc3b2xjFub3pNXDUZm5Zoprxr1KEdBhgyeY0GNnbA
	8shETsKPCnIErFonSP9E1mpfCL7GwBOIxalEzoJlZSALIJ+ws0utOWcuB+GJpowW
	WeAwL0gSVUB856Q5R8N8M5A3OUWj3WkmqM0/oTCP5Eq1o50LyZVrFr3VKVidd+Nk
	=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:message-id:date:from:mime-version:to:subject
	:references:in-reply-to:content-type:content-transfer-encoding;
	 s=default; bh=3f2XkMDzENIvkXZVVYRm9BCYbTM=; b=LaxsrcLmn6nh7WxoW
	K1iQkXrOCblmDEl0QAXynaFQXOcScr0bYFO6s0EbcFerk/LtHb9zG0jRRBl14XrJ
	ifbzcgel230WvloKBPoRFhuDCEpxSp4gVYZOiwX+/KLO/TESUx2EyvXHihp08kJ4
	9XKoj4DbALFpxHp6cG8Vf/bx5M=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id: <cygwin.cygwin.com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
Authentication-Results: sourceware.org; auth=none
X-Virus-Found: No
X-Spam-SWARE-Status: No, score=-1.3 required=5.0 tests=AWL,BAYES_00,T_RP_MATCHES_RCVD autolearn=ham version=3.3.2
X-HELO: Ishtar.hs.tlinx.org
Message-ID: <54374D09.3020504@tlinx.org>
Date: Thu, 09 Oct 2014 20:05:45 -0700
From: Linda Walsh <cygwin AT tlinx DOT org>
User-Agent: Thunderbird
MIME-Version: 1.0
To: "cygwin AT cygwin DOT com" <cygwin AT cygwin DOT com>
Subject: Re: Running 4096 parallel cat processes on remote share results in only 1018 succeeding
References: <113885868 DOT 183866 DOT 1412807725028 DOT JavaMail DOT yahoo AT jws10675 DOT mail DOT bf1 DOT yahoo DOT com>
In-Reply-To: <113885868.183866.1412807725028.JavaMail.yahoo@jws10675.mail.bf1.yahoo.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-IsSubscribed: yes

Nathan Fairchild wrote:
> When I run a script like so:
> cat: /u/pe/env_files/transpath.map: No such file or directory 
> ./run_many.sh: fork: retry: Resource temporarily unavailable 
> ./run_many.sh: fork: Resource temporarily unavailable 
> 
> $ grep -l PATH out* | wc -l 
> 1018 

> I think I'm probably hitting the 256 process limit because of the I/O slowdown the network presents? I don't get this issue running on (much faster) local disk. 
---
You are only reading from the net? or are you copying
to the to the net too?  I.e. what is your CWD?  is out$i.log
on the net or local?  I tried it locally and couldn't
reproduce your symptoms.

Your problem is more likely the server hosting the remote file system.

While you can write files locally that fast, the remote server adds
enough "ms"/file delay that it can't keep up.  Even processing your
Network requests take cpu time.

/i/fdlims> cat mrun
#!/bin/bash
ulimit -n 3200
for i in $(seq $1)
do exec cat mrun >/tmp/tmp$i.log&
done
/i/fdlims> ll /tmp/tmp*|wc
    4096   28672  191405
---
I'm not sure how accurate the ulimit command inside
cygwin is... may be accurate, just saying I don't know.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple