DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:from:reply-to:to:message-id:subject
	:mime-version:content-type:content-transfer-encoding; q=dns; s=
	default; b=Wu5SpLtQcOrHWinXZwb7GoeDUx2rRJB2PXfDrnsbkgyi0o3osCX3B
	mK9j2sG7ZtMy3aw/oUyznEHCFWCpQxMcIHNHEkyiBdJoxiLF5B4mnJR85U8qLp8o
	qgKTri6yxd5qrK/zeNci9C/qpp5GxFLaDnxcVpL3T+WEcigcxKiQ28=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Date: Wed, 8 Oct 2014 22:35:25 +0000 (UTC)
From: Nathan Fairchild <hashburydotnet AT yahoo DOT com>
Reply-To: Nathan Fairchild <nfairc AT dnai DOT com>
To: "cygwin AT cygwin DOT com" <cygwin AT cygwin DOT com>
Message-ID: <113885868.183866.1412807725028.JavaMail.yahoo@jws10675.mail.bf1.yahoo.com>
Subject:  Running 4096 parallel cat processes on remote share results in only 1018 succeeding
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit


When I run a script like so: 

#!/bin/bash 
ulimit -n 3200 
for i in $(seq $1) 
do exec cat /u/pe/env_files/transpath.map > out$i.log & 
done 

Where /u maps to a Windows share in /etc/fstab I get the following when running with 4096 instances (this happens whether I use a mapped /etc/fstab path or UNC path): 

$ ./run_many.sh 4096 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: Resource temporarily unavailable 

$ grep -l PATH out* | wc -l 
1018 


I think I'm probably hitting the 256 process limit because of the I/O slowdown the network presents? I don't get this issue running on (much faster) local disk. 

$ ulimit -a 
core file size          (blocks, -c) unlimited 
data seg size           (kbytes, -d) unlimited 
file size               (blocks, -f) unlimited 
open files                      (-n) 3200 
pipe size            (512 bytes, -p) 8 
stack size              (kbytes, -s) 2025 
cpu time               (seconds, -t) unlimited 
max user processes              (-u) 256 
virtual memory          (kbytes, -v) unlimited 


Is there a way to increase max user processes so I don't run into this limit? For my workload I need about 256 per cpu core (call it 8192), not 256 total. 

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple