X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:from:reply-to:to:message-id:subject
	:mime-version:content-type:content-transfer-encoding; q=dns; s=
	default; b=Wu5SpLtQcOrHWinXZwb7GoeDUx2rRJB2PXfDrnsbkgyi0o3osCX3B
	mK9j2sG7ZtMy3aw/oUyznEHCFWCpQxMcIHNHEkyiBdJoxiLF5B4mnJR85U8qLp8o
	qgKTri6yxd5qrK/zeNci9C/qpp5GxFLaDnxcVpL3T+WEcigcxKiQ28=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:from:reply-to:to:message-id:subject
	:mime-version:content-type:content-transfer-encoding; s=default;
	bh=VGeh65dgH/DRkq44/kv554ulQxw=; b=vG5sphPhn/bdF4bXvtdjMt4lFhW1
	AWGhlbYDIDkb8pn4UKSMwEARiOUBowF6EihGgV2xk1Z6d//3XRYxugda105w+spZ
	xTx/0urTbJ98x0sLTTrKlcS4EHmBvCiQsdATT+AnoxuecRgDc7vroAREO3wIg7vB
	1ouGcAYR6IWO4Sc=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Virus-Found:	No
X-Spam-SWARE-Status:	No, score=3.3 required=5.0 tests=AWL,BAYES_50,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_RP_MATCHES_RCVD autolearn=ham version=3.3.2
X-HELO:	nm50-vm3.bullet.mail.gq1.yahoo.com
Date:	Wed, 8 Oct 2014 22:35:25 +0000 (UTC)
From:	Nathan Fairchild <hashburydotnet AT yahoo DOT com>
Reply-To:	Nathan Fairchild <nfairc AT dnai DOT com>
To:	"cygwin AT cygwin DOT com" <cygwin AT cygwin DOT com>
Message-ID:	<113885868.183866.1412807725028.JavaMail.yahoo@jws10675.mail.bf1.yahoo.com>
Subject:	Running 4096 parallel cat processes on remote share results in only 1018 succeeding
MIME-Version:	1.0

When I run a script like so: 

#!/bin/bash 
ulimit -n 3200 
for i in $(seq $1) 
do exec cat /u/pe/env_files/transpath.map > out$i.log & 
done 

Where /u maps to a Windows share in /etc/fstab I get the following when running with 4096 instances (this happens whether I use a mapped /etc/fstab path or UNC path): 

$ ./run_many.sh 4096 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: Resource temporarily unavailable 

$ grep -l PATH out* | wc -l 
1018 


I think I'm probably hitting the 256 process limit because of the I/O slowdown the network presents? I don't get this issue running on (much faster) local disk. 

$ ulimit -a 
core file size          (blocks, -c) unlimited 
data seg size           (kbytes, -d) unlimited 
file size               (blocks, -f) unlimited 
open files                      (-n) 3200 
pipe size            (512 bytes, -p) 8 
stack size              (kbytes, -s) 2025 
cpu time               (seconds, -t) unlimited 
max user processes              (-u) 256 
virtual memory          (kbytes, -v) unlimited 


Is there a way to increase max user processes so I don't run into this limit? For my workload I need about 256 per cpu core (call it 8192), not 256 total. 

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -