X-Recipient: archive-cygwin AT delorie DOT com DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:date:from:reply-to:to:message-id:subject :mime-version:content-type:content-transfer-encoding; q=dns; s= default; b=Wu5SpLtQcOrHWinXZwb7GoeDUx2rRJB2PXfDrnsbkgyi0o3osCX3B mK9j2sG7ZtMy3aw/oUyznEHCFWCpQxMcIHNHEkyiBdJoxiLF5B4mnJR85U8qLp8o qgKTri6yxd5qrK/zeNci9C/qpp5GxFLaDnxcVpL3T+WEcigcxKiQ28= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:date:from:reply-to:to:message-id:subject :mime-version:content-type:content-transfer-encoding; s=default; bh=VGeh65dgH/DRkq44/kv554ulQxw=; b=vG5sphPhn/bdF4bXvtdjMt4lFhW1 AWGhlbYDIDkb8pn4UKSMwEARiOUBowF6EihGgV2xk1Z6d//3XRYxugda105w+spZ xTx/0urTbJ98x0sLTTrKlcS4EHmBvCiQsdATT+AnoxuecRgDc7vroAREO3wIg7vB 1ouGcAYR6IWO4Sc= Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner AT cygwin DOT com Mail-Followup-To: cygwin AT cygwin DOT com Delivered-To: mailing list cygwin AT cygwin DOT com Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=3.3 required=5.0 tests=AWL,BAYES_50,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_RP_MATCHES_RCVD autolearn=ham version=3.3.2 X-HELO: nm50-vm3.bullet.mail.gq1.yahoo.com Date: Wed, 8 Oct 2014 22:35:25 +0000 (UTC) From: Nathan Fairchild Reply-To: Nathan Fairchild To: "cygwin AT cygwin DOT com" Message-ID: <113885868.183866.1412807725028.JavaMail.yahoo@jws10675.mail.bf1.yahoo.com> Subject: Running 4096 parallel cat processes on remote share results in only 1018 succeeding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit When I run a script like so: #!/bin/bash ulimit -n 3200 for i in $(seq $1) do exec cat /u/pe/env_files/transpath.map > out$i.log & done Where /u maps to a Windows share in /etc/fstab I get the following when running with 4096 instances (this happens whether I use a mapped /etc/fstab path or UNC path): $ ./run_many.sh 4096 cat: /u/pe/env_files/transpath.map: No such file or directory cat: /u/pe/env_files/transpath.map: No such file or directory cat: /u/pe/env_files/transpath.map: No such file or directory cat: /u/pe/env_files/transpath.map: No such file or directory cat: /u/pe/env_files/transpath.map: No such file or directory ./run_many.sh: fork: retry: Resource temporarily unavailable ./run_many.sh: fork: retry: Resource temporarily unavailable ./run_many.sh: fork: retry: Resource temporarily unavailable ./run_many.sh: fork: retry: Resource temporarily unavailable ./run_many.sh: fork: Resource temporarily unavailable $ grep -l PATH out* | wc -l 1018 I think I'm probably hitting the 256 process limit because of the I/O slowdown the network presents? I don't get this issue running on (much faster) local disk. $ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 3200 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 2025 cpu time (seconds, -t) unlimited max user processes (-u) 256 virtual memory (kbytes, -v) unlimited Is there a way to increase max user processes so I don't run into this limit? For my workload I need about 256 per cpu core (call it 8192), not 256 total. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple