From: "David Tazartes" <davidt81 AT earthlink DOT net>
To: <cygwin AT cygwin DOT com>
Subject: Simple bash script is slow to execute - appears to be time spent starting commands like ls
Date: Wed, 2 Sep 2009 13:01:06 -0400
MIME-Version: 1.0
Content-Type: text/plain; 	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-ID: <E1MitCo-0004Gx-Qn@elasmtp-scoter.atl.sa.earthlink.net>
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com

All my bash scripts appear to run very slowly. But I have one very simple
example in particular here, so I figure I'd use it to ask if there's
anything I can do.

This very simple script (below) takes about 2 seconds per for loop
execution. That seems very high to me, based on my experience as a developer
on Linux, and comparing to a slow Atom-based ubuntu server. All the script
is doing is checking the file date and then moving the file to a
subdirectory (no file copy) based on the date.

LS=`which ls`
CUT=`which cut`
MV=`which mv`
MKDIR=`which mkdir`

for picfile in P[0-9][0-9][0-9][0-9][0-9][0-9][0-9].[Jj][Pp][Gg]
P[0-9][0-9][0-9][0-9][0-9][0-9][0-9].[Mm][Oo][Vv] ; do
	pictime=$($LS --time-style='+%Y%m%d' -ong $picfile | $CUT -d ' ' -f
4)
	picdate=${pictime:3:5}
	
	echo "Moving $picfile (time=$pictime) to $picdate"
	$MKDIR -p $picdate
	$MV -uv $picfile $picdate
done

System facts:
- Vista 32-bit on two-year-old low-end laptop
- ThreatFire and Symantec Antivirus both running normally; but disabling
them doesn't improve performance
- There is a network drive (w) mounting to a Samba share, but it's not
involved in this operation
- My username has a space in it, and the CWD for the script is
"C:\Users\user name\Desktop\another folder" ("another folder" has a space)
aka /cygdrive/c/Users/user name/Desktop/another folder

Cygwin config and other file facts:
- Same problem on both 1.5.25-15 and 1.7 (timings below are for 1.5, 1.7 is
slightly faster but still slow)
- Tried running bash from cmd, and using rxvt. No (or undetectable)
difference.
- PATH set to /usr/bin only
- umask set to 077 to avoid "Everyone" issue when creating directories
- File permissions start out as 500
- This sorted about 2000 files, but was no faster when there were fewer
files remaining.

Observations:
- When using bash -x, most of the time appears to be spent in "cut" and
"mkdir".
- When looking at Task Manager sorting by CPU, I see Windows Explorer
(explorer.exe, NOT Internet Explorer) suddenly popping up at around 10-20%
after every file is processed. This happens even if I have no Explorer
windows open.
- I also see ls, cut, mkdir and mv all popping up, but in pretty slow
succession.
- Even just typing "ls" in an empty directory pauses for ~ 1 second.

- "time mkdir abcd" results in:
real    0m0.544s
user    0m0.108s
sys     0m0.233s
My ubuntu server gives:
real    0m0.005s
user    0m0.000s
sys     0m0.004s

- "time echo hi | /usr/bin/cut -d ' ' -f 1" results in:
real    0m0.816s
user    0m0.310s
sys     0m0.311s
My ubuntu server gives:
real    0m0.006s
user    0m0.004s
sys     0m0.000s

Hypotheses:
- The "slow forking" issue may be biting me. It seems like each sub-process
takes forever to spin up. I haven't found any documentation as to how to
avoid this, though.
- strace shows endless filename conversions, and though they're not terribly
slow (1-2 ms each), there are lots of them.
- I thought it might be the "window title" issue but running from rxvt
should have taken care of that, and it doesn't.

Answers I'm not interested in:
- "Windows sucks, that's why it's slow"
- "Cygwin is just an emulation layer so you should expect it to be slow"
- "Why should you think it would take less than 2 seconds per loop
execution?" I think timings of the ubuntu server, and common sense
especially in the "echo | cut" case, sort of answer that question.

Thanks in advance for any advice.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple