Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
Subject: Re: slow handling of large sets of files?
From: Ken Sheldon <Ken_Sheldon AT ACM DOT org>
To: Cygwin List <cygwin AT cygwin DOT com>
In-Reply-To: <6.2.0.14.0.20050201144916.05a42210@pop.prospeed.net>
References: <1107286130 DOT 3063 DOT 65 DOT camel AT slb163188094105 DOT sugar-land DOT nam DOT slb DOT com> <6 DOT 2 DOT 0 DOT 14 DOT 0 DOT 20050201144916 DOT 05a42210 AT pop DOT prospeed DOT net>
Content-Type: text/plain
Date: Tue, 01 Feb 2005 14:22:41 -0600
Message-Id: <1107289362.3063.78.camel@slb163188094105.sugar-land.nam.slb.com>
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
X-ELNK-Trace: 4fdb074ad6db0e67d9b80cf7b625bd5c9ef193a6bfc3dd488fd317fe66b35e0ee1e0b92cf11b13c67ef9f80aaf77e5a4350badd9bab72f9c350badd9bab72f9c
X-IsSubscribed: yes

On Tue, 2005-02-01 at 14:51 -0500, Larry Hall wrote:
> At 02:28 PM 2/1/2005, you wrote:
> >I too have been seeing a problem with very slow file access in large
> >directories.
> >
> >Specifically,
> >
> >On a Cygwin/Win2k box, I have a mirror of an FTP site.  The site has 2.5
> >million files spread between 100 directories.  
> >(20,000 - 30,000 files per directory)  I have previously run this number
> >of files in an NT4 NTFS filesystem without significant performance
> >problems.
> >
> >In this site, operations like these are __VERY__ slow.
> >ls ${some_dir}
> >ls ${some_dir}/${some_path}
> >cp ${some_file} ${some_path}
> >cp -R ${some_path_with _only_a_few_files} ${some_path}
> >
> >
> >If I look at the performance monitor, I can see a queue depth of 1-2 and
> >300-500 disk reads per second.  (That's real.  It's a fast array)  The
> >reads appear to be single-block reads, as the throughput during these
> >events is 1.5 - 3MB/sec.
> >
> >I am beginning to think the disk activity relates to NTFS permission
> >checking, which can be complex under Win2k.
> >
> >I don't know how to debug or tune this.
> >
> >Any ideas?
> 
> 
> Have you looked at the '-x', '-X', and '-E' flags of mount?
> 

Hmm... Thats an interesting guess, but neither -e, -X nor -x seem to
have any measurable effect.  
 
$ /bin/ls -F 
00/ 07/ 14/ 21/ 28/ 35/ 42/ 49/ 56/ 63/ 70/ 77/ 84/ 91/ 98/
01/ 08/ 15/ 22/ 29/ 36/ 43/ 50/ 57/ 64/ 71/ 78/ 85/ 92/ 99/
02/ 09/ 16/ 23/ 30/ 37/ 44/ 51/ 58/ 65/ 72/ 79/ 86/ 93/
03/ 10/ 17/ 24/ 31/ 38/ 45/ 52/ 59/ 66/ 73/ 80/ 87/ 94/
04/ 11/ 18/ 25/ 32/ 39/ 46/ 53/ 60/ 67/ 74/ 81/ 88/ 95/
05/ 12/ 19/ 26/ 33/ 40/ 47/ 54/ 61/ 68/ 75/ 82/ 89/ 96/
06/ 13/ 20/ 27/ 34/ 41/ 48/ 55/ 62/ 69/ 76/ 83/ 90/ 97/
 
$ time /bin/ls 04 | wc -l
29551
 
real    0m53.297s
user    0m0.232s
sys     0m0.733s
 
More information:
Not only Cygwin apps incur this large performance penalty.  Something
similar happens with the cmd.exe prompt command "DIR", with the windows
file explorer, or with IIS (FTP server).  This only seems to happen in
the directory structures created by my CygWin scripts (using apps: tar,
wget, cp)


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/