Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Subject:	Re: slow handling of large sets of files?
From:	Ken Sheldon <Ken_Sheldon AT ACM DOT org>
To:	cygwin AT cygwin DOT com
Date:	Tue, 01 Feb 2005 13:28:50 -0600
Message-Id:	<1107286130.3063.65.camel@slb163188094105.sugar-land.nam.slb.com>
Mime-Version:	1.0
X-ELNK-Trace:	4fdb074ad6db0e67d9b80cf7b625bd5c9ef193a6bfc3dd488fd317fe66b35e0e4bff538fac14e290350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c
X-IsSubscribed:	yes

I too have been seeing a problem with very slow file access in large
directories.

Specifically,

On a Cygwin/Win2k box, I have a mirror of an FTP site.  The site has 2.5
million files spread between 100 directories.  
(20,000 - 30,000 files per directory)  I have previously run this number
of files in an NT4 NTFS filesystem without significant performance
problems.

In this site, operations like these are __VERY__ slow.
ls ${some_dir}
ls ${some_dir}/${some_path}
cp ${some_file} ${some_path}
cp -R ${some_path_with _only_a_few_files} ${some_path}


If I look at the performance monitor, I can see a queue depth of 1-2 and
300-500 disk reads per second.  (That's real.  It's a fast array)  The
reads appear to be single-block reads, as the throughput during these
events is 1.5 - 3MB/sec.

I am beginning to think the disk activity relates to NTFS permission
checking, which can be complex under Win2k.

I don't know how to debug or tune this.

Any ideas?


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

- Raw text -