DMARC-Filter:	OpenDMARC Filter v1.4.2 delorie.com 539AgmPZ2666789
Authentication-Results:	delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results:	delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter:	OpenDKIM Filter v2.11.0 delorie.com 539AgmPZ2666789
Authentication-Results:	delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=XwhBHenZ
X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 0C1D1384CBB5
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1744195366;
	bh=nzzKi01gyMgUZngh/WcFk4ofMzdxtnkBmo1I2cu9sww=;
	h=Subject:To:Cc:References:Date:In-Reply-To:List-Id:
	List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe:
	From:Reply-To:From;
	b=XwhBHenZxeMHZwgujTa/YCFy9isDcLlubiy1Iv0gDAs7NTBjPja6yEMhRkeou8B6I
	fJiaxBuC+5eOHlF9AUH3dAXCm4Wghsg6bKDEJq7wOKCfoncGzmfN3Ft8ySJrAqBcSy
	tzUgy1Km0E5DaITMUBR7hFdDU9yhNitNavaul+mo=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.2 sourceware.org 13481385841F
ARC-Filter:	OpenARC Filter v1.0.0 sourceware.org 13481385841F
ARC-Seal:	i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1744195302; cv=none;
	b=Wk3e0lw6tY9Ooqn8RWYVFY/XJydZqbYTkYA94WKKYiCkrWda6fQ1S2HJQAABuh+bqS9l/IeKzjQxXY5GbgOcM2EnuOxoXVXud2UFzbLsduMyE/ptFHfB0NVLp0riUu7KfkPn0BXUVMKG6UzfqET6jozTXho64bwWvX+LbGfQeaM=
ARC-Message-Signature:	i=1; a=rsa-sha256; d=sourceware.org; s=key;
	t=1744195302; c=relaxed/simple;
	bh=WABZaB4zJm7V2NUxoCBQQ21akudZrkLOEbLGx1I6FBM=;
	h=Subject:To:From:Message-ID:Date:MIME-Version;
	b=sN00yTv6jCUvo9e8juM0ioREyy8n4CwiZ+UKImPSe1ywA+5B2fH3YoeOKi5W6UZmIsB2nF7Gu32j4mVvhpB/WkhIFxZhj0TANmrxB1Yq1tEV8MkGSuQGQoWjuxlbenGU9U5tBRb1Gk9ZMbJksJZCstt0TJ20gXSRCmOLGnErWJc=
ARC-Authentication-Results:	i=1; server2.sourceware.org
Subject:	Re: Tuning ls, ls -l, find ., find . -ls performance for very large
	dirs (60000+ files/dir)
To:	cygwin AT cygwin DOT com
Cc:	cygwin AT cygwin DOT com
References:	<CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A AT mail DOT gmail DOT com>
	<4efd217b-eb94-42d8-bcc7-0f19de188ac0 AT gmx DOT com>
	<6e80b179-02c0-4972-b4fe-7fe0c2fef8a6 AT SystematicSW DOT ab DOT ca>
Message-ID:	<8f4eba88-858b-b535-eab0-5e930317700f@t-online.de>
Date:	Wed, 9 Apr 2025 12:41:24 +0200
User-Agent:	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101
	SeaMonkey/2.53.20
MIME-Version:	1.0
In-Reply-To:	<6e80b179-02c0-4972-b4fe-7fe0c2fef8a6@SystematicSW.ab.ca>
X-TOI-EXPURGATEID:	150726::1744195284-217E9509-D4D983B1/0/0 CLEAN NORMAL
X-TOI-MSGID:	38a488c1-8273-4067-94a6-0deb2a4f5f81
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.30
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe:	<https://cygwin.com/mailman/options/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	Christian Franke via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	Christian Franke <Christian DOT Franke AT t-online DOT de>
Errors-To:	cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender:	"Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
X-MIME-Autoconverted:	from base64 to 8bit by delorie.com id 539AgmPZ2666789

Brian Inglis via Cygwin wrote:
> On 08/04/2025 19:05, Mark Liam Brown via Cygwin wrote:
>> Are there tuning variables to improve ls, ls -l, find ., find . -ls
>> performance for very large dirs?
>>
>> If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
>> minutes (22+mins), while cmd.exe dir just floods the terminal with
>> results immediately.
>
> I just remembered noticing my /var/log/ entries jumped looking at my 
> last Cygwin upgrade log, so before I clean it up into subdirs, let's 
> try some stuff!
>
> If you just want a directory name dump, use \ls -U|--sort=none or \ls 
> -f which is like -aU; command prefix \ overrides the usual aliases 
> like -CF --color=auto:
>
> $ time \ls -1U /var/log/ | \wc -lwmcL
>   54049   54049 1885787 1885787      38
>
> real    0m0.336s
> user    0m0.046s
> sys    0m0.295s
> $ time \ls -1f /var/log/ | \wc -lwmcL
>   54051   54051 1885792 1885792      38
>
> real    0m0.347s
> user    0m0.140s
> sys    0m0.232s
>
> even with -l, it's only a few seconds:
>
> $ time \ls -lU /var/log/ | \wc -lwmcL
>   54050  486455 4966716 4966716     102
>
> real    0m15.891s
> user    0m1.421s
> sys    0m14.560s
> $ time \ls -lf /var/log/ | \wc -lwmcL
>   54052  486473 4966835 4966835     102
>
> real    0m15.858s
> user    0m1.405s
> sys    0m14.374s
>

One reason for the above difference is possibly that Cygwin supports the 
'dirent.d_type' field for quite some time and tools like 'ls' and 'find' 
use it to avoid stat() calls.

Fast - stat() calls avoided:

   ls -R
   find
   find -type f -name 'some*file'

Fast if only few entries require stat():

   find -type f -name 'some*file' -mtime +42 # only -mtime requires stat()
   find -mtime +42 -type f -name 'some*file'  # find does 'cheap' checks 
first

Slow - requires stat() for each entry:

   ls -Rt
   ls -RS
   ls -Rl
   find -mtime +42
   find -ls

-- 
Regards,
Christian


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -