DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 539AgmPZ2666789
Authentication-Results: delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results: delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter: OpenDKIM Filter v2.11.0 delorie.com 539AgmPZ2666789
Authentication-Results: delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=XwhBHenZ
X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 0C1D1384CBB5
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1744195366;
	bh=nzzKi01gyMgUZngh/WcFk4ofMzdxtnkBmo1I2cu9sww=;
	h=Subject:To:Cc:References:Date:In-Reply-To:List-Id:
	 List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe:
	 From:Reply-To:From;
	b=XwhBHenZxeMHZwgujTa/YCFy9isDcLlubiy1Iv0gDAs7NTBjPja6yEMhRkeou8B6I
	 fJiaxBuC+5eOHlF9AUH3dAXCm4Wghsg6bKDEJq7wOKCfoncGzmfN3Ft8ySJrAqBcSy
	 tzUgy1Km0E5DaITMUBR7hFdDU9yhNitNavaul+mo=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 13481385841F
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 13481385841F
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1744195302; cv=none;
 b=Wk3e0lw6tY9Ooqn8RWYVFY/XJydZqbYTkYA94WKKYiCkrWda6fQ1S2HJQAABuh+bqS9l/IeKzjQxXY5GbgOcM2EnuOxoXVXud2UFzbLsduMyE/ptFHfB0NVLp0riUu7KfkPn0BXUVMKG6UzfqET6jozTXho64bwWvX+LbGfQeaM=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1744195302; c=relaxed/simple;
 bh=WABZaB4zJm7V2NUxoCBQQ21akudZrkLOEbLGx1I6FBM=;
 h=Subject:To:From:Message-ID:Date:MIME-Version;
 b=sN00yTv6jCUvo9e8juM0ioREyy8n4CwiZ+UKImPSe1ywA+5B2fH3YoeOKi5W6UZmIsB2nF7Gu32j4mVvhpB/WkhIFxZhj0TANmrxB1Yq1tEV8MkGSuQGQoWjuxlbenGU9U5tBRb1Gk9ZMbJksJZCstt0TJ20gXSRCmOLGnErWJc=
ARC-Authentication-Results: i=1; server2.sourceware.org
Subject: Re: Tuning ls, ls -l, find ., find . -ls performance for very large
 dirs (60000+ files/dir)
To: cygwin AT cygwin DOT com
Cc: cygwin AT cygwin DOT com
References: <CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A AT mail DOT gmail DOT com>
 <4efd217b-eb94-42d8-bcc7-0f19de188ac0 AT gmx DOT com>
 <6e80b179-02c0-4972-b4fe-7fe0c2fef8a6 AT SystematicSW DOT ab DOT ca>
Message-ID: <8f4eba88-858b-b535-eab0-5e930317700f@t-online.de>
Date: Wed, 9 Apr 2025 12:41:24 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101
 SeaMonkey/2.53.20
MIME-Version: 1.0
In-Reply-To: <6e80b179-02c0-4972-b4fe-7fe0c2fef8a6@SystematicSW.ab.ca>
X-TOI-EXPURGATEID: 150726::1744195284-217E9509-D4D983B1/0/0 CLEAN NORMAL
X-TOI-MSGID: 38a488c1-8273-4067-94a6-0deb2a4f5f81
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Christian Franke via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Christian Franke <Christian DOT Franke AT t-online DOT de>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 539AgmPZ2666789

Brian Inglis via Cygwin wrote:
> On 08/04/2025 19:05, Mark Liam Brown via Cygwin wrote:
>> Are there tuning variables to improve ls, ls -l, find ., find . -ls
>> performance for very large dirs?
>>
>> If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
>> minutes (22+mins), while cmd.exe dir just floods the terminal with
>> results immediately.
>
> I just remembered noticing my /var/log/ entries jumped looking at my 
> last Cygwin upgrade log, so before I clean it up into subdirs, let's 
> try some stuff!
>
> If you just want a directory name dump, use \ls -U|--sort=none or \ls 
> -f which is like -aU; command prefix \ overrides the usual aliases 
> like -CF --color=auto:
>
> $ time \ls -1U /var/log/ | \wc -lwmcL
>   54049   54049 1885787 1885787      38
>
> real    0m0.336s
> user    0m0.046s
> sys    0m0.295s
> $ time \ls -1f /var/log/ | \wc -lwmcL
>   54051   54051 1885792 1885792      38
>
> real    0m0.347s
> user    0m0.140s
> sys    0m0.232s
>
> even with -l, it's only a few seconds:
>
> $ time \ls -lU /var/log/ | \wc -lwmcL
>   54050  486455 4966716 4966716     102
>
> real    0m15.891s
> user    0m1.421s
> sys    0m14.560s
> $ time \ls -lf /var/log/ | \wc -lwmcL
>   54052  486473 4966835 4966835     102
>
> real    0m15.858s
> user    0m1.405s
> sys    0m14.374s
>

One reason for the above difference is possibly that Cygwin supports the 
'dirent.d_type' field for quite some time and tools like 'ls' and 'find' 
use it to avoid stat() calls.

Fast - stat() calls avoided:

   ls -R
   find
   find -type f -name 'some*file'

Fast if only few entries require stat():

   find -type f -name 'some*file' -mtime +42 # only -mtime requires stat()
   find -mtime +42 -type f -name 'some*file'  # find does 'cheap' checks 
first

Slow - requires stat() for each entry:

   ls -Rt
   ls -RS
   ls -Rl
   find -mtime +42
   find -ls

-- 
Regards,
Christian


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple