DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 53977fDE2576525
Authentication-Results: delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results: delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter: OpenDKIM Filter v2.11.0 delorie.com 53977fDE2576525
Authentication-Results: delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=KGIn2/6a
X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E1ED43864816
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1744182459;
	bh=VfBmqsfFyzPVCL9ndv7oiiTjCSecOfg74QRRVT3iocM=;
	h=Date:To:Subject:References:In-Reply-To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	 From;
	b=KGIn2/6aUBpNbKoObOld4AgJc9i/iHLor3t3YyVrdJMQ09Pgg6JLu53Bv1Miof5D8
	 PdS4AKTNZ28Mu295MVG15EBwxd0/vGW8iS7G6mUnNhT4MUr4Q4qngygc/9c0Rd5xh9
	 7jP2An/jfO6fVQMVU40yI2BtCaATSbwlZVH0BA9o=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 24C5A384DEF2
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 24C5A384DEF2
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1744182246; cv=none;
 b=q8ZQk+of+5KMLSz4g7bL7pFS3P0gE9U3HCHgexlXbJjJo4lLO1/kCkpPXUgiGCiAaoLsKJ14Pr/4x3vNUrcWha+IoJNUM3bgMul2rZK5Mjy4an5G5+Kvj753fh9hscTLAYmPuPp0zT7oSuptequEyI4cxm9Au0G57TK5XXM5PAs=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1744182246; c=relaxed/simple;
 bh=miba90yIYwm+MxohRTDRltBtvc/PdJ/lx78XLdDR6Fw=;
 h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version;
 b=e3rkM8e4FDwrA6VcbeaoE00bhGQ/tQYNSnUupfjK0/fYInLs4UgRv66HwqnuIWnbcDM1sxkemqajTLgLU53jLHsH+VEvEScfP+VltodhETCRwh5Gezp/zHOddWphjyJJDp9HKWXxHzLZFvCXSAALRuGvNHqelu/8uGrBrqnfgbk=
ARC-Authentication-Results: i=1; server2.sourceware.org
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 24C5A384DEF2
Date: Wed, 9 Apr 2025 03:04:02 -0400
To: cygwin AT cygwin DOT com
Subject: Re: Tuning ls, ls -l, find ., find . -ls performance for very large
 dirs (60000+ files/dir)
Message-ID: <Z_Yb4kX6oUKbMUTX@xps13>
References: <CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A AT mail DOT gmail DOT com>
 <4efd217b-eb94-42d8-bcc7-0f19de188ac0 AT gmx DOT com>
 <6e80b179-02c0-4972-b4fe-7fe0c2fef8a6 AT SystematicSW DOT ab DOT ca>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <6e80b179-02c0-4972-b4fe-7fe0c2fef8a6@SystematicSW.ab.ca>
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Glenn Strauss via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Glenn Strauss <gs-cygwin DOT com AT gluelogic DOT com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>

On Tue, Apr 08, 2025 at 10:36:20PM -0600, Brian Inglis via Cygwin wrote:
> On 08/04/2025 19:05, Mark Liam Brown via Cygwin wrote:
> > Are there tuning variables to improve ls, ls -l, find ., find . -ls
> > performance for very large dirs?
> > 
> > If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
> > minutes (22+mins), while cmd.exe dir just floods the terminal with
> > results immediately.
> 
> I just remembered noticing my /var/log/ entries jumped looking at my last
> Cygwin upgrade log, so before I clean it up into subdirs, let's try some
> stuff!
> 
> If you just want a directory name dump, use \ls -U|--sort=none or \ls -f
> which is like -aU; command prefix \ overrides the usual aliases like -CF
> --color=auto:
> 
> $ time \ls -1U /var/log/ | \wc -lwmcL
>   54049   54049 1885787 1885787      38
> 
> real	0m0.336s
> user	0m0.046s
> sys	0m0.295s
> $ time \ls -1f /var/log/ | \wc -lwmcL
>   54051   54051 1885792 1885792      38
> 
> real	0m0.347s
> user	0m0.140s
> sys	0m0.232s
> 
> even with -l, it's only a few seconds:
> 
> $ time \ls -lU /var/log/ | \wc -lwmcL
>   54050  486455 4966716 4966716     102
> 
> real	0m15.891s
> user	0m1.421s
> sys	0m14.560s
> $ time \ls -lf /var/log/ | \wc -lwmcL
>   54052  486473 4966835 4966835     102
> 
> real	0m15.858s
> user	0m1.405s
> sys	0m14.374s
> 
> and skipping user and group lookup does not save anything:
> 
> $ time \ls -gloU /var/log/ | \wc -lwmcL
>   54050  378357 3777638 3777638      80
> 
> real	0m15.699s
> user	0m1.562s
> sys	0m14.155s
> $ time \ls -gGlf /var/log/ | \wc -lwmcL
>   54052  378371 3777713 3777713      80
> 
> real	0m15.908s
> user	0m1.389s
> sys	0m14.498s
> 
> -- 
> Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

That reminds me:

OP should run cygserver to cache the AD LDAP queries.
That will likely have a measurable impact on performance.

https://cygwin.com/cygwin-ug-net/using-cygserver.html

Cheers, Glenn

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple