DMARC-Filter:	OpenDMARC Filter v1.4.2 delorie.com 5394baSu2516507
Authentication-Results:	delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results:	delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter:	OpenDKIM Filter v2.11.0 delorie.com 5394baSu2516507
Authentication-Results:	delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=eZjhoB3p
X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 9C9D13850856
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1744173454;
	bh=iXuMk1iApD0VMQXwmxf2s1KCAR5yXfx2qjMUeD9oqeU=;
	h=Date:Subject:To:References:In-Reply-To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:
	From;
	b=eZjhoB3pALNzLzqLgpLy8o5IYFRbx60odRRvBaUdS9I0WVDCY6ete+kNzvjHWNKb+
	gLJmKDtGUr6tuEBY+8aLb10UJuBQtb1lOdARxUHigcBfiqieK+ijon331CQdEQdjQ+
	dLpRKhn9ZKx0PBZSdcTW9w4bLee1jN+Yzhw8bzYU=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.2 sourceware.org 32A2D385117F
ARC-Filter:	OpenARC Filter v1.0.0 sourceware.org 32A2D385117F
ARC-Seal:	i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1744173383; cv=none;
	b=xDIruuIkaY0FaOq6qRRuBtadod5tVNr6DsTXOtCyu7sSmuETmRHyG2tTPPvzHiP3RhFc6Bjs8LiDc6okxk+F4D0Nixc2jSvv6OwKLkkSMGs/Nj16OQbqJlGm2eWwhAko8F7oWZYEWDQNQoszFWeuMrWsU9e2veJczHmgsLybtiM=
ARC-Message-Signature:	i=1; a=rsa-sha256; d=sourceware.org; s=key;
	t=1744173383; c=relaxed/simple;
	bh=iKMd2gK27/+1KHHK+bC1pZP1k9vBC83NyT2/kc8bj/s=;
	h=Message-ID:Date:MIME-Version:From:Subject:To:DKIM-Signature;
	b=uabsoPiTwrfk3JWytvUx99v38ZwvnrUUHQPLLyxU46Vf4EU+C3WjL/pb9Oap1IAi1jgW7UahrkfUQXoAEf6PrlDN/q3wuUK/NXgYTZhjC4uXV4lZaULCzXNgR1ESSIV8F2/NZPcwss+HExwsYIV8wPJj1Ksvo+I8vPKIGYrECns=
ARC-Authentication-Results:	i=1; server2.sourceware.org
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 32A2D385117F
Message-ID:	<6e80b179-02c0-4972-b4fe-7fe0c2fef8a6@SystematicSW.ab.ca>
Date:	Tue, 8 Apr 2025 22:36:20 -0600
MIME-Version:	1.0
User-Agent:	Mozilla Thunderbird
Subject:	Re: Tuning ls, ls -l, find ., find . -ls performance for very large
	dirs (60000+ files/dir)
To:	cygwin AT cygwin DOT com
References:	<CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A AT mail DOT gmail DOT com>
	<4efd217b-eb94-42d8-bcc7-0f19de188ac0 AT gmx DOT com>
Organization:	Systematic Software
In-Reply-To:	<4efd217b-eb94-42d8-bcc7-0f19de188ac0@gmx.com>
X-Rspamd-Queue-Id:	5854E2000D
X-Stat-Signature:	oxs4dpj4syztqkmfjqasnchgrm8yoaii
X-Rspamd-Server:	rspamout07
X-Session-Marker:	427269616E2E496E676C69734053797374656D6174696353572E61622E6361
X-Session-ID:	U2FsdGVkX1+xloWBRkNXESyjDBz0QOxnYfSHDO+B1Wg=
X-HE-Tag:	1744173381-402570
X-HE-Meta:	U2FsdGVkX1/fcofaJDB1HuDpZDctcmn31VxTCoB/0iziTNwY/nA3XrmdbVE3xr8mTrq+34JWVRuTtYNAT2dAHTotI1BaC1TwYl7tb0TMKTQ5zT4QykDwYJNBWFNKj6PK6DQ8J754xf60or8ZOWyxqdFMnlaQ97nqKhE1jJSlU0UG3X4NHF+EycNtdGb6HIka8IpG8hcgGvS6u/nFVICJQYb5uyAxx1sjBinOIPYHCu6g2Yj8MRkhNUyHGveTJoY6s3RvdqZ4T0hScKodiZfqx/2ed9ZWDju+5vs/oS6GWyNmCyFHyGtbKumQaQsaO79YIvFG5ETUsLZJp0BhavA9HFKTuY8zA0Ay
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.30
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe:	<https://cygwin.com/mailman/options/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	Brian Inglis via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	cygwin AT cygwin DOT com
Cc:	Brian Inglis <Brian DOT Inglis AT SystematicSW DOT ab DOT ca>
Errors-To:	cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender:	"Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
X-MIME-Autoconverted:	from base64 to 8bit by delorie.com id 5394baSu2516507

On 08/04/2025 19:05, Mark Liam Brown via Cygwin wrote:
> Are there tuning variables to improve ls, ls -l, find ., find . -ls
> performance for very large dirs?
>
> If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
> minutes (22+mins), while cmd.exe dir just floods the terminal with
> results immediately.

I just remembered noticing my /var/log/ entries jumped looking at my last Cygwin 
upgrade log, so before I clean it up into subdirs, let's try some stuff!

If you just want a directory name dump, use \ls -U|--sort=none or \ls -f which 
is like -aU; command prefix \ overrides the usual aliases like -CF --color=auto:

$ time \ls -1U /var/log/ | \wc -lwmcL
   54049   54049 1885787 1885787      38

real	0m0.336s
user	0m0.046s
sys	0m0.295s
$ time \ls -1f /var/log/ | \wc -lwmcL
   54051   54051 1885792 1885792      38

real	0m0.347s
user	0m0.140s
sys	0m0.232s

even with -l, it's only a few seconds:

$ time \ls -lU /var/log/ | \wc -lwmcL
   54050  486455 4966716 4966716     102

real	0m15.891s
user	0m1.421s
sys	0m14.560s
$ time \ls -lf /var/log/ | \wc -lwmcL
   54052  486473 4966835 4966835     102

real	0m15.858s
user	0m1.405s
sys	0m14.374s

and skipping user and group lookup does not save anything:

$ time \ls -gloU /var/log/ | \wc -lwmcL
   54050  378357 3777638 3777638      80

real	0m15.699s
user	0m1.562s
sys	0m14.155s
$ time \ls -gGlf /var/log/ | \wc -lwmcL
   54052  378371 3777713 3777713      80

real	0m15.908s
user	0m1.389s
sys	0m14.498s

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retrancher  but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -