DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 5394baSu2516507
Authentication-Results: delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results: delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter: OpenDKIM Filter v2.11.0 delorie.com 5394baSu2516507
Authentication-Results: delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=eZjhoB3p
X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9C9D13850856
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1744173454;
	bh=iXuMk1iApD0VMQXwmxf2s1KCAR5yXfx2qjMUeD9oqeU=;
	h=Date:Subject:To:References:In-Reply-To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:
	 From;
	b=eZjhoB3pALNzLzqLgpLy8o5IYFRbx60odRRvBaUdS9I0WVDCY6ete+kNzvjHWNKb+
	 gLJmKDtGUr6tuEBY+8aLb10UJuBQtb1lOdARxUHigcBfiqieK+ijon331CQdEQdjQ+
	 dLpRKhn9ZKx0PBZSdcTW9w4bLee1jN+Yzhw8bzYU=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 32A2D385117F
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 32A2D385117F
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1744173383; cv=none;
 b=xDIruuIkaY0FaOq6qRRuBtadod5tVNr6DsTXOtCyu7sSmuETmRHyG2tTPPvzHiP3RhFc6Bjs8LiDc6okxk+F4D0Nixc2jSvv6OwKLkkSMGs/Nj16OQbqJlGm2eWwhAko8F7oWZYEWDQNQoszFWeuMrWsU9e2veJczHmgsLybtiM=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1744173383; c=relaxed/simple;
 bh=iKMd2gK27/+1KHHK+bC1pZP1k9vBC83NyT2/kc8bj/s=;
 h=Message-ID:Date:MIME-Version:From:Subject:To:DKIM-Signature;
 b=uabsoPiTwrfk3JWytvUx99v38ZwvnrUUHQPLLyxU46Vf4EU+C3WjL/pb9Oap1IAi1jgW7UahrkfUQXoAEf6PrlDN/q3wuUK/NXgYTZhjC4uXV4lZaULCzXNgR1ESSIV8F2/NZPcwss+HExwsYIV8wPJj1Ksvo+I8vPKIGYrECns=
ARC-Authentication-Results: i=1; server2.sourceware.org
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 32A2D385117F
Message-ID: <6e80b179-02c0-4972-b4fe-7fe0c2fef8a6@SystematicSW.ab.ca>
Date: Tue, 8 Apr 2025 22:36:20 -0600
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Tuning ls, ls -l, find ., find . -ls performance for very large
 dirs (60000+ files/dir)
Content-Language: en-CA
To: cygwin AT cygwin DOT com
References: <CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A AT mail DOT gmail DOT com>
 <4efd217b-eb94-42d8-bcc7-0f19de188ac0 AT gmx DOT com>
Organization: Systematic Software
In-Reply-To: <4efd217b-eb94-42d8-bcc7-0f19de188ac0@gmx.com>
X-Rspamd-Queue-Id: 5854E2000D
X-Stat-Signature: oxs4dpj4syztqkmfjqasnchgrm8yoaii
X-Rspamd-Server: rspamout07
X-Session-Marker: 427269616E2E496E676C69734053797374656D6174696353572E61622E6361
X-Session-ID: U2FsdGVkX1+xloWBRkNXESyjDBz0QOxnYfSHDO+B1Wg=
X-HE-Tag: 1744173381-402570
X-HE-Meta: U2FsdGVkX1/fcofaJDB1HuDpZDctcmn31VxTCoB/0iziTNwY/nA3XrmdbVE3xr8mTrq+34JWVRuTtYNAT2dAHTotI1BaC1TwYl7tb0TMKTQ5zT4QykDwYJNBWFNKj6PK6DQ8J754xf60or8ZOWyxqdFMnlaQ97nqKhE1jJSlU0UG3X4NHF+EycNtdGb6HIka8IpG8hcgGvS6u/nFVICJQYb5uyAxx1sjBinOIPYHCu6g2Yj8MRkhNUyHGveTJoY6s3RvdqZ4T0hScKodiZfqx/2ed9ZWDju+5vs/oS6GWyNmCyFHyGtbKumQaQsaO79YIvFG5ETUsLZJp0BhavA9HFKTuY8zA0Ay
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Brian Inglis via Cygwin <cygwin AT cygwin DOT com>
Reply-To: cygwin AT cygwin DOT com
Cc: Brian Inglis <Brian DOT Inglis AT SystematicSW DOT ab DOT ca>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 5394baSu2516507

On 08/04/2025 19:05, Mark Liam Brown via Cygwin wrote:
> Are there tuning variables to improve ls, ls -l, find ., find . -ls
> performance for very large dirs?
>
> If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
> minutes (22+mins), while cmd.exe dir just floods the terminal with
> results immediately.

I just remembered noticing my /var/log/ entries jumped looking at my last Cygwin 
upgrade log, so before I clean it up into subdirs, let's try some stuff!

If you just want a directory name dump, use \ls -U|--sort=none or \ls -f which 
is like -aU; command prefix \ overrides the usual aliases like -CF --color=auto:

$ time \ls -1U /var/log/ | \wc -lwmcL
   54049   54049 1885787 1885787      38

real	0m0.336s
user	0m0.046s
sys	0m0.295s
$ time \ls -1f /var/log/ | \wc -lwmcL
   54051   54051 1885792 1885792      38

real	0m0.347s
user	0m0.140s
sys	0m0.232s

even with -l, it's only a few seconds:

$ time \ls -lU /var/log/ | \wc -lwmcL
   54050  486455 4966716 4966716     102

real	0m15.891s
user	0m1.421s
sys	0m14.560s
$ time \ls -lf /var/log/ | \wc -lwmcL
   54052  486473 4966835 4966835     102

real	0m15.858s
user	0m1.405s
sys	0m14.374s

and skipping user and group lookup does not save anything:

$ time \ls -gloU /var/log/ | \wc -lwmcL
   54050  378357 3777638 3777638      80

real	0m15.699s
user	0m1.562s
sys	0m14.155s
$ time \ls -gGlf /var/log/ | \wc -lwmcL
   54052  378371 3777713 3777713      80

real	0m15.908s
user	0m1.389s
sys	0m14.498s

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retrancher  but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple