DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 538Ncl6x2349266
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 235A1385EC3E
Date: Tue, 8 Apr 2025 19:38:20 -0400
To: Mark Liam Brown <brownmarkliam@gmail.com>
Cc: cygwin@cygwin.com
Subject: Re: Tuning ls, ls -l, find ., find . -ls performance for very large
 dirs (60000+ files/dir)
Message-ID: <Z_WzbNaNgcxiLcE9@xps13>
References: <CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A@mail.gmail.com>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A@mail.gmail.com>
Precedence: list
From: Glenn Strauss via Cygwin <cygwin@cygwin.com>
Reply-To: Glenn Strauss <gs-cygwin.com@gluelogic.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: cygwin-bounces~archive-cygwin=delorie.com@cygwin.com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie.com@cygwin.com>

On Tue, Apr 08, 2025 at 08:05:21PM +0200, Mark Liam Brown via Cygwin wrote:
> Greetings!
> 
> Are there tuning variables to improve ls, ls -l, find ., find . -ls
> performance for very large dirs?
> 
> If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
> minutes (22+mins), while cmd.exe dir just floods the terminal with
> results immediately.

`man ls`

This might help a little bit:

`ls -1f`  # one (-1) for single line output, and -f to disable sorting


Of course, please also ask why there are 60000 files in one directory
rather than sharded or stored some other way.  e.g. for log files,
create an archive/ subfolder and rotate all non-recently,
less-frequently accessed files to that subfolder so that the main
folder has (many) fewer entries.

For a folder that size, `ls` and `ls -l` are less appropriate tools.

A web server could serve a single index.html with the directory listing,
and the directory listing could be updated when the folder changes, or
on a periodic basis.

Various solutions depend on context and usage of the folder and its
contents.

Whether or not cmd.exe handles such a large directory better is
immaterial to the assessment that 60000+ entries in a single folder
is lacking better folder organization/file management and access methods

Cheers, Glenn

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple