X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 460993858289
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1673968949;
	bh=Cz6XvwUBLY5jO3Fq2ShlDUUsIxldKtcxUoX1Intj7yM=;
	h=Subject:To:References:Date:In-Reply-To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	 From;
	b=shmssAXSG+5H6zgCm9WU9X0NRfrXxVmv5/LFK+wIt/v247dMmHAD3X0K95hz2dvlo
	 ddf4P+PZco960eOOuSA0gt2Mzt0C3y4U9jbMbuQBx8KLO3Sj13fo1zB3Ps240Htx7j
	 Y9effo51ZG0XDz/LxBtDTwAmhBkcT9uf5Gh40t08=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6D4D93858D28
Subject: Re: Question about slow access to file information
To: cygwin <cygwin AT cygwin DOT com>
References: <797a8935-e38b-0c0f-87d8-b8df1e9fd76f AT cs DOT umass DOT edu>
 <0c9c111e-9e63-bf8c-8049-06fd23f66351 AT t-online DOT de>
 <cbc3fce3-64a5-47af-0cf1-5ca55daf176a AT cs DOT umass DOT edu>
Message-ID: <08d1d7ae-32b7-45b3-595a-ae92eff9f8e1@t-online.de>
Date: Tue, 17 Jan 2023 16:21:51 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101
 SeaMonkey/2.53.14
MIME-Version: 1.0
In-Reply-To: <cbc3fce3-64a5-47af-0cf1-5ca55daf176a@cs.umass.edu>
X-TOI-EXPURGATEID: 150726::1673968910-736BB7D8-D99EF2BD/0/0 CLEAN NORMAL
X-TOI-MSGID: 099f1691-f176-4445-91c5-67537fa662ae
X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00, FREEMAIL_FROM,
 KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_DNSWL_LOW,
 RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE,
 TXREP autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Christian Franke via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Christian Franke <Christian DOT Franke AT t-online DOT de>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 30HFMsP1025318

Eliot Moss via Cygwin wrote:
> On 1/15/2023 3:38 AM, Christian Franke via Cygwin wrote:
>> Eliot Moss via Cygwin wrote:
>>> I have a separate drive mounted this way:
>>>
>>> d:/ /cygdrive/d ntfs binary,posix=0,user,noacl,auto 0 0
>>>
>>> One thing I use it for is to store backup files.  These tend to be 2 Gb
>>> chunks, and there can be hundreds of them in the backup directory. 
>>> (The drive
>>> is 5Tb.)  The Windows Disk Management tool describes it as NTFS, 
>>> Basic Data
>>> Partition.
>>>
>>> Doing ls (for example) takes a very perceptible numbers of seconds 
>>> (though
>>> whatever takes a long time seems to be cached, at least for a while, 
>>> since a
>>> second ls soon after is fast).
>>
>> The problem is the 'noacl' mount option and the fact that POSIX only 
>> offers the *stat*() functions to retrieve file information. These 
>> functions always need to provide the full file information, even if 
>> only a small subset is needed.
>>
>> To determine the 'x'-permission bits in the 'stat.st_mode' field on a 
>> 'noacl'-mount, Cygwin reads the first bytes of most files (all except 
>> *.exe, *.lnk, *.com). The 'x' bits are set if the file starts with 
>> "#!" (script), ":\n" (?) or "MZ" (Windows executable).
>>
>> On 'noacl' mounts, this behavior could be suppressed by 'exec' or 
>> 'noexec' mount options.
>
> Interesting.  I removed the noacl from /etc/fstab and restarted all 
> Cygwin processes.
> The mount program now shows that drive without noacl.  It still takes 
> surprisingly
> long to ls if I have not done so recently.  The directory contains 
> ~1200 files.

This depends on storage device, sometimes (HDD) on filesystem 
fragmentation and always on 'ls' options. Plain '/bin/ls' without any 
arguments does not call stat(). 'ls -s' or 'ls --color=yes' call stat() 
for each file. 'ls -l' additionally calls getfacl() for each file if on 
an 'acl' mount. The latter is apparently slower than expected, see below.

Here a quick test on a directory with 10000 ~3KB files on a NTFS USB 
drive connected via USB-2 (~28MB/s raw read speed). The first test of 
each mount variant was done immediately after connecting the drive:

$ TIMEFORMAT='%R'

1. mount [-o acl]

$ time ls -l > /dev/null
4.282
$ time ls -l > /dev/null
1.322
$ time ls -s > /dev/null
0.404
$ time ls > /dev/null
0.032


2. mount -o noacl

$ time ls -l > /dev/null
13.452
$ time ls -l > /dev/null
0.789
$ time ls -s > /dev/null
0.764
$ time ls > /dev/null
0.033


3. mount -o noacl,noexec

$ time ls -l > /dev/null
3.215
$ time ls -l > /dev/null
0.368
$ time ls -s > /dev/null
0.355
$ time ls > /dev/null
0.032

-- 
Regards,
Christian


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple