X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2FE183857400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1709903746;
	bh=Ozh/AfL30XMMb9CIfwp5QaiPgjEB/UMGDf9vy73BtKM=;
	h=Date:Subject:To:References:In-Reply-To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	 From;
	b=FCC4rhKLqNZhwtr5lsZ+i8QizNDgYTTFY2JXIRbmiMHQ8VgF4il+ew9tmS628Bg/I
	 t14smztdDWUR7/q2uEKcpMaOSLpw/pf555xFIb2sjx3ZVBapY/SNadi6pYi89PttH4
	 3kJbLFv6QAPsZlfOF2V5cD4+c4JmcV3o3GEozyJo=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org AC1D13858D33
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org AC1D13858D33
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1709903724; cv=none;
 b=dwb5TQ1lbEPuLufB8AAD/6wkFUGdY/5jO1s1LTQ+BfBbrR/jBllzAO1YHLbQ16PDDCRRmsT93TbE2u4nDAyY72MOKDUm0E49JJZaQYxLD2sYT6CDy3+Y/kUbQpVSpNE/h1XksoKFZDs3sFEqw9uQ6V3lBQ5HfQk13PYSFm+we7Y=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1709903724; c=relaxed/simple;
 bh=op0c5qHNq5drd1rkPoM1F3akuKVlM4IUZHYmi2hHxK4=;
 h=DKIM-Signature:Message-ID:Date:MIME-Version:Subject:To:From;
 b=KXHbnrRKuZRTzzaYalNw1YLbco0g1QaNaDy5OFUTkNdzqog9pOj86IHxPjDccNPjjdsRlbTbJyQoMhfcedZZhg+s7aLv7oCTALjdPRJUrhZVde3GV5an175C9TVT1PHUtLLBNBI3FozQYxq5bfCTgaqSrUSXXg2JWOhJFToc640=
ARC-Authentication-Results: i=1; server2.sourceware.org
X-MDAV-Result: clean
X-MDAV-Processed: sequoia-grove.ad.secure-endpoints.com, Fri,
 08 Mar 2024 08:15:21 -0500
X-Spam-Processed: sequoia-grove.ad.secure-endpoints.com, Fri,
 08 Mar 2024 08:15:21 -0500
 (not processed: message from trusted or authenticated source)
X-MDRemoteIP: 2603:7000:73c:bb00:2477:ddf9:9ccb:a8de
X-MDHelo: [IPV6:2603:7000:73c:bb00:2477:ddf9:9ccb:a8de]
X-MDArrival-Date: Fri, 08 Mar 2024 08:15:21 -0500
X-MDOrigin-Country: US, NA
X-Authenticated-Sender: acct-jaltman AT secure-endpoints DOT com
X-Return-Path: prvs=1797105c20=jaltman AT secure-endpoints DOT com
X-Envelope-From: jaltman AT secure-endpoints DOT com
X-MDaemon-Deliver-To: cygwin AT cygwin DOT com
Message-ID: <64b65385-cad3-4807-b6c3-e463507da164@secure-endpoints.com>
Date: Fri, 8 Mar 2024 08:15:16 -0500
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: ls/stat on OneDrive causes download of files
Content-Language: en-US
To: Thomas Wolff <towo AT towo DOT net>, cygwin AT cygwin DOT com
References: <CAC9GOO-RbcR5YOPyk6qVDuVSdDK4M=f1V8bHvz7yvoaHjGUZnQ AT mail DOT gmail DOT com>
 <ZehuDAKd3ceXlfba AT calimero DOT vinschen DOT de>
 <ZehvbhVizZTBSaRR AT calimero DOT vinschen DOT de>
 <208962d1-78cd-4e2d-9e4a-a1edcbf5e809 AT SystematicSW DOT ab DOT ca>
 <ZeilkJK7Csryuzkc AT calimero DOT vinschen DOT de>
 <7d9fe460-5704-424b-a89b-e34ef2176d38 AT secure-endpoints DOT com>
 <ZerqWzAr3b_mpork AT calimero DOT vinschen DOT de>
 <a5ed70b0-8ac4-453f-b5cf-ce8f77f6bdf7 AT towo DOT net>
Organization: Secure Endpoints, Inc.
Autocrypt: addr=jaltman AT secure-endpoints DOT com; keydata=
 xsFNBEwLlO0BEACu6yWFkd1+qwsGg8ZzgslSkcAKhSegWt5j86DpaRL0W8fxg6YjxwEPvwoH
 BGa/rpSdBd1gkmzeYxD3hVZdj75r6nVS9f/mxNQzW+o1sW4vaeSxKgZSQz5RqHmwPDcqQP66
 +ZSnjV+G88MKwZ9DIzA9AwpJhNAAlAlj3OvsQVsxd1ipc6C4/U3qjHL7Ih22UbPBM71ltIZx
 kqcrAlXPnUTeraJXtfzYbq4mJFJ9JC6/o1NRSjsBvRD+ADxlG50+KccZN4SS5xxdGuh1tA9U
 TydYBQB3YtJbq7CYau2kIYt/3HnyLYGo1s6Ti6cuAJJ/40iIE1xkqhvMiIz/Q+1ztmksJbLQ
 aCtW8kF42nF8MpPdIPTSPr2uGvpRtCjRbh4lgMXgyNUx1wpCEY0X11xce++H8HySmFwryE2y
 kkxUQeMUjaaXZDHYUSyQz7riChFiZ9ax9dmX0wUY/A05v0qcualglpk4wJ2kcsGKUEGkLvnV
 wwvya8zifPwKOw5JlGPvzX8t2m7jB2GXKzvVAsImqOqnDBTKUXWQQZCW9Rqt7acdE8bQ2vqr
 vP+3Ykf4SrPwcuNCDt6QSgjVbhc3hA3hCtE1iW/HhuBAzKiuzJ9era+q9QjTtLPIkQDHRpcC
 MMWvK0Y1uQ34Ql1BfKRA4gc8A7CuVUY6+Ga7PuJWd+FSglvmKQARAQABzS1KZWZmcmV5IEFs
 dG1hbiA8amFsdG1hbkBzZWN1cmUtZW5kcG9pbnRzLmNvbT7CwXYEEwECACAFAkwLltMCGyMG
 CwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRD3enNVkraaBIO5D/sHkpJx2Vz7mgUDthBuro7f
 y/FK9MxC6dUTF0H54OjRXPBRQmzDDzpTNmG+8mtPO+FR1QlkupQJpGgFjpUdVnjlq2nGF8sH
 ecCCIKylI9VM3+bqZOEiRWHYxmmfMNDLJTBY5ES8YFsiOvWiCmKJe5YicwysBEcjD75O5JsX
 2ylRJ/NRF3VLI2A/jE1HI7DDgZ8hDkYU2yhV37wUHZPQUS5XYiSaUtn71MI2kIS8rHRPrxHg
 qmRB4uKJDVVW1jdo0/43YAJQr8CitlstluZMmXYOjGZdiG9MDdtWJz7KxEQsQD9XLPvb/aD5
 sGCoBbu26ODnSkz0k2UM5uWknul4zbgt6s2V/HD+uwxmbhJEpzMeLMbWNVDXQH+EcmOnfVYV
 ZalkMA19zkGt0jEbNq3CTx2zSsg1Gq+j6UzRm+kANo+VN3y2izRVtVreQrw1MX6DhvwQsqdI
 gVVINvTaa+2rZhHaDXRsnb3jYaqsvLo7gOUGdwGp3jin4YNHAR8JZ9e+ueUMDEFiYdCHMPJf
 y2HgVh4vNTtO9P6BDIG/DdpOPSlOgKKulbE0BGKMWCkqwAr42SDF0lk5uCvOkEpfDHPLgDbK
 rBehbmhCidGiZsR5ijdhh65CgKFUrBY50/aBeSSjMUCo2kC10oV+N5SdjbOH/M5TC8Yt1oUf
 ATH/Dt7/FZ/GHM7BTQRMC5TtARAA88hJdpgcg2RU/uAWfAL46XZHA59cVpPNNly1tPWCSbG6
 +ONH6nOG/NarmNVxX6Mb9YRkEU6wmrZS85inz3otdyz/zlyNSWma8qGNUlMbiwFQqfXWVBAP
 GoRC0a0aJrd4IayLuvv1UqEwx7Otp7y5RNHtRv35/kho0Z+UheYVdGm2I06xIc+aNKW2LO7R
 5BNtjpADPIG+NSdsVIeamhAWPvLrwbf6mUb//eA9pF0w0QixLVrH/cCoz+S27gCGJvY6zF22
 NgdhnkIqNz8E/LKt6S36ZI9Mw/ixpQTozqRmdNzVQNgTHUZClbJj4iq1EPHB7XqpxOv+awrx
 Sxq2jt8GFD0rU+sAuzW+F7cBoIw434/IrxKYwcPHpHLEVQ1tLP7d3ZpZR30p3oqoliGiLsWv
 HHxyXjuMBF4XJ6MRXmD65/qOhuo2DKduHMNlmxzgSzvWgXZeNJq+OcS8jQZDt2Na2pMKjWyt
 au7xQu2ndm0FwS48ngMrDYRQMxzL1NfnBnT9BCwjiU+/6NBSwcNKIqyea9IpTwsVfkF4/iui
 7xD9+LtzqeUkBAe7q5jEJmJMZhAfh7usZGT8TGxXegCaF4Jwz2nxS4Fv7VRza/yUAOJlc0da
 R11TPeiUNCQWY7PpL1AXO9vaSyjFuOzTnU8vzXvI9fGoxIxKGRQpKMU8PROIFw0AEQEAAcLB
 XwQYAQIACQUCTAuU7QIbDAAKCRD3enNVkraaBIxXD/4xlaBwW2TLFfMvlcY/2XDSm6NO4JaJ
 G2Nzp35xaaBVwMVzWvI+GgTgKNSFot9f4jiLBNQdnq3UKoEThR2ORKVL0ZJS1QYR7yyrOo0M
 teDSy8ofU1FJ6xu4ND3ekOjP20BTrihDpqUdahir2uaRfMkwM+0imOlcutGMhJNF/LAjrhoD
 p9SeDMYBXZ1wfrbrEo/EEu0PbkGyzqPyEPqwN1iSJkcAnjuIA0rTf1jQtJAaDov7yHsSRwUM
 +qTGsjOGQAN3wtYwjPpw7hI01sE+x0uq0pVeo4qeWTZ2TE4Vtp8FKXFAkqnP878q+kNk9Ve+
 DRs8UlRfa9Lgf5ETjXOTVGaT/UGxi9B4oo8k0lzvM/A1txexL/lLw8AULhUeGtyS6D2X9vFi
 6azna+o918R9BV86uXPiDOf1nMwqKchNCxmgH9vd0aQm8TKCrWAW4kU1Ig6aMNuZiWloVZfK
 rmWizbgeGKE9rhNPNqxkqBaA4lrJ8L6bdKbhAOe3NQjO2vUAXB53JphlF74GwEsh+85i9/yI
 bvwJVcsFYhdZz7fCAUOcnFkGnyrwIgkizQ3xXShPW8mqkgUk4kYMnucC4kG/E7pI/4lke5X5
 X9vroXRHB7tkpAgT46SqSM/XTwCaseXG9orDgz3duRTUp6K0++S/qsqTakGVmjD5917A1HqW
 fMmiKA==
In-Reply-To: <a5ed70b0-8ac4-453f-b5cf-ce8f77f6bdf7@towo.net>
X-MDCFSigsAdded: secure-endpoints.com
X-Spam-Status: No, score=-0.6 required=5.0 tests=BAYES_00, BODY_8BITS,
 DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE,
 SPF_HELO_PASS, SPF_PASS, TXREP,
 T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Jeffrey Altman via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Jeffrey Altman <jaltman AT secure-endpoints DOT com>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 428DFljW351103

On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote:
>
> Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin:
>> Hi Jeffrey,
>>
>> On Mar  6 13:55, Jeffrey Altman via Cygwin wrote:
>>> On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote:
>>>> We can add an explicit call to
>>>>
>>>>     RtlSetProcessPlaceholderCompatibilityMode 
>>>> (PHCM_EXPOSE_PLACEHOLDERS);
>>>> [...]
>>> Files and directories that are placeholders should have either the
>>> FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or 
>>> FILE_ATTRIBUTE_RECALL_ON_OPEN file
>>> attributes set. When these attributes are set, applications and mini 
>>> filters
>>> are advised not to "read" or "open" the files or directories unless 
>>> they
>>> absolutely need to because doing so will cause the placeholder to be
>>> replaced by an object containing the actual data which might take a 
>>> long
>>> time to fetch,
>> Yesterday I stumbled over a certain NtCreateFile flag:
>>
>>    FILE_OPEN_NO_RECALL (0x00400000)
>>
>>      Instructs any filters that perform offline storage or 
>> virtualization
>>      to not recall the contents of the file as a result of this open.
>>
>> MS-CIFS described it like this:
>>
>>    FILE_OPEN_NO_RECALL
>>    0x00400000
>>
>>      In a hierarchical storage management environment, this option
>>      requests that the file SHOULD NOT be recalled from tertiary storage
>>      such as tape. A file recall can take up to several minutes in a
>>      hierarchical storage management environment. The clients can 
>> specify
>>      this option to avoid such delays.
>>
>> This sounds like we could simply add this flag to all NtOpenFile
>> used for path conversion or stat-like calls, without having to care
>> for any file attributes specificially.
>>
>> Does that make sense?
> Sounds good, without even studying the other details...
> I speculate some more handling would still be needed to avoid executable
> detection via magic tags.
>
Agreed.   FILE_OPEN_NO_RECALL has been defined for at least a decade but 
was not documented by Microsoft relatively recently.

Another suggestion would be to try opening the file with 
FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not 
required.  See

https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493

Jeffrey Altman



-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple