DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B3DB23858D34
Message-ID: <7d9fe460-5704-424b-a89b-e34ef2176d38@secure-endpoints.com>
Date: Wed, 6 Mar 2024 13:55:17 -0500
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: ls/stat on OneDrive causes download of files
Content-Language: en-US
To: cygwin@cygwin.com
References: <CAC9GOO-RbcR5YOPyk6qVDuVSdDK4M=f1V8bHvz7yvoaHjGUZnQ@mail.gmail.com>
 <ZehuDAKd3ceXlfba@calimero.vinschen.de>
 <ZehvbhVizZTBSaRR@calimero.vinschen.de>
 <208962d1-78cd-4e2d-9e4a-a1edcbf5e809@SystematicSW.ab.ca>
 <ZeilkJK7Csryuzkc@calimero.vinschen.de>
Organization: Secure Endpoints, Inc.
Autocrypt: addr=jaltman@secure-endpoints.com; keydata=
 xsFNBEwLlO0BEACu6yWFkd1+qwsGg8ZzgslSkcAKhSegWt5j86DpaRL0W8fxg6YjxwEPvwoH
 BGa/rpSdBd1gkmzeYxD3hVZdj75r6nVS9f/mxNQzW+o1sW4vaeSxKgZSQz5RqHmwPDcqQP66
 +ZSnjV+G88MKwZ9DIzA9AwpJhNAAlAlj3OvsQVsxd1ipc6C4/U3qjHL7Ih22UbPBM71ltIZx
 kqcrAlXPnUTeraJXtfzYbq4mJFJ9JC6/o1NRSjsBvRD+ADxlG50+KccZN4SS5xxdGuh1tA9U
 TydYBQB3YtJbq7CYau2kIYt/3HnyLYGo1s6Ti6cuAJJ/40iIE1xkqhvMiIz/Q+1ztmksJbLQ
 aCtW8kF42nF8MpPdIPTSPr2uGvpRtCjRbh4lgMXgyNUx1wpCEY0X11xce++H8HySmFwryE2y
 kkxUQeMUjaaXZDHYUSyQz7riChFiZ9ax9dmX0wUY/A05v0qcualglpk4wJ2kcsGKUEGkLvnV
 wwvya8zifPwKOw5JlGPvzX8t2m7jB2GXKzvVAsImqOqnDBTKUXWQQZCW9Rqt7acdE8bQ2vqr
 vP+3Ykf4SrPwcuNCDt6QSgjVbhc3hA3hCtE1iW/HhuBAzKiuzJ9era+q9QjTtLPIkQDHRpcC
 MMWvK0Y1uQ34Ql1BfKRA4gc8A7CuVUY6+Ga7PuJWd+FSglvmKQARAQABzS1KZWZmcmV5IEFs
 dG1hbiA8amFsdG1hbkBzZWN1cmUtZW5kcG9pbnRzLmNvbT7CwXYEEwECACAFAkwLltMCGyMG
 CwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRD3enNVkraaBIO5D/sHkpJx2Vz7mgUDthBuro7f
 y/FK9MxC6dUTF0H54OjRXPBRQmzDDzpTNmG+8mtPO+FR1QlkupQJpGgFjpUdVnjlq2nGF8sH
 ecCCIKylI9VM3+bqZOEiRWHYxmmfMNDLJTBY5ES8YFsiOvWiCmKJe5YicwysBEcjD75O5JsX
 2ylRJ/NRF3VLI2A/jE1HI7DDgZ8hDkYU2yhV37wUHZPQUS5XYiSaUtn71MI2kIS8rHRPrxHg
 qmRB4uKJDVVW1jdo0/43YAJQr8CitlstluZMmXYOjGZdiG9MDdtWJz7KxEQsQD9XLPvb/aD5
 sGCoBbu26ODnSkz0k2UM5uWknul4zbgt6s2V/HD+uwxmbhJEpzMeLMbWNVDXQH+EcmOnfVYV
 ZalkMA19zkGt0jEbNq3CTx2zSsg1Gq+j6UzRm+kANo+VN3y2izRVtVreQrw1MX6DhvwQsqdI
 gVVINvTaa+2rZhHaDXRsnb3jYaqsvLo7gOUGdwGp3jin4YNHAR8JZ9e+ueUMDEFiYdCHMPJf
 y2HgVh4vNTtO9P6BDIG/DdpOPSlOgKKulbE0BGKMWCkqwAr42SDF0lk5uCvOkEpfDHPLgDbK
 rBehbmhCidGiZsR5ijdhh65CgKFUrBY50/aBeSSjMUCo2kC10oV+N5SdjbOH/M5TC8Yt1oUf
 ATH/Dt7/FZ/GHM7BTQRMC5TtARAA88hJdpgcg2RU/uAWfAL46XZHA59cVpPNNly1tPWCSbG6
 +ONH6nOG/NarmNVxX6Mb9YRkEU6wmrZS85inz3otdyz/zlyNSWma8qGNUlMbiwFQqfXWVBAP
 GoRC0a0aJrd4IayLuvv1UqEwx7Otp7y5RNHtRv35/kho0Z+UheYVdGm2I06xIc+aNKW2LO7R
 5BNtjpADPIG+NSdsVIeamhAWPvLrwbf6mUb//eA9pF0w0QixLVrH/cCoz+S27gCGJvY6zF22
 NgdhnkIqNz8E/LKt6S36ZI9Mw/ixpQTozqRmdNzVQNgTHUZClbJj4iq1EPHB7XqpxOv+awrx
 Sxq2jt8GFD0rU+sAuzW+F7cBoIw434/IrxKYwcPHpHLEVQ1tLP7d3ZpZR30p3oqoliGiLsWv
 HHxyXjuMBF4XJ6MRXmD65/qOhuo2DKduHMNlmxzgSzvWgXZeNJq+OcS8jQZDt2Na2pMKjWyt
 au7xQu2ndm0FwS48ngMrDYRQMxzL1NfnBnT9BCwjiU+/6NBSwcNKIqyea9IpTwsVfkF4/iui
 7xD9+LtzqeUkBAe7q5jEJmJMZhAfh7usZGT8TGxXegCaF4Jwz2nxS4Fv7VRza/yUAOJlc0da
 R11TPeiUNCQWY7PpL1AXO9vaSyjFuOzTnU8vzXvI9fGoxIxKGRQpKMU8PROIFw0AEQEAAcLB
 XwQYAQIACQUCTAuU7QIbDAAKCRD3enNVkraaBIxXD/4xlaBwW2TLFfMvlcY/2XDSm6NO4JaJ
 G2Nzp35xaaBVwMVzWvI+GgTgKNSFot9f4jiLBNQdnq3UKoEThR2ORKVL0ZJS1QYR7yyrOo0M
 teDSy8ofU1FJ6xu4ND3ekOjP20BTrihDpqUdahir2uaRfMkwM+0imOlcutGMhJNF/LAjrhoD
 p9SeDMYBXZ1wfrbrEo/EEu0PbkGyzqPyEPqwN1iSJkcAnjuIA0rTf1jQtJAaDov7yHsSRwUM
 +qTGsjOGQAN3wtYwjPpw7hI01sE+x0uq0pVeo4qeWTZ2TE4Vtp8FKXFAkqnP878q+kNk9Ve+
 DRs8UlRfa9Lgf5ETjXOTVGaT/UGxi9B4oo8k0lzvM/A1txexL/lLw8AULhUeGtyS6D2X9vFi
 6azna+o918R9BV86uXPiDOf1nMwqKchNCxmgH9vd0aQm8TKCrWAW4kU1Ig6aMNuZiWloVZfK
 rmWizbgeGKE9rhNPNqxkqBaA4lrJ8L6bdKbhAOe3NQjO2vUAXB53JphlF74GwEsh+85i9/yI
 bvwJVcsFYhdZz7fCAUOcnFkGnyrwIgkizQ3xXShPW8mqkgUk4kYMnucC4kG/E7pI/4lke5X5
 X9vroXRHB7tkpAgT46SqSM/XTwCaseXG9orDgz3duRTUp6K0++S/qsqTakGVmjD5917A1HqW
 fMmiKA==
In-Reply-To: <ZeilkJK7Csryuzkc@calimero.vinschen.de>
Precedence: list
From: Jeffrey Altman via Cygwin <cygwin@cygwin.com>
Reply-To: Jeffrey Altman <jaltman@secure-endpoints.com>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: cygwin-bounces+archive-cygwin=delorie.com@cygwin.com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie.com@cygwin.com>

On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote:
> We can add an explicit call to
>
>    RtlSetProcessPlaceholderCompatibilityMode (PHCM_EXPOSE_PLACEHOLDERS);
>
> and we can recognize the IO_REPARSE_TAG_FILE_PLACEHOLDER and
> IO_REPARSE_TAG_CLOUD_* tags during symlink evaluation, but even then
> we still have to know what the reparse point buffer actually contains.
>
> Given that the content of reparse points with these reparse tags are
> undocumented, some people using cloud services should examine these
> reparse points so we can add some suitable code to Cygwin.
>
>
> Corinna
I'm not an expert in this area by any means but here are my 
recollections from when Microsoft presented in-person on cloud 
placeholders to filter and filesystem developers many years ago.

Files and directories that are placeholders should have either the 
FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN 
file attributes set. When these attributes are set, applications and 
mini filters are advised not to "read" or "open" the files or 
directories unless they absolutely need to because doing so will cause 
the placeholder to be replaced by an object containing the actual data 
which might take a long time to fetch, might cost the end user money, or 
might fail depending upon the network connectivity. In particular, 
anti-malware should ignore them during scans and only analyze the data 
when it is fetched locally by an end user application.

I believe that IO_REPARSE_TAG_FILE_PLACEHOLDER was replaced by 
IO_REPARSE_TAG_CLOUD_1 ..IO_REPARSE_TAG_CLOUD_F. Any reparse tag 
attached to a placeholder object is for the interpretation of the filter 
associated with the back-end storage and not for the consumption of 
applications. The content of the reparse tags can be back-end 
proprietary; different reparse data for onedrive, icloud, dropbox, etc.

The default ProcessPlaceholderCompaibilityMode is 
PHCM_EXPOSE_PLACEHOLDERS which makes the FILE_ATTRIBUTE flags and 
reparse tags visible. Microsoft maintains a database of processes for 
which PHCM_DISGUISE_PLACEHOLDER is set which hides that information. Its 
unclear to me that explicitly setting the placeholder compatibility mode 
is useful.

I'm not sure that exposing the object as a symlink is a good idea. A 
posix symlink is an object whose type and target information cannot 
change. In the case of a placeholder, the placeholder is silently 
replaced by the actual object either when the object is opened or the 
object's data is accessed. An application that believes it knows that 
the object is a symlink will be mighty confused when it turns out to be 
a file or a directory.

Perhaps the question that needs to be asked is whether there are opens 
that can be skipped if an object is known to not be locally present 
(either of the FILE_ATTRIBUTE flags are set)?

Jeffrey Altman


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple