DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 50FHQiTA3968372
Date: Wed, 15 Jan 2025 18:26:14 +0100
To: cygwin@cygwin.com
Subject: Re: Cygwin 3.6 possible issue handling compressed .pdb files on SSD?
Message-ID: <Z4fvtuywpUttG-66@calimero.vinschen.de>
Mail-Followup-To: cygwin@cygwin.com
References: <CAKAoaQmwna7sSfpYFn0OUsqpdXAgJ5FyaNkhKv6K+otr0Oa5QA@mail.gmail.com>
 <4712dcf7-1d4d-4d27-b7c1-b705d3a0a553@SystematicSW.ab.ca>
 <CAKAoaQ=aj+xB2J65Rgv1tV4oSK9fq0tk8Bq6ZoXAoOG_hBR5eA@mail.gmail.com>
 <bf9d2afd-68e4-46b3-a9e6-37de7ffb8fb1@SystematicSW.ab.ca>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <bf9d2afd-68e4-46b3-a9e6-37de7ffb8fb1@SystematicSW.ab.ca>
Precedence: list
From: Corinna Vinschen via Cygwin <cygwin@cygwin.com>
Reply-To: cygwin@cygwin.com
Cc: Corinna Vinschen <corinna-cygwin@cygwin.com>
Content-Type: text/plain; charset="utf-8"
Errors-To: cygwin-bounces~archive-cygwin=delorie.com@cygwin.com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie.com@cygwin.com>
Content-Transfer-Encoding: 8bit

On Jan 15 09:12, Brian Inglis via Cygwin wrote:
> On 2025-01-14 03:13, Roland Mainz via Cygwin wrote:
> > On Tue, Jan 14, 2025 at 7:19 AM Brian Inglis via Cygwin
> > <cygwin@cygwin.com> wrote:
> > > On 2025-01-13 13:10, Roland Mainz via Cygwin wrote:
> > > > I just hit an endless loop with /usr/bin/cp from "coreutils" version
> > > > 9.5-1 copying a larger *.pdb file (it seems that only this specific
> > > > file is affected...) from Visual Studio 19.
> > > > [...]
> > I think I found the problem:
> > The *.pdb file uses NTFS compression:
> > ---- snip ----
> > /bin/winfsinfo filebasicinfo "$(cygpath -w
> > $PWD/../build.vc19/x64/Debug/nfs41_driver.pdb)"
> > (
> >         filename='C:\cygwin64\home\roland_mainz\work\msnfs41_uidmapping\ms-nfs41-client-kofemannvacation\build.vc19\x64\Debug\nfs41_driver.pdb'
> >         CreationTime=133812707624654816
> >         LastAccessTime=133813220892976366
> >         LastWriteTime=133812707639811081
> >         ChangeTime=133812707639811081
> >         typeset -a FileAttributes=(
> >                 FILE_ATTRIBUTE_ARCHIVE
> >                 FILE_ATTRIBUTE_COMPRESSED
> >         )
> > )
> > ---- snip ----
> > 
> > If I remove the "FILE_ATTRIBUTE_COMPRESSED" flag /bin/cp works without problems.
> > I think the issues here are:
> > 1. Coreutils 9.5-1 /bin/cp erroneously assumes that a file is sparse
> > if the number of blocks is smaller than $((filesize / fs_blocksize)) -
> > but in this case the file is NOT sparse, just compressed.
> > 2. The loop to copy a sparse file is faulty because there are no holes
> > in that file. That itself is IMHO already a bad idea to have a
> > separate codepath for sparse files, just the normal codepath should
> > use SEEK_HOLE and just skip those in the destination
> 
> A possible issue is that Cygwin assumes sparse files on SSD

No, that's not the problem, because SPARSE handling requires that
the FILE_ATTRIBUTE_SPARSE_FILE flag of the file is actually set.

For instance, see lseek w/ SEEK_DATA/SEEK_HOLE, which handles
non-sparse files as documented by the Linux man page
https://man7.org/linux/man-pages/man2/lseek.2.html:

        if (!has_attribute (FILE_ATTRIBUTE_SPARSE_FILE))
          {
            /* Default behaviour if sparse files are not supported:
               SEEK_DATA: seek to offset
               SEEK_HOLE: seek to EOF */
            fpi.CurrentByteOffset.QuadPart = (whence == SEEK_DATA)
                                             ? offset
                                             : fsi.EndOfFile.QuadPart;
            break;
          }

> so we need
> fhandler/disk_file:fstat_helper to allow cp to handle compressed files
> normally.

Say again?  What exactly are you expecting from stat()?  Fibbing about
the number of blocks taken by a FS-compressed file?

For the time being, it seems the assumption that #blocks * blocksize < 
filesize is not really correct.

Actually cp(1) should test if lseek(SEEK_HOLE) returns EOF.  If so,
the file isn't sparse.

On Cygwin it could also check with

  (ioctl (fd, FS_IOC_GETFLAGS, &flags) & FS_SPARSE_FL) != 0

that the file is sparse, but the lseek test is target-independent.

On Cygwin and Linux you could also test with 

  (ioctl (fd, FS_IOC_GETFLAGS, &flags) & FS_COMPR_FL) != 0

that the file is compressed, but there's a problem so far.  The
Linux flag is called FS_COMPR_FL but the Cygwin flag is called
FS_COMPRESSED_FL.  THis flag is only supported on btrfs and I'm
not sure it already existed when I added FS_IOC_GETFLAGS...

Anyway, I'll change FS_COMPR_FL to FS_COMPRESSED_FL and make
FS_COMPRESSED_FL an alias for backward compat...


Corinna

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple