DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 50FHQiTA3968372
Authentication-Results: delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results: delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter: OpenDKIM Filter v2.11.0 delorie.com 50FHQiTA3968372
Authentication-Results: delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=xkhipaOt
X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9488B3851A8B
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1736962002;
	bh=hJnOy2kCsgz56BRwg9jAe1BjIo2yjhoTs9f/CdBFltc=;
	h=Date:To:Subject:References:In-Reply-To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:
	 From;
	b=xkhipaOthaRbiqUSWd/h2vihJhYYdU+Yb3pME4JhBFwM/u0/IPxSqauBqc4bhL+U6
	 MAuEGFTfU/F94ADLdtJMQiCLcE9/7eGyLvU7r7a9dVnoefrQhwTVC62qaiSP+YLLIG
	 YZRZjbpZAampfgI3eVWAep4i2o+YsVZ2Hu2e6bXk=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 086113857014
Date: Wed, 15 Jan 2025 18:26:14 +0100
To: cygwin AT cygwin DOT com
Subject: Re: Cygwin 3.6 possible issue handling compressed .pdb files on SSD?
Message-ID: <Z4fvtuywpUttG-66@calimero.vinschen.de>
Mail-Followup-To: cygwin AT cygwin DOT com
References: <CAKAoaQmwna7sSfpYFn0OUsqpdXAgJ5FyaNkhKv6K+otr0Oa5QA AT mail DOT gmail DOT com>
 <4712dcf7-1d4d-4d27-b7c1-b705d3a0a553 AT SystematicSW DOT ab DOT ca>
 <CAKAoaQ=aj+xB2J65Rgv1tV4oSK9fq0tk8Bq6ZoXAoOG_hBR5eA AT mail DOT gmail DOT com>
 <bf9d2afd-68e4-46b3-a9e6-37de7ffb8fb1 AT SystematicSW DOT ab DOT ca>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <bf9d2afd-68e4-46b3-a9e6-37de7ffb8fb1@SystematicSW.ab.ca>
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Corinna Vinschen via Cygwin <cygwin AT cygwin DOT com>
Reply-To: cygwin AT cygwin DOT com
Cc: Corinna Vinschen <corinna-cygwin AT cygwin DOT com>
Content-Type: text/plain; charset="utf-8"
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 50FHQiTA3968372

On Jan 15 09:12, Brian Inglis via Cygwin wrote:
> On 2025-01-14 03:13, Roland Mainz via Cygwin wrote:
> > On Tue, Jan 14, 2025 at 7:19 AM Brian Inglis via Cygwin
> > <cygwin AT cygwin DOT com> wrote:
> > > On 2025-01-13 13:10, Roland Mainz via Cygwin wrote:
> > > > I just hit an endless loop with /usr/bin/cp from "coreutils" version
> > > > 9.5-1 copying a larger *.pdb file (it seems that only this specific
> > > > file is affected...) from Visual Studio 19.
> > > > [...]
> > I think I found the problem:
> > The *.pdb file uses NTFS compression:
> > ---- snip ----
> > /bin/winfsinfo filebasicinfo "$(cygpath -w
> > $PWD/../build.vc19/x64/Debug/nfs41_driver.pdb)"
> > (
> >         filename='C:\cygwin64\home\roland_mainz\work\msnfs41_uidmapping\ms-nfs41-client-kofemannvacation\build.vc19\x64\Debug\nfs41_driver.pdb'
> >         CreationTime=133812707624654816
> >         LastAccessTime=133813220892976366
> >         LastWriteTime=133812707639811081
> >         ChangeTime=133812707639811081
> >         typeset -a FileAttributes=(
> >                 FILE_ATTRIBUTE_ARCHIVE
> >                 FILE_ATTRIBUTE_COMPRESSED
> >         )
> > )
> > ---- snip ----
> > 
> > If I remove the "FILE_ATTRIBUTE_COMPRESSED" flag /bin/cp works without problems.
> > I think the issues here are:
> > 1. Coreutils 9.5-1 /bin/cp erroneously assumes that a file is sparse
> > if the number of blocks is smaller than $((filesize / fs_blocksize)) -
> > but in this case the file is NOT sparse, just compressed.
> > 2. The loop to copy a sparse file is faulty because there are no holes
> > in that file. That itself is IMHO already a bad idea to have a
> > separate codepath for sparse files, just the normal codepath should
> > use SEEK_HOLE and just skip those in the destination
> 
> A possible issue is that Cygwin assumes sparse files on SSD

No, that's not the problem, because SPARSE handling requires that
the FILE_ATTRIBUTE_SPARSE_FILE flag of the file is actually set.

For instance, see lseek w/ SEEK_DATA/SEEK_HOLE, which handles
non-sparse files as documented by the Linux man page
https://man7.org/linux/man-pages/man2/lseek.2.html:

        if (!has_attribute (FILE_ATTRIBUTE_SPARSE_FILE))
          {
            /* Default behaviour if sparse files are not supported:
               SEEK_DATA: seek to offset
               SEEK_HOLE: seek to EOF */
            fpi.CurrentByteOffset.QuadPart = (whence == SEEK_DATA)
                                             ? offset
                                             : fsi.EndOfFile.QuadPart;
            break;
          }

> so we need
> fhandler/disk_file:fstat_helper to allow cp to handle compressed files
> normally.

Say again?  What exactly are you expecting from stat()?  Fibbing about
the number of blocks taken by a FS-compressed file?

For the time being, it seems the assumption that #blocks * blocksize < 
filesize is not really correct.

Actually cp(1) should test if lseek(SEEK_HOLE) returns EOF.  If so,
the file isn't sparse.

On Cygwin it could also check with

  (ioctl (fd, FS_IOC_GETFLAGS, &flags) & FS_SPARSE_FL) != 0

that the file is sparse, but the lseek test is target-independent.

On Cygwin and Linux you could also test with 

  (ioctl (fd, FS_IOC_GETFLAGS, &flags) & FS_COMPR_FL) != 0

that the file is compressed, but there's a problem so far.  The
Linux flag is called FS_COMPR_FL but the Cygwin flag is called
FS_COMPRESSED_FL.  THis flag is only supported on btrfs and I'm
not sure it already existed when I added FS_IOC_GETFLAGS...

Anyway, I'll change FS_COMPR_FL to FS_COMPRESSED_FL and make
FS_COMPRESSED_FL an alias for backward compat...


Corinna

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple