X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org F234E3846405
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1712132115;
	bh=LWpiE65sa+42w+b5/xjq8M30NCtufjWcR2J58zmE1No=;
	h=Date:To:Subject:References:In-Reply-To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:
	 From;
	b=ymNg4SLpJt7Jcdlz/COt3TmFrV6evOAupfopkN1C8YdAlgJDhrbCPSTVa6TjI2yEl
	 RCQuwDzei+xjGhEUe7iwWCw3MIGimIXNglbtOebHv7hd5MXGP+/wNWy5Mxb3knBqPb
	 7P8U5yJ4SY+96r0VWnQ8QqYfm1mOI6Pu6ome/nLM=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D12293846411
Date: Wed, 3 Apr 2024 10:14:48 +0200
To: cygwin AT cygwin DOT com
Subject: Re: Cygwin&Win32 file prefetch, block sizes?
Message-ID: <Zg0P-JRyEyiyZJCP@calimero.vinschen.de>
Mail-Followup-To: cygwin AT cygwin DOT com
References: <CANH4o6Mxai4_C=d1P93Prrimb8_H=trTwm-Eg+WBwpomN3tNJw AT mail DOT gmail DOT com>
 <ZgwFNde2z804koS_ AT calimero DOT vinschen DOT de>
 <CANH4o6P8cts9TJgpdjR4mi+sj2YvuDa=d49XLcEVvYnRB81KRw AT mail DOT gmail DOT com>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <CANH4o6P8cts9TJgpdjR4mi+sj2YvuDa=d49XLcEVvYnRB81KRw@mail.gmail.com>
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Corinna Vinschen via Cygwin <cygwin AT cygwin DOT com>
Reply-To: cygwin AT cygwin DOT com
Cc: Corinna Vinschen <corinna-cygwin AT cygwin DOT com>
Content-Type: text/plain; charset="utf-8"
Errors-To: cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 4338FFqO2173134

On Apr  3 00:35, Martin Wege via Cygwin wrote:
> On Tue, Apr 2, 2024 at 3:17 PM Corinna Vinschen via Cygwin
> <cygwin AT cygwin DOT com> wrote:
> >
> > On Apr  2 02:04, Martin Wege via Cygwin wrote:
> > > Hello,
> > >
> > > Is there any document which describes how Cygwin and Win32 file
> > > prefetch and readahead work, and which sizes are used (e.g. always
> > > read one full page even if only 16 bytes are requested?)?
> >
> > I'm not aware of any docs, but again, keep in mind that Cygwin is a
> > usersapce DLL. We basically do what Windows does for low-level file
> > access.
> >
> > > Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that
> > > mean the file's block size is really 64k? Is this info per filesystem,
> > > or hardcoded in Cygwin?
> >
> > Hardcoded in Cygwin since 2017, based on a discussion in terms of
> > file access performance, especially when using stdio.h functions:
> >
> >   https://cygwin.com/cgit/newlib-cygwin/commit/?id=7bef7db5ccd9c
> 
> OUCH.
> 
> While I can understand the motivation, FAT32 on multi-GB-devices
> having 64k block size, and Win32 API on Win95/98/ME/Win7 being
> optimized to that insane block size, it is absolutely WRONG with
> today's NTFS and even more so with ReFS. This only works if you stream
> files, but as soon as you are doing random read/writes the performance
> is terrible due to cache thrashing. That could explain the many
> complaints about Cygwin's IO performance.

The above patch *only* sets stat::st_blksize to 64K. Nothing else
happens!

This usually means that stdio.h functions use this size for their buffer
and readahead.  It doesn't affect direct calls to read(2)/write(2) and
fread(3)/fwrite(3) at all!

> So, what can be done? I'm not a benchmarking guru, so I'd like to
> propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to

No.

We have two ways to handle this *iff* there's really a reason to
handle this.

- Either we just lower PREFERRED_IO_BLKSIZE to 4K or 8K, but that's
  kind of bad in terms of pipes, the clipboard, etc.

- So we keep PREFERRED_IO_BLKSIZE at 64K but don't use it for disk
  files.  Rather, we read this info from the filesystem:

  https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddk/ns-ntddk-_file_fs_sector_size_information

  If the filesystem is local and SSINFO_FLAGS_NO_SEEK_PENALTY is set, we
  could stick to 64K.

  Otherwise the PhysicalBytesPerSectorForPerformance member might be
  helpful I guess.  Needs checking, of course.

  If this isn't any good, we can still fallback to
  FILE_FS_FULL_SIZE_INFORMATION as in fhandler_base::fstatvfs_by_handle,
  https://cygwin.com/cgit/newlib-cygwin/tree/winsup/cygwin/fhandler/disk_file.cc#n661

Corinna


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple