X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D1CA1384772F
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1712166262;
	bh=XvFApa+VmEtoBjh+uf5uW8ThgmuZPIS2+BLk9lMbpcM=;
	h=References:In-Reply-To:Date:Subject:To:Cc:List-Id:
	 List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe:
	 From:Reply-To:From;
	b=JGJugLGo9lnwgrdXFfllVr+4Tm+ALoW3GDmN4wQ/2didgIOQdZ+BLqS6lHv34P9MC
	 MrSu7OM3JT+wFz4GaKLtfu8r9e1pwmAiWx5BleZIT5TrwHAh6lZy+gSBfIo3K4utg2
	 XEwNlC/u1151BEIGiZR10MtgNga5FUWA4LSFt+I8=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 974193847725
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 974193847725
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1712166213; cv=none;
 b=TB9p1EIz5V3S9qI3mT6eghFaydOb53fGVC38SSn2uR5c/O8FM9FH3Gmk/ivRj6/9bXkPRkd5f79sxqDKZp8whJjsPRK778W6Hq/RP/oiRDL5fo8gB+qECqaxqmNYglQN8Q9jI+ntD4VpDwpThkpntgJnjjo8lXJ7y1J4CVoEpfo=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1712166213; c=relaxed/simple;
 bh=fVtyHRD9E/KMUidjJys3zVTOXdbc09/Wluw9NDepOI4=;
 h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To;
 b=kVJKDVxUdyJTYMj7owaORjMmQnHs5d4cWXHnaQQz95WgMZKSm9OXvFX/Gh1TOErhgQ9DDzPSkpoqxxk6wvP3DoSNyk9TaMnh7nc7RnflDKHrMEHpBI4VE7fFBcIhfceu+sAN5IeOSFugk2L3N6qb+4o5rZHYDsdML4XpegljXfE=
ARC-Authentication-Results: i=1; server2.sourceware.org
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1712166209; x=1712771009;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=S2fVG3rugi+7EaJgBoVyEOJ9p6EdMG67Q0mMv9L/cqA=;
 b=DBizfvXsavqwtaOUN4fCWCBGMquhBMz/QZmrqBvJ/AYWtkYXZYzIL2iYjLQHCtDucL
 0dsLjcu80Od6gzfVuj6sY38gdklQ5CxlqlGePANgoZot9OjkaK/Q/rSz+hTkNM1WGOcg
 jCqbyhZRdqzfHinY4n1/NNw3YT8fivJEaLodjn++YcT+mHHLgqLgUle5iHebf/29k9W3
 xkVOPXW+3S4VfTVQ/XWYrlXAEJf1OX+d7tJQSmyS/NZVGVKLy24soTiJ857MMJ9OMZLP
 P/EAvOEyq2q3yINsp5MnYwFCA7YT2BCbwgyQPYmldD/LzLHv4DCy0NgAmY72Z9y3J7kd
 YgoA==
X-Gm-Message-State: AOJu0YzmpTAJjjL4t0WSm6udrz+rmeio97hnORfzZyQXHJqH/RR94u77
 Jp5Kpa2nmuXjERPXO5BAO0RXkS8P/RtkNL+7YfVF7mcGqYHiSnFtetYiDswItubcOdlyqdxTbl6
 H3OYy82ZPJro/bijLtjjFuclu6G0+i9d/I6M=
X-Google-Smtp-Source: AGHT+IGvPSeei9j3h/VtHTO6OF3Nj0xarGIrlI8TBG1a8YKOl3fnIxi8zB3OPODMBGtqDPLMhCyzJI4eqiMOUUHJG4Y=
X-Received: by 2002:a50:9e81:0:b0:56e:ee2:7897 with SMTP id
 a1-20020a509e81000000b0056e0ee27897mr127657edf.5.1712166208666; Wed, 03 Apr
 2024 10:43:28 -0700 (PDT)
MIME-Version: 1.0
References: <CANH4o6Mxai4_C=d1P93Prrimb8_H=trTwm-Eg+WBwpomN3tNJw AT mail DOT gmail DOT com>
 <ZgwFNde2z804koS_ AT calimero DOT vinschen DOT de>
 <CANH4o6P8cts9TJgpdjR4mi+sj2YvuDa=d49XLcEVvYnRB81KRw AT mail DOT gmail DOT com>
 <Zg0P-JRyEyiyZJCP AT calimero DOT vinschen DOT de>
In-Reply-To: <Zg0P-JRyEyiyZJCP@calimero.vinschen.de>
Date: Wed, 3 Apr 2024 19:42:51 +0200
Message-ID: <CALXu0UcBTXOQ9yGf5_BPOHrdQw9SPyzD9n0sW9ikM4kBQPMLTg@mail.gmail.com>
Subject: Re: Cygwin&Win32 file prefetch, block sizes?
To: cygwin AT cygwin DOT com
Cc: Corinna Vinschen <corinna-cygwin AT cygwin DOT com>
X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_00, DKIM_SIGNED,
 DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,
 SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Cedric Blancher via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Cedric Blancher <cedric DOT blancher AT gmail DOT com>
Content-Type: text/plain; charset="utf-8"
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 433HiNvU2340139

On Wed, 3 Apr 2024 at 10:15, Corinna Vinschen via Cygwin
<cygwin AT cygwin DOT com> wrote:
>
> On Apr  3 00:35, Martin Wege via Cygwin wrote:
> > On Tue, Apr 2, 2024 at 3:17 PM Corinna Vinschen via Cygwin
> > <cygwin AT cygwin DOT com> wrote:
> > >
> > > On Apr  2 02:04, Martin Wege via Cygwin wrote:
> > > > Hello,
> > > >
> > > > Is there any document which describes how Cygwin and Win32 file
> > > > prefetch and readahead work, and which sizes are used (e.g. always
> > > > read one full page even if only 16 bytes are requested?)?
> > >
> > > I'm not aware of any docs, but again, keep in mind that Cygwin is a
> > > usersapce DLL. We basically do what Windows does for low-level file
> > > access.
> > >
> > > > Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that
> > > > mean the file's block size is really 64k? Is this info per filesystem,
> > > > or hardcoded in Cygwin?
> > >
> > > Hardcoded in Cygwin since 2017, based on a discussion in terms of
> > > file access performance, especially when using stdio.h functions:
> > >
> > >   https://cygwin.com/cgit/newlib-cygwin/commit/?id=7bef7db5ccd9c
> >
> > OUCH.
> >
> > While I can understand the motivation, FAT32 on multi-GB-devices
> > having 64k block size, and Win32 API on Win95/98/ME/Win7 being
> > optimized to that insane block size, it is absolutely WRONG with
> > today's NTFS and even more so with ReFS. This only works if you stream
> > files, but as soon as you are doing random read/writes the performance
> > is terrible due to cache thrashing. That could explain the many
> > complaints about Cygwin's IO performance.
>
> The above patch *only* sets stat::st_blksize to 64K. Nothing else
> happens!

Yes, but applications use that information, and then make wrong choices.

>
> This usually means that stdio.h functions use this size for their buffer
> and readahead.  It doesn't affect direct calls to read(2)/write(2) and
> fread(3)/fwrite(3) at all!
>
> > So, what can be done? I'm not a benchmarking guru, so I'd like to
> > propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to
>
> No.
>
> We have two ways to handle this *iff* there's really a reason to
> handle this.
>
> - Either we just lower PREFERRED_IO_BLKSIZE to 4K or 8K, but that's
>   kind of bad in terms of pipes, the clipboard, etc.

I think the env variable was strictly meant for benchmarking **ONLY**,
so someone with good benchmarking expertise can grab the Cygwin 3.6
alpha binaries and does benchmarking with them.

>
> - So we keep PREFERRED_IO_BLKSIZE at 64K but don't use it for disk
>   files.  Rather, we read this info from the filesystem:
>
>   https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddk/ns-ntddk-_file_fs_sector_size_information

@Corinna Vinschen  Could you please do me a favour and check whether
SMB supports FILE_FS_SECTOR_SIZE_INFORMATION?

>
>   If the filesystem is local and SSINFO_FLAGS_NO_SEEK_PENALTY is set, we
>   could stick to 64K.

That still means that buffers will be very large, and associated
memcpy(), read(), write() are large too. Also harmful is that such
buffers might be allocated with alloca(), because the original authors
didn't anticipate 64k buffer sizes...

>
>   Otherwise the PhysicalBytesPerSectorForPerformance member might be
>   helpful I guess.  Needs checking, of course.
>
>   If this isn't any good, we can still fallback to
>   FILE_FS_FULL_SIZE_INFORMATION as in fhandler_base::fstatvfs_by_handle,
>   https://cygwin.com/cgit/newlib-cygwin/tree/winsup/cygwin/fhandler/disk_file.cc#n661

You mean when FILE_FS_SECTOR_SIZE_INFORMATION is not supported you use
FILE_FS_FULL_SIZE_INFORMATION instead?

Ced
-- 
Cedric Blancher <cedric DOT blancher AT gmail DOT com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple