delorie.com/archives/browse.cgi | search |
X-Recipient: | archive-cygwin AT delorie DOT com |
DKIM-Filter: | OpenDKIM Filter v2.11.0 sourceware.org F234E3846405 |
DKIM-Signature: | v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com; |
s=default; t=1712132115; | |
bh=LWpiE65sa+42w+b5/xjq8M30NCtufjWcR2J58zmE1No=; | |
h=Date:To:Subject:References:In-Reply-To:List-Id:List-Unsubscribe: | |
List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: | |
From; | |
b=ymNg4SLpJt7Jcdlz/COt3TmFrV6evOAupfopkN1C8YdAlgJDhrbCPSTVa6TjI2yEl | |
RCQuwDzei+xjGhEUe7iwWCw3MIGimIXNglbtOebHv7hd5MXGP+/wNWy5Mxb3knBqPb | |
7P8U5yJ4SY+96r0VWnQ8QqYfm1mOI6Pu6ome/nLM= | |
X-Original-To: | cygwin AT cygwin DOT com |
Delivered-To: | cygwin AT cygwin DOT com |
DKIM-Filter: | OpenDKIM Filter v2.11.0 sourceware.org D12293846411 |
Date: | Wed, 3 Apr 2024 10:14:48 +0200 |
To: | cygwin AT cygwin DOT com |
Subject: | Re: Cygwin&Win32 file prefetch, block sizes? |
Message-ID: | <Zg0P-JRyEyiyZJCP@calimero.vinschen.de> |
Mail-Followup-To: | cygwin AT cygwin DOT com |
References: | <CANH4o6Mxai4_C=d1P93Prrimb8_H=trTwm-Eg+WBwpomN3tNJw AT mail DOT gmail DOT com> |
<ZgwFNde2z804koS_ AT calimero DOT vinschen DOT de> | |
<CANH4o6P8cts9TJgpdjR4mi+sj2YvuDa=d49XLcEVvYnRB81KRw AT mail DOT gmail DOT com> | |
MIME-Version: | 1.0 |
In-Reply-To: | <CANH4o6P8cts9TJgpdjR4mi+sj2YvuDa=d49XLcEVvYnRB81KRw@mail.gmail.com> |
X-BeenThere: | cygwin AT cygwin DOT com |
X-Mailman-Version: | 2.1.30 |
List-Id: | General Cygwin discussions and problem reports <cygwin.cygwin.com> |
List-Unsubscribe: | <https://cygwin.com/mailman/options/cygwin>, |
<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe> | |
List-Archive: | <https://cygwin.com/pipermail/cygwin/> |
List-Post: | <mailto:cygwin AT cygwin DOT com> |
List-Help: | <mailto:cygwin-request AT cygwin DOT com?subject=help> |
List-Subscribe: | <https://cygwin.com/mailman/listinfo/cygwin>, |
<mailto:cygwin-request AT cygwin DOT com?subject=subscribe> | |
From: | Corinna Vinschen via Cygwin <cygwin AT cygwin DOT com> |
Reply-To: | cygwin AT cygwin DOT com |
Cc: | Corinna Vinschen <corinna-cygwin AT cygwin DOT com> |
Errors-To: | cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com |
Sender: | "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com> |
X-MIME-Autoconverted: | from base64 to 8bit by delorie.com id 4338FFqO2173134 |
On Apr 3 00:35, Martin Wege via Cygwin wrote: > On Tue, Apr 2, 2024 at 3:17 PM Corinna Vinschen via Cygwin > <cygwin AT cygwin DOT com> wrote: > > > > On Apr 2 02:04, Martin Wege via Cygwin wrote: > > > Hello, > > > > > > Is there any document which describes how Cygwin and Win32 file > > > prefetch and readahead work, and which sizes are used (e.g. always > > > read one full page even if only 16 bytes are requested?)? > > > > I'm not aware of any docs, but again, keep in mind that Cygwin is a > > usersapce DLL. We basically do what Windows does for low-level file > > access. > > > > > Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that > > > mean the file's block size is really 64k? Is this info per filesystem, > > > or hardcoded in Cygwin? > > > > Hardcoded in Cygwin since 2017, based on a discussion in terms of > > file access performance, especially when using stdio.h functions: > > > > https://cygwin.com/cgit/newlib-cygwin/commit/?id=7bef7db5ccd9c > > OUCH. > > While I can understand the motivation, FAT32 on multi-GB-devices > having 64k block size, and Win32 API on Win95/98/ME/Win7 being > optimized to that insane block size, it is absolutely WRONG with > today's NTFS and even more so with ReFS. This only works if you stream > files, but as soon as you are doing random read/writes the performance > is terrible due to cache thrashing. That could explain the many > complaints about Cygwin's IO performance. The above patch *only* sets stat::st_blksize to 64K. Nothing else happens! This usually means that stdio.h functions use this size for their buffer and readahead. It doesn't affect direct calls to read(2)/write(2) and fread(3)/fwrite(3) at all! > So, what can be done? I'm not a benchmarking guru, so I'd like to > propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to No. We have two ways to handle this *iff* there's really a reason to handle this. - Either we just lower PREFERRED_IO_BLKSIZE to 4K or 8K, but that's kind of bad in terms of pipes, the clipboard, etc. - So we keep PREFERRED_IO_BLKSIZE at 64K but don't use it for disk files. Rather, we read this info from the filesystem: https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddk/ns-ntddk-_file_fs_sector_size_information If the filesystem is local and SSINFO_FLAGS_NO_SEEK_PENALTY is set, we could stick to 64K. Otherwise the PhysicalBytesPerSectorForPerformance member might be helpful I guess. Needs checking, of course. If this isn't any good, we can still fallback to FILE_FS_FULL_SIZE_INFORMATION as in fhandler_base::fstatvfs_by_handle, https://cygwin.com/cgit/newlib-cygwin/tree/winsup/cygwin/fhandler/disk_file.cc#n661 Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation: https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
webmaster | delorie software privacy |
Copyright © 2019 by DJ Delorie | Updated Jul 2019 |