X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4FC863858CDA
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1712121733;
	bh=fmbMRFzK/jvS79l5mpGtqDNg7Nw1EqaUvep8/PwASjg=;
	h=References:In-Reply-To:Date:Subject:To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	 From;
	b=YZHnItH6gb67OCN/0eLeE+DPVW0KsLNx4oGwGOp2hT/srRmqYu+Fq8Z7o0CBFv/Zz
	 lz2nc9yFWW6/p6EuKVmU1A2nTU22yGAw9a+gxKOPJWmqO4FRwYucBP2PjXD2awky0m
	 UhXzr1NziI0QkQ5R/5w8We8vDOqZgkgW1hpCXfbc=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4AEBE3858CDA
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 4AEBE3858CDA
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1712121713; cv=none;
 b=KtKC0dplkv/SZB/VZfVma/pn4RA9b26NYZgqCW8/VD6Rgl09liGyvjQ6+TDm5OYaBIpdHP6dqYplo80lMl+rOVgt5WDmWPgD7v72EJO3mvS6Kh6zCmd7Agchhml90LRH25bX//xiz9VI7SvTZiOOAxp+unbCyJleZqDVQkRNu1o=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1712121713; c=relaxed/simple;
 bh=gviTmeKr+gdphgx8x8KlV1ulJRuyv/BM+Jgv8P4dHko=;
 h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To;
 b=KnLGm4OICuLbs+RqmCd85qcp6FeUnePIR1qqn2oKctL3njPIe4hX43125UHyu27Av+64IUo0Lk4fnWQB3hvnalwrDlDxShiPQDX7eWNID8cs1JLSIyUFRhEQzDETEepW0YV5w4whJjsN6aQeeJa6YfpgQLHmCPZp7XrVIwn1ql4=
ARC-Authentication-Results: i=1; server2.sourceware.org
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1712121708; x=1712726508;
 h=content-transfer-encoding:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=pUnY6GhsqNWt0gZLO3pzlxVkV9EPtJcTIQlCVaBFxz4=;
 b=Bov7/P9auAoZMBzM6yr6X/ogtpmIodO4nAoiRenZLiP1eGR63p/fxSbsHMr00R+BLm
 zbAm+nfa+801QhYtwo2Ljpi6Ox4pe0LqCaXIK/4G9vRlqQjsAFeFxPPZrugXprFCn0gG
 7hkoSt57mS4IosqRvyua0zntHJeKFDUIbcZcMSTiHU5rS1fRcpIfWmJVrlTCBTAbi1A7
 S91g3rN8G/luXs0eoLJw4+Q56Zp2SBfn8XBeQ+hXg4aEYZGb1hwVf524alUMkuVDW33v
 Uk3GV6l0O97cXWNTdKFkhHFdflFsrRFL3jDDa+wyz7wqBxREQ8UNsZ4mQ6TKRDG4GSOG
 2bCw==
X-Gm-Message-State: AOJu0YxH/xfhb4K0f5tecafzMB95cdjcGXCVBACE7apqIl0OVGcaaY9z
 POEuGe7h/gWfgi5VBIoZ5bs82YOdpYxMxkiSCSRuM+n5QuumYLrTFtLZcR004uZpv+fgn/qM5pJ
 pL1nlYlYU9RhypQi/qEIbRsvFzpzskXU9
X-Google-Smtp-Source: AGHT+IGDUXWTVpS032gZw1JQjOQ1GncrsmNq59ta0zwcHuuMOqK9kTBEJtaoSipKHG0KVuZ/oYkh3RdD547MM9sLHsk=
X-Received: by 2002:a50:8d41:0:b0:56c:4db:33f7 with SMTP id
 t1-20020a508d41000000b0056c04db33f7mr11200334edt.10.1712121708267; Tue, 02
 Apr 2024 22:21:48 -0700 (PDT)
MIME-Version: 1.0
References: <CANH4o6Mxai4_C=d1P93Prrimb8_H=trTwm-Eg+WBwpomN3tNJw AT mail DOT gmail DOT com>
 <ZgwFNde2z804koS_ AT calimero DOT vinschen DOT de>
 <CANH4o6P8cts9TJgpdjR4mi+sj2YvuDa=d49XLcEVvYnRB81KRw AT mail DOT gmail DOT com>
In-Reply-To: <CANH4o6P8cts9TJgpdjR4mi+sj2YvuDa=d49XLcEVvYnRB81KRw@mail.gmail.com>
Date: Wed, 3 Apr 2024 07:21:00 +0200
Message-ID: <CALXu0UdzRZ-7e2Vc=z1ABKYPK8X8xg6uea41Up3VZqJZ3w=Pzg@mail.gmail.com>
Subject: Re: Cygwin&Win32 file prefetch, block sizes?
To: cygwin AT cygwin DOT com
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_00, DKIM_SIGNED,
 DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,
 SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Cedric Blancher via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Cedric Blancher <cedric DOT blancher AT gmail DOT com>
Content-Type: text/plain; charset="utf-8"
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 4335MFvX2126254

On Wed, 3 Apr 2024 at 00:36, Martin Wege via Cygwin <cygwin AT cygwin DOT com> wrote:
>
> On Tue, Apr 2, 2024 at 3:17 PM Corinna Vinschen via Cygwin
> <cygwin AT cygwin DOT com> wrote:
> >
> > On Apr  2 02:04, Martin Wege via Cygwin wrote:
> > > Hello,
> > >
> > > Is there any document which describes how Cygwin and Win32 file
> > > prefetch and readahead work, and which sizes are used (e.g. always
> > > read one full page even if only 16 bytes are requested?)?
> >
> > I'm not aware of any docs, but again, keep in mind that Cygwin is a
> > usersapce DLL. We basically do what Windows does for low-level file
> > access.
> >
> > > Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that
> > > mean the file's block size is really 64k? Is this info per filesystem,
> > > or hardcoded in Cygwin?
> >
> > Hardcoded in Cygwin since 2017, based on a discussion in terms of
> > file access performance, especially when using stdio.h functions:
> >
> >   https://cygwin.com/cgit/newlib-cygwin/commit/?id=7bef7db5ccd9c
>
> OUCH.
>
> While I can understand the motivation, FAT32 on multi-GB-devices
> having 64k block size, and Win32 API on Win95/98/ME/Win7 being
> optimized to that insane block size, it is absolutely WRONG with
> today's NTFS and even more so with ReFS. This only works if you stream
> files, but as soon as you are doing random read/writes the performance
> is terrible due to cache thrashing. That could explain the many
> complaints about Cygwin's IO performance.
>

It can also explain why Cygwin is so slow on SMB filesystems. If it
really works on 64k blocks, then this would pull all small files over
the wire, even if only a bit gets touched.

Thinking more about this, this basically defeats every
driver/tcp/ip/ethernet/net/switch optimization like jumbo frames et
al.
/usr/bin/stat %o (optimum IO block size hint) also returns 65536 on
SMB, which is NOT GOOD:

This needs to be confirmed, and if this is really true, then it should
be fixed for SMB.

I need a coffee to think about this...

> So, what can be done? I'm not a benchmarking guru, so I'd like to
> propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to
> the CYGWIN env variable (marked as "experimental"), so the
> benchmarking guys can do performance testing without recompiling
> everything, get perf results for Cygwin 3.6, and decide what to do for
> Cygwin 3.7.

I would agree, but I would also clamp the minimum at page size (4096
bytes on x86) and 8M (4x 2M hugepage size) to prevent abuse.

Does Cygwin partake in the GSOC programm (Google Summer Of Code)? This
would IMO be a high priority item.

Ced
-- 
Cedric Blancher <cedric DOT blancher AT gmail DOT com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple