DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1B4A83858D20
Message-ID: <af8da25d-d290-4db3-8343-dee5587f5fe3@systematicsw.ab.ca>
Date: Mon, 21 Oct 2024 22:27:21 -0600
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Installer freezes in windows-2022 GitHub Action
Content-Language: en-CA
To: cygwin AT cygwin DOT com
References: <CAGen01ickv9cenzeT3SkV9buugEh5CSGW-JWKkWfHUQUsCoVxg AT mail DOT gmail DOT com>
 <52556213-2914-449d-a719-e1a0826f6ba1 AT dronecode DOT org DOT uk>
 <CAGen01iESMAAzWL+hdp2nNpCBHiD0fhsuK-Fhw6yyFDSzV388A AT mail DOT gmail DOT com>
 <CAGen01gf91LXQrLp32DWbVFkAHn2A92WqjQOmwRm3LMtoZR2Nw AT mail DOT gmail DOT com>
 <d98dc0e8-d6be-47c6-af8f-fae93fd04bbf AT SystematicSW DOT ab DOT ca>
 <CAGen01j1FUAA0XAhH1ygEC30Bjvq2DPaWKhJ5CSoabh7trcSHg AT mail DOT gmail DOT com>
Organization: Systematic Software
In-Reply-To: <CAGen01j1FUAA0XAhH1ygEC30Bjvq2DPaWKhJ5CSoabh7trcSHg@mail.gmail.com>
Precedence: list
From: Brian Inglis via Cygwin <cygwin AT cygwin DOT com>
Reply-To: cygwin AT cygwin DOT com
Cc: Brian Inglis <brian DOT inglis AT systematicsw DOT ab DOT ca>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit

On 2024-10-21 11:36, Michele Locati via Cygwin wrote:
> Il giorno lun 21 ott 2024 alle ore 19:03 Brian Inglis via Cygwin
> <cygwin AT cygwin DOT com> ha scritto:
>>> I think it was probably an issue caused by a malformed executable in a
>>> mirror/CDN (which would explain why it worked sometimes and sometimes
>>> not).
>>
>> That's kind of worrying, but also kind of typical of MS service expectations ;^>
> 
> The problem still persists: there's really something wrong in the
> distribution of the installer...
> 
> Try running this posix script (I just tried it on a bare metal linux machine):
> 
> 
> #!/bin/sh
> set -o errexit
> set -o nounset
> url=https://cygwin.com/setup-x86_64.exe
> printf 'Downloading %s multiple times (press Ctrl+C to stop)\n' "$url"
> while :; do
>      printf 'Downloading... '
>      size=$(curl -s -o /dev/null -w '%{size_download}' "$url")
>      printf 'done: %s bytes\n' $size
>      sleep 1
> done
> 
> 
> Here's what I get:
> 
> Downloading https://cygwin.com/setup-x86_64.exe multiple times (press
> Ctrl+C to stop)
> Downloading... done: 1407507 bytes
> Downloading... done: 1407507 bytes
> Downloading... done: 0 bytes
> Downloading... done: 0 bytes
> Downloading... done: 0 bytes
> Downloading... done: 0 bytes
> Downloading... done: 0 bytes
> Downloading... ^C

I get similar results, but a proxy and/or CDN is upstream: see below for my 
opinions about some of this.

> So, unless the file is served by a Microsoft service, I don't think MS
> has anything to do with this issue...
> 
> Maybe there's some sort of protection from downloading the installer
> multiple times from the same IP?
> That would break big companies (who may have many PC internally but
> just 1 public IP), as well as services like GitHub Actions (which may
> have just a few of public IPs)....

Modern networks and utilities can deal appropriately with NATs.

> That's really something that should be fixed...

I typically use wget -N -nv for downloads, without any issues for decades, on 
any service.

These urls say do not cache:

$ curl -I https://cygwin.com/setup-x86_64.exe{,.sig}
HTTP/2 200
date: Mon, 21 Oct 2024 17:41:25 GMT
server: Apache/2.4.37 (Red Hat Enterprise Linux) OpenSSL/1.1.1k mod_qos/11.74 
mod_wsgi/4.6.4 Python/3.6 mod_perl/2.0.12 Perl/v5.26.3
vary: User-Agent
last-modified: Tue, 16 Apr 2024 18:39:28 GMT
etag: "236-6163b0ed908ba"
accept-ranges: bytes
content-length: 566
cache-control: max-age=0
expires: Mon, 21 Oct 2024 17:41:25 GMT
content-security-policy: default-src 'self' http: https:
strict-transport-security: max-age=16070400
x-cache: MISS from cygwin.com
content-type: application/pgp-signature

HTTP/2 200
date: Mon, 21 Oct 2024 17:41:25 GMT
server: Apache/2.4.37 (Red Hat Enterprise Linux) OpenSSL/1.1.1k mod_qos/11.74 
mod_wsgi/4.6.4 Python/3.6 mod_perl/2.0.12 Perl/v5.26.3
vary: User-Agent
last-modified: Tue, 16 Apr 2024 18:39:04 GMT
etag: "157a13-6163b0d6e8405"
accept-ranges: bytes
content-length: 1407507
cache-control: max-age=0
expires: Mon, 21 Oct 2024 17:41:25 GMT
content-security-policy: default-src 'self' http: https:
strict-transport-security: max-age=16070400
x-cache: MISS from cygwin.com
content-type: application/octet-stream

Note:
CDNs use proxies and add X-... headers to results:
X-Cache-Lookup	proxy has cache-able resource	HIT yes, MISS no
X-Cache		proxy result served from cache	HIT yes, MISS no

If those headers are ignored by GitHub or their or your upstream, or an 
intermediate proxy, or they have *policies*, or they or you have an AntiVirus 
product, who knows what happens?

As this infrastructure is shared by other sourceware.org projects, and we have 
not heard anything else, it may be a problem nearer yourself.
If you have networking support, ask them why you get nothing from a request?

Also note the response cache hint "vary: User-Agent" allowing the CDN/proxy 
cache to make assumptions about how different user agents will handle requests: 
in the case of requests from command line clients, they can perhaps assume that 
the previously downloaded file will be left intact if nothing is provided.

Remember that these are really browser queries and it is not a problem for the 
CDN/proxy cache if the client does not actually have a local cache from which 
they can serve the file: the previously downloaded copy may have to be handled 
as if it is a cached copy.

Also, you can run curl as if it were like wget using such options as:

	-R, --remote-time
	-Z, --parallel
	-z, --time-cond TIME|FILE
also
	--etag-save 	FILE
	--etag-compare	FILE

similar to cookies.

And Cygwin curl is packaged with easy front end wcurl which works like wget 
using curl and a bunch of useful assumptions about downloading to set 
appropriate options.

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple