X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 655A43952027
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1592502827;
	bh=g2mN9C968jbRfQXMNluDF03mHE1HsD5SCPkz/bJ9PQ0=;
	h=References:In-Reply-To:Date:Subject:To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	From;
	b=T5v5/lUIXsZUDqi4wd5MRRkRC2MYJMHrULKnL/lQ6btweaSdn8GZYuDuVREvlhT9c
	bSxHZ4kYpS1/2v/OGLnMNnKGij6vn870+jIa3kLnbitCmxTCsq3Izt96Enm5wm0itl
	HUp2iYWMlpnVdjLhUwH3JScIBrkcvumvGAxa+itU=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.3.2 sourceware.org 6B1F33860017
X-Google-DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20161025;
	h=x-gm-message-state:mime-version:references:in-reply-to:from:date
	:message-id:subject:to;
	bh=2KaUUOMPna4Hu0YD1mHDQ2tCIkY/JGYWBXIUsaCgFZE=;
	b=WZgXSLDLaFMAHXw79QedtS6Qtxj24266nEiQkeFfOw9uxjyi6KYY812nFycCLOH+OZ
	yMrMc2Bl6Nwnd4lj/2mergroHxzliGlcU57otDq+o/D3xrjSYcj6Up4YqIdXCtEcpz83
	N3BqfXk8Ab1KVv9RRamjmkrdGsjyO6ir+0FHYo1F2/fjRVqykvLPAnNoMOT75QMxPG/1
	+KuMRcwlg2TJm/foazMM/8MI8WS8XT4KQfzclDnZGkxgpzhczYhBTwjrAQeqDXwX2kO8
	T2fi/BHqUkWMCFJauf1J0nCcuDoYgXpN21tHMZdml1CNiD4OtD4y6oceRJd5GheCui0f
	PgOA==
X-Gm-Message-State:	AOAM530sXaMhEGahGgUmpYJ5xbJoTgHqO6JG+Y9EbPbU3qch97xgHiEZ
	gw2h3/XuLvujs4FrFXKHEV3hbiRtUtLuNo7eAlRpcM/4
X-Google-Smtp-Source:	ABdhPJwoWsfmEZqLhqfnhWjVM+M29YyUJbbdFFPdWW1EwWX33zgzeq+v8jxuHL2mlUJZaYyf9A+3AD1qM69CCNo+igo=
X-Received:	by 2002:a05:6808:4b:: with SMTP id
	v11mr3862714oic.31.1592502822573;
	Thu, 18 Jun 2020 10:53:42 -0700 (PDT)
MIME-Version:	1.0
References:	<CAA5t8VqpU2KiQ_Kk5czjLzeCDzf21Gb2iBLu25xB43ZhpfmaPQ AT mail DOT gmail DOT com>
	<b68e371b-4e3d-3ab3-9a0e-2461697d814d AT gmail DOT com>
	<CAA5t8Vpym95FDhvanJSNd910kWssJqK46ZzmF2jckQbY_7YxOw AT mail DOT gmail DOT com>
	<f224a4a9-ea7d-e22c-0285-19d09ebd572f AT gmail DOT com>
	<CAA5t8Vpanxxop9THkbod115y_Xd2Q2QvYDXvBYOA7N9BqQw-Bw AT mail DOT gmail DOT com>
	<70ec2ece-c85e-c122-b4a0-3089da5c8a2c AT SystematicSw DOT ab DOT ca>
	<CAA5t8Vo=bVo9n-PHMrJW00HJ1LzeJxAaHZb4KOujnb+gyh9GKg AT mail DOT gmail DOT com>
	<3de6e80d-5ba5-3d12-f527-97b8ff3eaae7 AT SystematicSw DOT ab DOT ca>
	<CAA5t8Vo6PdHvSGwd49cBtD09QWOThiiHnwsMOkpVFV9-XG-h9w AT mail DOT gmail DOT com>
	<CAA5t8Vrao2csnVGNy4p396+nDBiU0eJGcsz=SV7WrUGYPwXfsw AT mail DOT gmail DOT com>
	<CAA5t8Vo8AcKg84acBhGefyO7WFR_Bwu_xLadDhPeUd4dnq1Z7Q AT mail DOT gmail DOT com>
	<d91c8335-77c8-c7b3-b10a-beec1ac2e016 AT SystematicSw DOT ab DOT ca>
In-Reply-To:	<d91c8335-77c8-c7b3-b10a-beec1ac2e016@SystematicSw.ab.ca>
Date:	Thu, 18 Jun 2020 10:53:30 -0700
Message-ID:	<CAA5t8VprFNw8RqAXFKBXn3-gcYMRq=179Jfd3Mbp3qjCGg=xaw@mail.gmail.com>
Subject:	Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not
	in cmd shell
To:	The Cygwin Mailing List <cygwin AT cygwin DOT com>
X-Spam-Status:	No, score=-3.6 required=5.0 tests=BAYES_00, DKIM_SIGNED,
	DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, HTML_MESSAGE,
	RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS,
	TXREP autolearn=ham autolearn_force=no version=3.4.2
X-Spam-Checker-Version:	SpamAssassin 3.4.2 (2018-09-13) on
	server2.sourceware.org
X-Content-Filtered-By:	Mailman/MimeDel 2.1.29
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.29
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<http://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	David Karr via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	David Karr <davidmichaelkarr AT gmail DOT com>
Sender:	"Cygwin" <cygwin-bounces AT cygwin DOT com>

On Wed, Jun 17, 2020 at 6:06 PM Brian Inglis  wrote:

> On 2020-06-17 10:39, David Karr via Cygwin wrote:
> > On Mon, Jun 15, 2020 at 9:10 AM David Karr wrote:
> >> On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote:
> >>> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis wrote:
> >>>> On 2020-06-14 12:16, David Karr via Cygwin wrote:
> >>>>> On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis wrote:
> >>>>>> On 2020-06-14 09:38, David Karr via Cygwin wrote:
> >>>>>>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
> >>>>>>>> On 14.06.2020 08:12, David Karr wrote:
> >>>>>>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
> >>>>>>>>>> On 13.06.2020 20:53, David Karr via Cygwin wrote:
> >>>>>>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a
> >>>> while,
> >>>>>>>>>>> to communicate to our in-house k8s clusters. I often use
> "kubectl
> >>>>>>>>>>> exec" to open a shell in a container or directly execute a
> shell
> >>>>>>>>>>> command.
> >>>>>>>>>>> This has worked perfectly fine for a long time.
> >>>>>>>>>>> A couple of days ago, I discovered that all of these attempts
> >>>> were
> >>>>>>>>>>> failing with "Upgrade request required".  I hadn't upgraded
> >>>> kubectl
> >>>>>>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s
> >>>> upgrade,
> >>>>>>>>>>> but it's entirely possible.
> >>>>>>>>>>> A colleague of mine has a very similar desktop configuration
> >>>>>>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
> >>>>>>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it
> >>>> shows
> >>>>>>>>>>> the resulting "curl" command that it runs. I tried that
> resulting
> >>>>>>>>>>> command, and it results in the same response. I then tried
> >>>> updating
> >>>>>>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
> >>>>>> entire resulting "kubectl exec" command line and ran
> >>>>>>>>>>> it in a "cmd" shell.  No problem at all.  No error.
> >>>>>>>>>>> I know I haven't provided much useful information yet. I wanted
> >>>> to
> >>>>>>>>>>> get an initial response before I started providing those
> >>>> diagnostics.
> >>>>>>>>>>> Is there a clear issue here that I'm not aware of?
> >>>>>>
> >>>>>>>>> from where is kubectl coming from ?
> >>>>>>>>> In cygwin I found only a kubectl.py in the ansible package
> >>>>>>
> >>>>>>>>>> It's from here:
> >>>>>>>>>>
> >>>>>>
> >>>>
> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
> >>>>>>
> >>>>>>>> so it is NOT a cygwin program.
> >>>>>>>> If the warning is coming about curl, it is likely
> >>>>>>>> that using from cygwin you are using the cygwin curl
> >>>>>>>> and from CMD the windows one
> >>>>>>>> $ which -a curl
> >>>>>>>> /usr/bin/curl
> >>>>>>>> /cygdrive/c/WINDOWS/system32/curl
> >>>>>>>> $ /cygdrive/c/WINDOWS/system32/curl -V
> >>>>>>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
> >>>>>>>> Release-Date: 2017-11-14, security patched: 2019-11-05
> >>>>>>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s
> smtp
> >>>>>> smtps
> >>>>>>>> telnet tftp
> >>>>>>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
> >>>>>>>>
> >>>>>>>> $ /usr/bin/curl -V
> >>>>>>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f
> >>>> zlib/1.2.11
> >>>>>>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
> >>>>>>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
> >>>>>>>> Release-Date: 2019-09-11
> >>>>>>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap
> >>>> ldaps
> >>>>>>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
> >>>>>>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN
> IPv6
> >>>>>>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL
> TLS-SRP
> >>>>>>>> TrackMemory UnixSockets
> >>>>>>>> the support Forum https://discuss.kubernetes.io/
> >>>>>>>> is probably the most indicate place for guidance
> >>>>>>
> >>>>>>> I thought it was obvious that it was not working because it was
> >>>> calling
> >>>>>> the
> >>>>>>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to
> >>>> me.
> >>>>>>> And since I'm well aware of the k8s community, I already posted
> >>>> questions
> >>>>>>> about this in the appropriate place, before I posted here.
> >>>>>>> What I was hoping to get here was some indication or thoughts on
> why
> >>>> a
> >>>>>>> process using Windows curl doesn't have a problem, but does have a
> >>>>>> problem
> >>>>>>> when using Cygwin Curl. This isn't likely something that Cygwin
> curl
> >>>> is
> >>>>>>> doing "wrong", it's just that it's doing something different.
> >>>>>>> If it matters, the following is an elided version of the resulting
> >>>> curl
> >>>>>>> command:
> >>>>>>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
> >>>>>> (windows/amd64)
> >>>>>>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
> >>>>>>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
> >>>>>>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
> >>>>>>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
> >>>>>>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
> >>>>>>>
> >>>>>>
> >>>>
> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
> >>>>>>> I can't tell from the logging what request body it sent. It's
> >>>> possible it
> >>>>>>> didn't send any.
> >>>>>>
> >>>>>> We can't tell, not knowing who you are, and from what you are
> posting,
> >>>>>> what may
> >>>>>> or may not be obvious to you, where you are seeing "Upgrade request
> >>>>>> required",
> >>>>>> or where that may be coming from.
> >>>>>> That you need diagnostic help indicates that, what may appear
> obvious
> >>>> to
> >>>>>> you,
> >>>>>> may not be the case, as it is often our assumptions which lead us
> >>>> astray,
> >>>>>> and we
> >>>>>> get daily proof that we are imperfect, which is why most of seek to
> >>>> talk
> >>>>>> over
> >>>>>> and explain our issues to inanimate objects or colleagues e.g. talk
> >>>> to the
> >>>>>> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks
> >>>> and a
> >>>>>> pelican awarded by former projects.)
> >>>>>>
> >>>>>> For help with Cygwin, we need to see *whole* commands and all the
> >>>> output
> >>>>>> between
> >>>>>> the shell prompts, preferably with context, in this case including
> >>>> PATH
> >>>>>> under
> >>>>>> Cygwin and MS Windows and/or program paths typed or invoked.
> >>>>>>
> >>>>>> You may retain privacy and security by substituting variables e.g.
> env
> >>>>>> vars $VAR
> >>>>>> to sanitize sensitive information: to avoid problems I use a script
> >>>> to do
> >>>>>> so
> >>>>>> that logs are never written containing sensitive content.
> >>>>>>
> >>>>>
> >>>>> Acknowledged. The "Upgrade request required" is coming from the
> >>>> kubernetes
> >>>>> server.  At this point, I think I'm going to need to understand
> exactly
> >>>>> what that message means (other threads talking about this didn't make
> >>>> that
> >>>>> clear). Someone in that community did respond to my question, not
> with
> >>>> an
> >>>>> answer, but with a query for more information. I'll see what I can
> >>>> find out.
> >>>>
> >>>> A short Google search turned up that the message is apparently an HTTP
> >>>> 400 Bad
> >>>> Request response with Upgrade request message, meaning you need to use
> >>>> the
> >>>> websocket API to talk to a kubernetes pod:
> >>>>
> >>>>
> >>>>
> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
> >>>> https://tools.ietf.org/html/rfc6455
> >>>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101
> >>>
> >>>
> >>> Yes, I saw that thread in my research on this, but note that that
> >>> conclusion about using WebSocket is only from the first sentence of the
> >>> answer, whereas by the end of the answer, including the comment, it's
> clear
> >>> that this should be mitigated by just using "kubectl exec", which is
> what
> >>> I'm doing. In any case, it appears that even kubectl uses curl under
> the
> >>> covers. I would assume that kubectl attempts to mitigate the issues
> around
> >>> not using WebSocket for this, but it seems like it's not working at
> least
> >>> in my case.
> >>>
> >>> Another thought I had is whether I can coerce kubectl to use the
> Windows
> >>> curl by setting the PATH just before calling kubectl.  This didn't
> help. Is
> >>> there possibly something happening under the covers that would still
> make
> >>> this execute the Cygwin curl?
> >>>
> >>
> >> I have managed to resolve this.  The challenge here is finding the
> correct
> >> version of kubectl to use for the corresponding cluster version.  The
> rule
> >> is that you have to use a version of kubectl within one minor version of
> >> the cluster version.  We're using version 1.13.5 in the cluster.  I
> >> originally saw this problem with version 1.17, then moved to 1.18, then
> to
> >> 1.13.5.  Finally, version 1.14.0 resolves the problem. Version 1.13.5
> >> should have worked.  I don't know why it didn't.  I also have no idea
> why
> >> it would work in a non-cygwin shell.
> >>
> >>
> > Actually, I was wrong about this being resolved. For some reason it was
> > working properly for a short period after I installed v1.14.0, but now it
> > is consistently failing with "Upgrade request required" again.
> >
> > I've verified that the resulting kubectl call works fine from a cmd
> window.
> >
> > Another thing that I didn't mention before is that I run a Linux VM on
> this
> > laptop, and I can run the same scripting layer, albeit with a Linux
> install
> > of kubectl, and it works perfectly fine.
>
> Looks like curl has to send an Upgrade and other headers:
> https://gist.github.com/htp/fbce19069187ec1cc486b594104f01d0
> which kubectl exec should supply.
>
> Maybe time to start looking at what is passing over the net at both ends,
> as
> well as from the middle.
>

I've managed to resolve this.  The key is the proxy, as you indicate here.
My first real indication that this was the problem was that I started to
see this happen from my LInux VM, just this morning.  It's very curious
that this hasn't been happening all along. However, there have been some
global changes in the firewall recently, but that mostly consisted of
retiring an obsolete proxy and setting to use a new one, which I've been
configured to use for quite a while.

My solution is simply to not set the proxy in my shell environment
variables.  This won't really be a problem, as anything that requires an
external connection has tool configurations. I'm now fully working again
both in Cygwin and Linux VM.

Our theory about why this was having trouble is that using WebSocket
results in a direct connection from the server to an IP address on the
client. I have a feeling the proxy doesn't like that.
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -