X-Recipient:	archive-cygwin AT delorie DOT com
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.3.2 sourceware.org E2868393C89A
Authentication-Results:	sourceware.org; dmarc=none (p=none dis=none)
	header.from=SystematicSw.ab.ca
Authentication-Results:	sourceware.org;
	spf=none smtp.mailfrom=brian DOT inglis AT systematicsw DOT ab DOT ca
X-Authority-Analysis:	v=2.3 cv=LKf9vKe9 c=1 sm=1 tr=0
	a=kiZT5GMN3KAWqtYcXc+/4Q==:117 a=kiZT5GMN3KAWqtYcXc+/4Q==:17
	a=IkcTkHD0fZMA:10 a=0MaIOF3bAAAA:8 a=uPZiAMpXAAAA:8 a=48vgC7mUAAAA:8
	a=pQs5aej7AAAA:8 a=J7xhV7lfAAAA:20 a=GwtC8ItWuLMr9VDuWoQA:9 a=QEXdDO2ut3YA:10
	a=q-wEALBr1heszoC4fXDA:22 a=w1C3t2QeGrPiZgrLijVG:22
Subject:	Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not
	in cmd shell
To:	cygwin AT cygwin DOT com
References:	<CAA5t8VqpU2KiQ_Kk5czjLzeCDzf21Gb2iBLu25xB43ZhpfmaPQ AT mail DOT gmail DOT com>
	<b68e371b-4e3d-3ab3-9a0e-2461697d814d AT gmail DOT com>
	<CAA5t8Vpym95FDhvanJSNd910kWssJqK46ZzmF2jckQbY_7YxOw AT mail DOT gmail DOT com>
	<f224a4a9-ea7d-e22c-0285-19d09ebd572f AT gmail DOT com>
	<CAA5t8Vpanxxop9THkbod115y_Xd2Q2QvYDXvBYOA7N9BqQw-Bw AT mail DOT gmail DOT com>
	<70ec2ece-c85e-c122-b4a0-3089da5c8a2c AT SystematicSw DOT ab DOT ca>
	<CAA5t8Vo=bVo9n-PHMrJW00HJ1LzeJxAaHZb4KOujnb+gyh9GKg AT mail DOT gmail DOT com>
	<3de6e80d-5ba5-3d12-f527-97b8ff3eaae7 AT SystematicSw DOT ab DOT ca>
	<CAA5t8Vo6PdHvSGwd49cBtD09QWOThiiHnwsMOkpVFV9-XG-h9w AT mail DOT gmail DOT com>
	<CAA5t8Vrao2csnVGNy4p396+nDBiU0eJGcsz=SV7WrUGYPwXfsw AT mail DOT gmail DOT com>
	<CAA5t8Vo8AcKg84acBhGefyO7WFR_Bwu_xLadDhPeUd4dnq1Z7Q AT mail DOT gmail DOT com>
From:	Brian Inglis <Brian DOT Inglis AT SystematicSw DOT ab DOT ca>
Autocrypt:	addr=Brian DOT Inglis AT SystematicSw DOT ab DOT ca; prefer-encrypt=mutual;
	keydata=
	mDMEXopx8xYJKwYBBAHaRw8BAQdAnCK0qv/xwUCCZQoA9BHRYpstERrspfT0NkUWQVuoePa0
	LkJyaWFuIEluZ2xpcyA8QnJpYW4uSW5nbGlzQFN5c3RlbWF0aWNTdy5hYi5jYT6IlgQTFggA
	PhYhBMM5/lbU970GBS2bZB62lxu92I8YBQJeinHzAhsDBQkJZgGABQsJCAcCBhUKCQgLAgQW
	AgMBAh4BAheAAAoJEB62lxu92I8Y0ioBAI8xrggNxziAVmr+Xm6nnyjoujMqWcq3oEhlYGAO
	WacZAQDFtdDx2koSVSoOmfaOyRTbIWSf9/Cjai29060fsmdsDLg4BF6KcfMSCisGAQQBl1UB
	BQEBB0Awv8kHI2PaEgViDqzbnoe8B9KMHoBZLS92HdC7ZPh8HQMBCAeIfgQYFggAJhYhBMM5
	/lbU970GBS2bZB62lxu92I8YBQJeinHzAhsMBQkJZgGAAAoJEB62lxu92I8YZwUBAJw/74rF
	IyaSsGI7ewCdCy88Lce/kdwX7zGwid+f8NZ3AQC/ezTFFi5obXnyMxZJN464nPXiggtT9gN5
	RSyTY8X+AQ==
Organization:	Systematic Software
Message-ID:	<d91c8335-77c8-c7b3-b10a-beec1ac2e016@SystematicSw.ab.ca>
Date:	Wed, 17 Jun 2020 18:27:34 -0600
User-Agent:	Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
	Thunderbird/68.9.0
MIME-Version:	1.0
In-Reply-To:	<CAA5t8Vo8AcKg84acBhGefyO7WFR_Bwu_xLadDhPeUd4dnq1Z7Q@mail.gmail.com>
X-CMAE-Envelope:	MS4wfLo9hgqDMYjHCr/g5aP9Jm2vjy2rSEdIi9der+Y/S2dtjshPRBPD3Rs16H92Qk0LtRoM8E2Q/z4oFHzkQxp1jkgR85SubL9xiFhzsSGYLg8Uds5IGkre
	Q7K96oyCaFCTT0c2teeQO8cbF6E9Wl3LIp770yYrahTmgl47CcAOQvwH4bTDDdHniQ7vL2y2FhuqIQ==
X-Spam-Status:	No, score=-11.3 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS,
	KAM_LAZY_DOMAIN_SECURITY, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE,
	TXREP autolearn=no autolearn_force=no version=3.4.2
X-Spam-Checker-Version:	SpamAssassin 3.4.2 (2018-09-13) on
	server2.sourceware.org
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.29
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe:	<http://cygwin.com/mailman/options/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<http://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Reply-To:	cygwin AT cygwin DOT com
Errors-To:	cygwin-bounces AT cygwin DOT com
Sender:	"Cygwin" <cygwin-bounces AT cygwin DOT com>

On 2020-06-17 10:39, David Karr via Cygwin wrote:
> On Mon, Jun 15, 2020 at 9:10 AM David Karr wrote:
>> On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote:
>>> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis wrote:
>>>> On 2020-06-14 12:16, David Karr via Cygwin wrote:
>>>>> On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis wrote:
>>>>>> On 2020-06-14 09:38, David Karr via Cygwin wrote:
>>>>>>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
>>>>>>>> On 14.06.2020 08:12, David Karr wrote:
>>>>>>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>>>>>>>>>> On 13.06.2020 20:53, David Karr via Cygwin wrote:
>>>>>>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a
>>>> while,
>>>>>>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl
>>>>>>>>>>> exec" to open a shell in a container or directly execute a shell
>>>>>>>>>>> command.
>>>>>>>>>>> This has worked perfectly fine for a long time.
>>>>>>>>>>> A couple of days ago, I discovered that all of these attempts
>>>> were
>>>>>>>>>>> failing with "Upgrade request required".  I hadn't upgraded
>>>> kubectl
>>>>>>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s
>>>> upgrade,
>>>>>>>>>>> but it's entirely possible.
>>>>>>>>>>> A colleague of mine has a very similar desktop configuration
>>>>>>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
>>>>>>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it
>>>> shows
>>>>>>>>>>> the resulting "curl" command that it runs. I tried that resulting
>>>>>>>>>>> command, and it results in the same response. I then tried
>>>> updating
>>>>>>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
>>>>>> entire resulting "kubectl exec" command line and ran
>>>>>>>>>>> it in a "cmd" shell.  No problem at all.  No error.
>>>>>>>>>>> I know I haven't provided much useful information yet. I wanted
>>>> to
>>>>>>>>>>> get an initial response before I started providing those
>>>> diagnostics.
>>>>>>>>>>> Is there a clear issue here that I'm not aware of?
>>>>>>
>>>>>>>>> from where is kubectl coming from ?
>>>>>>>>> In cygwin I found only a kubectl.py in the ansible package
>>>>>>
>>>>>>>>>> It's from here:
>>>>>>>>>>
>>>>>>
>>>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
>>>>>>
>>>>>>>> so it is NOT a cygwin program.
>>>>>>>> If the warning is coming about curl, it is likely
>>>>>>>> that using from cygwin you are using the cygwin curl
>>>>>>>> and from CMD the windows one
>>>>>>>> $ which -a curl
>>>>>>>> /usr/bin/curl
>>>>>>>> /cygdrive/c/WINDOWS/system32/curl
>>>>>>>> $ /cygdrive/c/WINDOWS/system32/curl -V
>>>>>>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
>>>>>>>> Release-Date: 2017-11-14, security patched: 2019-11-05
>>>>>>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
>>>>>> smtps
>>>>>>>> telnet tftp
>>>>>>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>>>>>>>>
>>>>>>>> $ /usr/bin/curl -V
>>>>>>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f
>>>> zlib/1.2.11
>>>>>>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
>>>>>>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
>>>>>>>> Release-Date: 2019-09-11
>>>>>>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap
>>>> ldaps
>>>>>>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
>>>>>>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
>>>>>>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
>>>>>>>> TrackMemory UnixSockets
>>>>>>>> the support Forum https://discuss.kubernetes.io/
>>>>>>>> is probably the most indicate place for guidance
>>>>>>
>>>>>>> I thought it was obvious that it was not working because it was
>>>> calling
>>>>>> the
>>>>>>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to
>>>> me.
>>>>>>> And since I'm well aware of the k8s community, I already posted
>>>> questions
>>>>>>> about this in the appropriate place, before I posted here.
>>>>>>> What I was hoping to get here was some indication or thoughts on why
>>>> a
>>>>>>> process using Windows curl doesn't have a problem, but does have a
>>>>>> problem
>>>>>>> when using Cygwin Curl. This isn't likely something that Cygwin curl
>>>> is
>>>>>>> doing "wrong", it's just that it's doing something different.
>>>>>>> If it matters, the following is an elided version of the resulting
>>>> curl
>>>>>>> command:
>>>>>>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
>>>>>> (windows/amd64)
>>>>>>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
>>>>>>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
>>>>>>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
>>>>>>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
>>>>>>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
>>>>>>>
>>>>>>
>>>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
>>>>>>> I can't tell from the logging what request body it sent. It's
>>>> possible it
>>>>>>> didn't send any.
>>>>>>
>>>>>> We can't tell, not knowing who you are, and from what you are posting,
>>>>>> what may
>>>>>> or may not be obvious to you, where you are seeing "Upgrade request
>>>>>> required",
>>>>>> or where that may be coming from.
>>>>>> That you need diagnostic help indicates that, what may appear obvious
>>>> to
>>>>>> you,
>>>>>> may not be the case, as it is often our assumptions which lead us
>>>> astray,
>>>>>> and we
>>>>>> get daily proof that we are imperfect, which is why most of seek to
>>>> talk
>>>>>> over
>>>>>> and explain our issues to inanimate objects or colleagues e.g. talk
>>>> to the
>>>>>> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks
>>>> and a
>>>>>> pelican awarded by former projects.)
>>>>>>
>>>>>> For help with Cygwin, we need to see *whole* commands and all the
>>>> output
>>>>>> between
>>>>>> the shell prompts, preferably with context, in this case including
>>>> PATH
>>>>>> under
>>>>>> Cygwin and MS Windows and/or program paths typed or invoked.
>>>>>>
>>>>>> You may retain privacy and security by substituting variables e.g. env
>>>>>> vars $VAR
>>>>>> to sanitize sensitive information: to avoid problems I use a script
>>>> to do
>>>>>> so
>>>>>> that logs are never written containing sensitive content.
>>>>>>
>>>>>
>>>>> Acknowledged. The "Upgrade request required" is coming from the
>>>> kubernetes
>>>>> server.  At this point, I think I'm going to need to understand exactly
>>>>> what that message means (other threads talking about this didn't make
>>>> that
>>>>> clear). Someone in that community did respond to my question, not with
>>>> an
>>>>> answer, but with a query for more information. I'll see what I can
>>>> find out.
>>>>
>>>> A short Google search turned up that the message is apparently an HTTP
>>>> 400 Bad
>>>> Request response with Upgrade request message, meaning you need to use
>>>> the
>>>> websocket API to talk to a kubernetes pod:
>>>>
>>>>
>>>> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
>>>> https://tools.ietf.org/html/rfc6455
>>>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101
>>>
>>>
>>> Yes, I saw that thread in my research on this, but note that that
>>> conclusion about using WebSocket is only from the first sentence of the
>>> answer, whereas by the end of the answer, including the comment, it's clear
>>> that this should be mitigated by just using "kubectl exec", which is what
>>> I'm doing. In any case, it appears that even kubectl uses curl under the
>>> covers. I would assume that kubectl attempts to mitigate the issues around
>>> not using WebSocket for this, but it seems like it's not working at least
>>> in my case.
>>>
>>> Another thought I had is whether I can coerce kubectl to use the Windows
>>> curl by setting the PATH just before calling kubectl.  This didn't help. Is
>>> there possibly something happening under the covers that would still make
>>> this execute the Cygwin curl?
>>>
>>
>> I have managed to resolve this.  The challenge here is finding the correct
>> version of kubectl to use for the corresponding cluster version.  The rule
>> is that you have to use a version of kubectl within one minor version of
>> the cluster version.  We're using version 1.13.5 in the cluster.  I
>> originally saw this problem with version 1.17, then moved to 1.18, then to
>> 1.13.5.  Finally, version 1.14.0 resolves the problem. Version 1.13.5
>> should have worked.  I don't know why it didn't.  I also have no idea why
>> it would work in a non-cygwin shell.
>>
>>
> Actually, I was wrong about this being resolved. For some reason it was
> working properly for a short period after I installed v1.14.0, but now it
> is consistently failing with "Upgrade request required" again.
> 
> I've verified that the resulting kubectl call works fine from a cmd window.
> 
> Another thing that I didn't mention before is that I run a Linux VM on this
> laptop, and I can run the same scripting layer, albeit with a Linux install
> of kubectl, and it works perfectly fine.

Looks like curl has to send an Upgrade and other headers:
https://gist.github.com/htp/fbce19069187ec1cc486b594104f01d0
which kubectl exec should supply.

Maybe time to start looking at what is passing over the net at both ends, as
well as from the middle.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -