X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 489C3389851F
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1592411970;
	bh=LBEUt04MaBwsB4XQOEBZgheC7fP6H+PeLiYpa9N4OVQ=;
	h=References:In-Reply-To:Date:Subject:To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	From;
	b=spbZywjj54vg8wJcLBd5D3Ti9gsQgujmX4QUFeVLbcVO5kmANRTd8uobg7D7KDR2C
	yOogPdtZk1WzhoPnv4TP+rlYREFmTCRTVnwx+c+G2y54I++YHFv/ZTusbhE2JJCATr
	mO1ltBk0fLmLbiygE68CMCAQNqyXrg2H1zimFrFk=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.3.2 sourceware.org A9CDA3890409
X-Google-DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20161025;
	h=x-gm-message-state:mime-version:references:in-reply-to:from:date
	:message-id:subject:to;
	bh=L7eDHEGqZTKWdDiLAqWD3faanMpTQroM2Fiu6vHN8ac=;
	b=XIYeYzB25KwXmAv/U1wng6ByNw1LiXSMOqbtNzdbxJzrnj1Nteb8AXJBNHtHTwjLyN
	v4vdj43DMi0V5YZDLGoqL0qdR7wogl5VsSpsexKEzWH+cGX02rXec5BlGyez17lUzI9r
	80e5AI73GFThfOpy50+6FEr683ryeyNfTs5h3P87zPE+i7NcYWyZe0vcT0o62JniDOO/
	hGGGtlBK8zGia1/lr3ev+tmw/gRbw8n56iMEiPZdisVLXsAxXRaUvHQC8em28Dg5JiJc
	nrw40eQoqNUbbHTv2+xPooiSV5iFca74GuL+xq+EoLc6ZNuK/RWPNFNfq9kxC4T9EZgb
	CgcQ==
X-Gm-Message-State:	AOAM530X+028471STqO3xzZmDf1OIA5GZa9qJVE1q4gJg1LicX3xO9uC
	cJRT6wIWtH8x+M9rPRwNHrzTiTxY6niCgmC9n7e67qNZ
X-Google-Smtp-Source:	ABdhPJxchp44dbu8EFCXwJLyuLnTzaZ90oqsDwGHACKOZW4tjTnO5qSnjaqc/KCXBRfutaAoi3de2/fGSEGKFaflna8=
X-Received:	by 2002:a05:6808:4b:: with SMTP id
	v11mr7804867oic.31.1592411964513;
	Wed, 17 Jun 2020 09:39:24 -0700 (PDT)
MIME-Version:	1.0
References:	<CAA5t8VqpU2KiQ_Kk5czjLzeCDzf21Gb2iBLu25xB43ZhpfmaPQ AT mail DOT gmail DOT com>
	<b68e371b-4e3d-3ab3-9a0e-2461697d814d AT gmail DOT com>
	<CAA5t8Vpym95FDhvanJSNd910kWssJqK46ZzmF2jckQbY_7YxOw AT mail DOT gmail DOT com>
	<f224a4a9-ea7d-e22c-0285-19d09ebd572f AT gmail DOT com>
	<CAA5t8Vpanxxop9THkbod115y_Xd2Q2QvYDXvBYOA7N9BqQw-Bw AT mail DOT gmail DOT com>
	<70ec2ece-c85e-c122-b4a0-3089da5c8a2c AT SystematicSw DOT ab DOT ca>
	<CAA5t8Vo=bVo9n-PHMrJW00HJ1LzeJxAaHZb4KOujnb+gyh9GKg AT mail DOT gmail DOT com>
	<3de6e80d-5ba5-3d12-f527-97b8ff3eaae7 AT SystematicSw DOT ab DOT ca>
	<CAA5t8Vo6PdHvSGwd49cBtD09QWOThiiHnwsMOkpVFV9-XG-h9w AT mail DOT gmail DOT com>
	<CAA5t8Vrao2csnVGNy4p396+nDBiU0eJGcsz=SV7WrUGYPwXfsw AT mail DOT gmail DOT com>
In-Reply-To:	<CAA5t8Vrao2csnVGNy4p396+nDBiU0eJGcsz=SV7WrUGYPwXfsw@mail.gmail.com>
Date:	Wed, 17 Jun 2020 09:39:13 -0700
Message-ID:	<CAA5t8Vo8AcKg84acBhGefyO7WFR_Bwu_xLadDhPeUd4dnq1Z7Q@mail.gmail.com>
Subject:	Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not
	in cmd shell
To:	The Cygwin Mailing List <cygwin AT cygwin DOT com>
X-Spam-Status:	No, score=-1.7 required=5.0 tests=BAYES_00, DKIM_SIGNED,
	DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, HTML_MESSAGE,
	RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS,
	TXREP autolearn=ham autolearn_force=no version=3.4.2
X-Spam-Checker-Version:	SpamAssassin 3.4.2 (2018-09-13) on
	server2.sourceware.org
X-Content-Filtered-By:	Mailman/MimeDel 2.1.29
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.29
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<http://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	David Karr via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	David Karr <davidmichaelkarr AT gmail DOT com>
Sender:	"Cygwin" <cygwin-bounces AT cygwin DOT com>

On Mon, Jun 15, 2020 at 9:10 AM David Karr wrote:

>
> On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote:
>
>>
>> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis <
>> Brian DOT Inglis AT systematicsw DOT ab DOT ca> wrote:
>>
>>> On 2020-06-14 12:16, David Karr via Cygwin wrote:
>>> > On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis <
>>> > Brian DOT Inglis AT systematicsw DOT ab DOT ca> wrote:
>>> >
>>> >> On 2020-06-14 09:38, David Karr via Cygwin wrote:
>>> >>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
>>> >>>> On 14.06.2020 08:12, David Karr wrote:
>>> >>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>>> >>>>>>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
>>> >>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a
>>> while,
>>> >>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl
>>> >>>>>>> exec" to open a shell in a container or directly execute a shell
>>> >>>>>>> command.
>>> >>>>>>> This has worked perfectly fine for a long time.
>>> >>>>>>> A couple of days ago, I discovered that all of these attempts
>>> were
>>> >>>>>>> failing with "Upgrade request required".  I hadn't upgraded
>>> kubectl
>>> >>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s
>>> upgrade,
>>> >>>>>>> but it's entirely possible.
>>> >>>>>>> A colleague of mine has a very similar desktop configuration
>>> >>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
>>> >>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it
>>> shows
>>> >>>>>>> the resulting "curl" command that it runs. I tried that resulting
>>> >>>>>>> command, and it results in the same response. I then tried
>>> updating
>>> >>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
>>> >> entire resulting "kubectl exec" command line and ran
>>> >>>>>>> it in a "cmd" shell.  No problem at all.  No error.
>>> >>>>>>> I know I haven't provided much useful information yet. I wanted
>>> to
>>> >>>>>>> get an initial response before I started providing those
>>> diagnostics.
>>> >>>>>>> Is there a clear issue here that I'm not aware of?
>>> >>
>>> >>>>> from where is kubectl coming from ?
>>> >>>>> In cygwin I found only a kubectl.py in the ansible package
>>> >>
>>> >>>>>> It's from here:
>>> >>>>>>
>>> >>
>>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
>>> >>
>>> >>>> so it is NOT a cygwin program.
>>> >>>> If the warning is coming about curl, it is likely
>>> >>>> that using from cygwin you are using the cygwin curl
>>> >>>> and from CMD the windows one
>>> >>>> $ which -a curl
>>> >>>> /usr/bin/curl
>>> >>>> /cygdrive/c/WINDOWS/system32/curl
>>> >>>> $ /cygdrive/c/WINDOWS/system32/curl -V
>>> >>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
>>> >>>> Release-Date: 2017-11-14, security patched: 2019-11-05
>>> >>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
>>> >> smtps
>>> >>>> telnet tftp
>>> >>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>>> >>>>
>>> >>>> $ /usr/bin/curl -V
>>> >>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f
>>> zlib/1.2.11
>>> >>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
>>> >>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
>>> >>>> Release-Date: 2019-09-11
>>> >>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap
>>> ldaps
>>> >>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
>>> >>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
>>> >>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
>>> >>>> TrackMemory UnixSockets
>>> >>>> the support Forum https://discuss.kubernetes.io/
>>> >>>> is probably the most indicate place for guidance
>>> >>
>>> >>> I thought it was obvious that it was not working because it was
>>> calling
>>> >> the
>>> >>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to
>>> me.
>>> >>> And since I'm well aware of the k8s community, I already posted
>>> questions
>>> >>> about this in the appropriate place, before I posted here.
>>> >>> What I was hoping to get here was some indication or thoughts on why
>>> a
>>> >>> process using Windows curl doesn't have a problem, but does have a
>>> >> problem
>>> >>> when using Cygwin Curl. This isn't likely something that Cygwin curl
>>> is
>>> >>> doing "wrong", it's just that it's doing something different.
>>> >>> If it matters, the following is an elided version of the resulting
>>> curl
>>> >>> command:
>>> >>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
>>> >> (windows/amd64)
>>> >>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
>>> >>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
>>> >>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
>>> >>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
>>> >>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
>>> >>>
>>> >>
>>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
>>> >>> I can't tell from the logging what request body it sent. It's
>>> possible it
>>> >>> didn't send any.
>>> >>
>>> >> We can't tell, not knowing who you are, and from what you are posting,
>>> >> what may
>>> >> or may not be obvious to you, where you are seeing "Upgrade request
>>> >> required",
>>> >> or where that may be coming from.
>>> >> That you need diagnostic help indicates that, what may appear obvious
>>> to
>>> >> you,
>>> >> may not be the case, as it is often our assumptions which lead us
>>> astray,
>>> >> and we
>>> >> get daily proof that we are imperfect, which is why most of seek to
>>> talk
>>> >> over
>>> >> and explain our issues to inanimate objects or colleagues e.g. talk
>>> to the
>>> >> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks
>>> and a
>>> >> pelican awarded by former projects.)
>>> >>
>>> >> For help with Cygwin, we need to see *whole* commands and all the
>>> output
>>> >> between
>>> >> the shell prompts, preferably with context, in this case including
>>> PATH
>>> >> under
>>> >> Cygwin and MS Windows and/or program paths typed or invoked.
>>> >>
>>> >> You may retain privacy and security by substituting variables e.g. env
>>> >> vars $VAR
>>> >> to sanitize sensitive information: to avoid problems I use a script
>>> to do
>>> >> so
>>> >> that logs are never written containing sensitive content.
>>> >>
>>> >
>>> > Acknowledged. The "Upgrade request required" is coming from the
>>> kubernetes
>>> > server.  At this point, I think I'm going to need to understand exactly
>>> > what that message means (other threads talking about this didn't make
>>> that
>>> > clear). Someone in that community did respond to my question, not with
>>> an
>>> > answer, but with a query for more information. I'll see what I can
>>> find out.
>>>
>>> A short Google search turned up that the message is apparently an HTTP
>>> 400 Bad
>>> Request response with Upgrade request message, meaning you need to use
>>> the
>>> websocket API to talk to a kubernetes pod:
>>>
>>>
>>> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
>>> https://tools.ietf.org/html/rfc6455
>>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101
>>
>>
>> Yes, I saw that thread in my research on this, but note that that
>> conclusion about using WebSocket is only from the first sentence of the
>> answer, whereas by the end of the answer, including the comment, it's clear
>> that this should be mitigated by just using "kubectl exec", which is what
>> I'm doing. In any case, it appears that even kubectl uses curl under the
>> covers. I would assume that kubectl attempts to mitigate the issues around
>> not using WebSocket for this, but it seems like it's not working at least
>> in my case.
>>
>> Another thought I had is whether I can coerce kubectl to use the Windows
>> curl by setting the PATH just before calling kubectl.  This didn't help. Is
>> there possibly something happening under the covers that would still make
>> this execute the Cygwin curl?
>>
>
> I have managed to resolve this.  The challenge here is finding the correct
> version of kubectl to use for the corresponding cluster version.  The rule
> is that you have to use a version of kubectl within one minor version of
> the cluster version.  We're using version 1.13.5 in the cluster.  I
> originally saw this problem with version 1.17, then moved to 1.18, then to
> 1.13.5.  Finally, version 1.14.0 resolves the problem. Version 1.13.5
> should have worked.  I don't know why it didn't.  I also have no idea why
> it would work in a non-cygwin shell.
>
>
Actually, I was wrong about this being resolved. For some reason it was
working properly for a short period after I installed v1.14.0, but now it
is consistently failing with "Upgrade request required" again.

I've verified that the resulting kubectl call works fine from a cmd window.

Another thing that I didn't mention before is that I run a Linux VM on this
laptop, and I can run the same scripting layer, albeit with a Linux install
of kubectl, and it works perfectly fine.
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -