X-Recipient: archive-cygwin AT delorie DOT com DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2BECB383E83C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com; s=default; t=1592237468; bh=S6lg6Fu+NXAu6d6oujtFZ3pUVlOdjnaw8GFe6O5h3VY=; h=References:In-Reply-To:Date:Subject:To:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=Nv5HYwviMqXHEUq0A1fOefCc5ItIRlUq/5x9H55V3kxnfv6fncmHHC/EbkuXIqac9 76vCQVtIWqTWI6s9x7K2ulBiocz28aJ6gMdZG5juTPiWfNAfTY0IAza+nI0YMTdJXl QJfk8N64gi5788lqU/K7EdNzitq4fZF9KvG0eBR8= X-Original-To: cygwin AT cygwin DOT com Delivered-To: cygwin AT cygwin DOT com DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org C26D4387088B X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=qByTC6zzi/6FGsU1HfO2Ihvf/mH0hwvTQoso6+bCdlw=; b=hGIzCkXvRe6iNmhweee8asOUIQJ0yMH4jNrwfmkLKzrVe8A9tJVr6UpabD+mophk8r IRkH/khcMRmXXR/ernWYXX272svyN6Ygpr+nesyyrLI4CrDjv0ivladA3Tk1qZ00ne2E I1ebFqp5RqaPkAlbfdWoTGSiu9joAPGxOzJC73WjW3DuBghRz/LKb/ZdTULOQnZc73I3 gjlxsKVdkVFlKQEXX1/pqfE8g5iMdgSJGgq/7U35d/Dx/fOFl+8/KKy0jTUvbmHfm3nJ ErW47XEdA/2NwbU4JtZihyj+crZIiHYoEy97R7AgAdAUq7MPNDRO+CTy46O9fiN1GkWF 76+g== X-Gm-Message-State: AOAM533ZDOLbtGnT00XzUc2C8BV0qv6hAOpd6gzlWm8fFoIqTEWy3NWv Vzxa2BCHJq3D0TXvS6d+GISPguBS6IzrLQ+Gjm3QLSfR X-Google-Smtp-Source: ABdhPJzROQ7ab5PDSM5n0p4gykOwsQIgwdEs3RJUr7ZXh7DNJLPrNmxbPpJPueJUEveXfPU9XRZmByfa0hUF+wxn1xY= X-Received: by 2002:a9d:6013:: with SMTP id h19mr23048218otj.50.1592237459890; Mon, 15 Jun 2020 09:10:59 -0700 (PDT) MIME-Version: 1.0 References:

<70ec2ece-c85e-c122-b4a0-3089da5c8a2c AT SystematicSw DOT ab DOT ca> <3de6e80d-5ba5-3d12-f527-97b8ff3eaae7 AT SystematicSw DOT ab DOT ca> In-Reply-To: Date: Mon, 15 Jun 2020 09:10:47 -0700 Message-ID: Subject: Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell To: The Cygwin Mailing List X-Spam-Status: No, score=-3.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, HTML_MESSAGE, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: cygwin AT cygwin DOT com X-Mailman-Version: 2.1.29 List-Id: General Cygwin discussions and problem reports List-Archive: List-Post: List-Help: List-Subscribe: , From: David Karr via Cygwin Reply-To: David Karr Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "Cygwin" On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote: > > On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis < > Brian DOT Inglis AT systematicsw DOT ab DOT ca> wrote: > >> On 2020-06-14 12:16, David Karr via Cygwin wrote: >> > On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis < >> > Brian DOT Inglis AT systematicsw DOT ab DOT ca> wrote: >> > >> >> On 2020-06-14 09:38, David Karr via Cygwin wrote: >> >>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote: >> >>>> On 14.06.2020 08:12, David Karr wrote: >> >>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote: >> >>>>>> On 13.06.2020 20:53, David Karr via Cygwin wrote: >> >>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a while, >> >>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl >> >>>>>>> exec" to open a shell in a container or directly execute a shell >> >>>>>>> command. >> >>>>>>> This has worked perfectly fine for a long time. >> >>>>>>> A couple of days ago, I discovered that all of these attempts were >> >>>>>>> failing with "Upgrade request required". I hadn't upgraded >> kubectl >> >>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s >> upgrade, >> >>>>>>> but it's entirely possible. >> >>>>>>> A colleague of mine has a very similar desktop configuration >> >>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom. >> >>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it >> shows >> >>>>>>> the resulting "curl" command that it runs. I tried that resulting >> >>>>>>> command, and it results in the same response. I then tried >> updating >> >>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the >> >> entire resulting "kubectl exec" command line and ran >> >>>>>>> it in a "cmd" shell. No problem at all. No error. >> >>>>>>> I know I haven't provided much useful information yet. I wanted to >> >>>>>>> get an initial response before I started providing those >> diagnostics. >> >>>>>>> Is there a clear issue here that I'm not aware of? >> >> >> >>>>> from where is kubectl coming from ? >> >>>>> In cygwin I found only a kubectl.py in the ansible package >> >> >> >>>>>> It's from here: >> >>>>>> >> >> >> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows >> >> >> >>>> so it is NOT a cygwin program. >> >>>> If the warning is coming about curl, it is likely >> >>>> that using from cygwin you are using the cygwin curl >> >>>> and from CMD the windows one >> >>>> $ which -a curl >> >>>> /usr/bin/curl >> >>>> /cygdrive/c/WINDOWS/system32/curl >> >>>> $ /cygdrive/c/WINDOWS/system32/curl -V >> >>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL >> >>>> Release-Date: 2017-11-14, security patched: 2019-11-05 >> >>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp >> >> smtps >> >>>> telnet tftp >> >>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL >> >>>> >> >>>> $ /usr/bin/curl -V >> >>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f >> zlib/1.2.11 >> >>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) >> >>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0 >> >>>> Release-Date: 2019-09-11 >> >>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps >> >>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp >> >>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6 >> >>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP >> >>>> TrackMemory UnixSockets >> >>>> the support Forum https://discuss.kubernetes.io/ >> >>>> is probably the most indicate place for guidance >> >> >> >>> I thought it was obvious that it was not working because it was >> calling >> >> the >> >>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to me. >> >>> And since I'm well aware of the k8s community, I already posted >> questions >> >>> about this in the appropriate place, before I posted here. >> >>> What I was hoping to get here was some indication or thoughts on why a >> >>> process using Windows curl doesn't have a problem, but does have a >> >> problem >> >>> when using Cygwin Curl. This isn't likely something that Cygwin curl >> is >> >>> doing "wrong", it's just that it's doing something different. >> >>> If it matters, the following is an elided version of the resulting >> curl >> >>> command: >> >>> curl -k -v -XPOST -H "User-Agent: kubectl.exe/v1.18.0 >> >> (windows/amd64) >> >>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H >> >>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H >> >>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H >> >>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H >> >>> "X-Stream-Protocol-Version: channel.k8s.io" 'https:// >> >>> >> >> >> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true' >> >>> I can't tell from the logging what request body it sent. It's >> possible it >> >>> didn't send any. >> >> >> >> We can't tell, not knowing who you are, and from what you are posting, >> >> what may >> >> or may not be obvious to you, where you are seeing "Upgrade request >> >> required", >> >> or where that may be coming from. >> >> That you need diagnostic help indicates that, what may appear obvious >> to >> >> you, >> >> may not be the case, as it is often our assumptions which lead us >> astray, >> >> and we >> >> get daily proof that we are imperfect, which is why most of seek to >> talk >> >> over >> >> and explain our issues to inanimate objects or colleagues e.g. talk to >> the >> >> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks >> and a >> >> pelican awarded by former projects.) >> >> >> >> For help with Cygwin, we need to see *whole* commands and all the >> output >> >> between >> >> the shell prompts, preferably with context, in this case including PATH >> >> under >> >> Cygwin and MS Windows and/or program paths typed or invoked. >> >> >> >> You may retain privacy and security by substituting variables e.g. env >> >> vars $VAR >> >> to sanitize sensitive information: to avoid problems I use a script to >> do >> >> so >> >> that logs are never written containing sensitive content. >> >> >> > >> > Acknowledged. The "Upgrade request required" is coming from the >> kubernetes >> > server. At this point, I think I'm going to need to understand exactly >> > what that message means (other threads talking about this didn't make >> that >> > clear). Someone in that community did respond to my question, not with >> an >> > answer, but with a query for more information. I'll see what I can find >> out. >> >> A short Google search turned up that the message is apparently an HTTP >> 400 Bad >> Request response with Upgrade request message, meaning you need to use the >> websocket API to talk to a kubernetes pod: >> >> >> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required >> https://tools.ietf.org/html/rfc6455 >> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101 > > > Yes, I saw that thread in my research on this, but note that that > conclusion about using WebSocket is only from the first sentence of the > answer, whereas by the end of the answer, including the comment, it's clear > that this should be mitigated by just using "kubectl exec", which is what > I'm doing. In any case, it appears that even kubectl uses curl under the > covers. I would assume that kubectl attempts to mitigate the issues around > not using WebSocket for this, but it seems like it's not working at least > in my case. > > Another thought I had is whether I can coerce kubectl to use the Windows > curl by setting the PATH just before calling kubectl. This didn't help. Is > there possibly something happening under the covers that would still make > this execute the Cygwin curl? > I have managed to resolve this. The challenge here is finding the correct version of kubectl to use for the corresponding cluster version. The rule is that you have to use a version of kubectl within one minor version of the cluster version. We're using version 1.13.5 in the cluster. I originally saw this problem with version 1.17, then moved to 1.18, then to 1.13.5. Finally, version 1.14.0 resolves the problem. Version 1.13.5 should have worked. I don't know why it didn't. I also have no idea why it would work in a non-cygwin shell. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation: https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple