X-Recipient:	archive-cygwin AT delorie DOT com
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.3.2 sourceware.org 574733858004
Authentication-Results:	sourceware.org;
	dmarc=none (p=none dis=none) header.from=pdinc.us
Authentication-Results:	sourceware.org; spf=pass smtp.mailfrom=jpyeron AT pdinc DOT us
DKIM-Filter:	OpenDKIM Filter v2.11.0 mail2.pdinc.us 14KNCLAO024850
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=pdinc.us; s=default;
	t=1621552341; bh=gyhVXTtTxtaEtCK14dyLPYudoULnQW2k0CVovJvqKMo=;
	h=From:To:Cc:References:In-Reply-To:Subject:Date:From;
	b=UOGwwq89TTRvb+RcQpMf3RBRvrxzaq0YIhNgruX7uadFQe63vSo+Snqxg564c3yTX
	J6wFlKHdzdHqkl0ObhoKW+Qtwsu1B12WZ+P2iKEuzRLFDitC2/4VRK6Cz3TkSi0Gek
	rbWKSaaLJpcbRhP5N6Jdr/f4EDVV5a3+awficLlILuSzeG73e5Sia7sJPL04jWJ24B
	BsD+YxNhbGupm2hiE2KjeOoRn+QwbDPrgv5iCxw/d2lUb8uCXiya1C9ymO2Hb77Fw2
	wqkNiUpfRF5G+o10pT+O+UOXt2ngcMTi+7WYRUT0WgGCT+hA2sjG/W66SciiDvYrw/
	8vC8RoUOd1L8w==
From:	"Jason Pyeron" <jpyeron AT pdinc DOT us>
To:	<cygwin AT cygwin DOT com>
References:	<cda40b2d-e10b-220c-ec08-ac3a1c48b36f AT gmail DOT com>
	<7bd9875a-691e-95fe-77ea-4c8ccd98a46d AT gmail DOT com>
	<42988d13958e4b298f649190c800bcf8 AT leidos DOT com>
	<18110643390 DOT 20210520233304 AT yandex DOT ru>
In-Reply-To:	<18110643390.20210520233304@yandex.ru>
Subject:	RE: [cygwin] Re: EXTERNAL: Re: sshd high cpu load
Date:	Thu, 20 May 2021 19:12:31 -0400
Organization:	PD Inc
Message-ID:	<0ce701d74dcd$9caff290$d60fd7b0$@pdinc.us>
MIME-Version:	1.0
X-Mailer:	Microsoft Outlook 16.0
Thread-Index:	AQIf+eGeldap6MbuDqPsT/bo3CCJkgGnqVCgAdPdincB9AleKaowKdsA
X-Spam-Status:	No, score=-1.8 required=5.0 tests=BAYES_00, DKIM_SIGNED,
	DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, KAM_INFOUSMEBIZ, SPF_HELO_PASS,
	SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.2
X-Spam-Checker-Version:	SpamAssassin 3.4.2 (2018-09-13) on
	server2.sourceware.org
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.29
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Cc:	support AT pdinc DOT us
Sender:	"Cygwin" <cygwin-bounces AT cygwin DOT com>

> -----Original Message-----
> From: Andrey Repin
> Sent: Thursday, May 20, 2021 4:33 PM
> To: Wells, Roger K. <ROGER DOT K DOT WELLS AT leidos DOT com>; cygwin AT cygwin DOT com
> Cc: Andrey Repin <anrdaemon AT yandex DOT ru>
> Subject: [cygwin] Re: EXTERNAL: Re: sshd high cpu load
> 
> Greetings, Wells, Roger K.!
> 
> >> On 5/19/2021 12:48 AM, A. Doggy wrote:
> >>>
> >>>
> >>> I am running cygwin openssh as a windows service. I have been doing
> >>> so for many years with out issue. Recently, I have been running into
> >>> an issue where it maxes out my cpu on any version newer than 8.4p1-1.
> >>> The solution is to downgrade to 8.4p1-1. My server machine is a dell
> >>> t330 running windows 10. I am not a business despite using business
> >>> grade hardware.I have tried both 20h2 and 21h1 but no luck. There are
> >>> no users signed in when the issues occur and occurs within minutes of
> >>> booting up. The only change from the default config is I have it
> >>> running on a nonstandard port. Any advice is welcome as I really
> >>> would like to upgrade to a newer version. Thanks
> 
> > I noticed your initial contact and tried to duplicate what you observed
> > to no avail.
> 
> https://cygwin.com/pipermail/cygwin/2021-April/248299.html
> 
> > I set up cygwin openssh as a windows service as you described and also
> > have been doing it this way for many years.
> > sshd.exe doesn't show any cpu load on task manager even after days (yes
> > it still works when I log in from another machine)
> > My system is a Lenovo Thinkpad-x240 running updated W10. Cygwin is at
> > 3.2.0(0.340/5/3)
> > and ssh is at OpenSSH_8.5p1, OpenSSL 1.1.1f  31 Mar 2020.
> > Let me know if you would like me to try something else.
> 
> Connect from remote machine to the usual shell prompt and force kill remote
> ssh process.
> The hung SSH session will cause full core CPU load.

If this test case works, I will debug the sshd.exe Sunday!!!! My team will be so happy.


This has been our workaround:

$FROM="XXXXX@$([System.Net.Dns]::GetHostByName($env:computerName).HostName)"
$TO="ZZZZ AT WWWW"
$process="sshd"
$cpu=90
$maindelay=60
$checkdelay=5
$logging=$false
$LOGFILE="D:\INC000008507666\INC000008507666.log"

while($true)
{
    $killedany=$false
    if ($logging) { echo "$(Get-Date):loop" >> $LOGFILE }
    $x=get-wmiobject Win32_PerfFormattedData_PerfProc_Process | Where-Object -FilterScript { $_.PercentProcessorTime -gt $cpu -and $_.Name.startswith($process) } | foreach { $_.IDProcess }

    if ($x -and $x.Count -gt 0)
    {
        if ($logging) { echo "$(Get-Date):now sleep, found: $x" >> $LOGFILE }
        for ($retry=$checkdelay; $retry -ge 1; $retry=$retry/2)
        {
            if ($logging) { echo "$(Get-Date):check again: $retry" >> $LOGFILE }
            Start-Sleep -s $retry;
            if ($logging) { echo "$(Get-Date):woke" >> $LOGFILE }
            $y=get-wmiobject Win32_PerfFormattedData_PerfProc_Process | Where-Object -FilterScript { $_.PercentProcessorTime -gt $cpu -and $_.Name.startswith($process) } | foreach { $_.IDProcess }
            if ($y -and $y.Count -gt 0)
            {
                if ($logging) { echo "$(Get-Date):still found: $y" >> $LOGFILE }
                $y | foreach {
                    $badpid=$_
                    if ($logging) { echo "$(Get-Date):to kill: $badpid" >> $LOGFILE }
                    $p = Get-Process -Id $badpid
                    try
                    {
                        Stop-Process -Id $badpid -Force -ErrorAction Stop
                        $killedany=$true
                        Send-MailMessage -Subject "INC000008507666: $($env:COMPUTERNAME) killed $($p.Path) with pid $($badpid)" -From "$FROM" -To "$TO"
                    }
                    catch
                    {
                        Get-Date >> "$LOGFILE"
                        $_.Exception.Message >> "$LOGFILE"
                        $_.Exception.ItemName >> "$LOGFILE"
                        $_ >> "$LOGFILE"
                        Get-Date >> "$LOGFILE"

                        Send-MailMessage -Subject "INC000008507666: $($env:COMPUTERNAME) broke, check logs" -From "$FROM" -To "$TO"
                    }
                }
            }
            else
            {
                if ($logging) { echo "$(Get-Date):oh well: $y" >> $LOGFILE }
            }
        }
        if ($killedany -and -not (Test-NetConnection -Port 22 -InformationLevel Quiet -ComputerName 127.0.0.1))
        {
            Start-Service sshd
        }
    }
    Start-Sleep $maindelay
}


--
Jason Pyeron  | Architect
PD Inc        |
10 w 24th St  |
Baltimore, MD |
 
.mil: jason DOT j DOT pyeron DOT ctr AT mail DOT mil
.com: jpyeron AT pdinc DOT us
tel : 202-741-9397





-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -