X-Recipient: archive-cygwin AT delorie DOT com
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 490F2385EC4E
Authentication-Results: sourceware.org; dmarc=none (p=none dis=none)
 header.from=SystematicSw.ab.ca
Authentication-Results: sourceware.org;
 spf=none smtp.mailfrom=brian DOT inglis AT systematicsw DOT ab DOT ca
X-Authority-Analysis: v=2.4 cv=bZHV7MDB c=1 sm=1 tr=0 ts=5f68d37a
 a=kiZT5GMN3KAWqtYcXc+/4Q==:117 a=kiZT5GMN3KAWqtYcXc+/4Q==:17
 a=IkcTkHD0fZMA:10 a=yMhMjlubAAAA:8 a=w_pzkKWiAAAA:8 a=Maudkj2rSRX00NJfr2cA:9
 a=QEXdDO2ut3YA:10 a=KyLo1vIQnU4A:10 a=sRI3_1zDfAgwuvI8zelB:22
Subject: Re: Slowness of Cygwin commands on Windows Server 2016
To: cygwin AT cygwin DOT com
References: <CAH8Ga39ji+3puQi1LBJevsyWTW5j=k9jW+yeNv3mnTTmOiU6Gw AT mail DOT gmail DOT com>
 <0f041eee-0219-eb16-8c8b-1eb02c57a469 AT cornell DOT edu>
From: Brian Inglis <Brian DOT Inglis AT SystematicSw DOT ab DOT ca>
Autocrypt: addr=Brian DOT Inglis AT SystematicSw DOT ab DOT ca; prefer-encrypt=mutual;
 keydata=
 mDMEXopx8xYJKwYBBAHaRw8BAQdAnCK0qv/xwUCCZQoA9BHRYpstERrspfT0NkUWQVuoePa0
 LkJyaWFuIEluZ2xpcyA8QnJpYW4uSW5nbGlzQFN5c3RlbWF0aWNTdy5hYi5jYT6IlgQTFggA
 PhYhBMM5/lbU970GBS2bZB62lxu92I8YBQJeinHzAhsDBQkJZgGABQsJCAcCBhUKCQgLAgQW
 AgMBAh4BAheAAAoJEB62lxu92I8Y0ioBAI8xrggNxziAVmr+Xm6nnyjoujMqWcq3oEhlYGAO
 WacZAQDFtdDx2koSVSoOmfaOyRTbIWSf9/Cjai29060fsmdsDLg4BF6KcfMSCisGAQQBl1UB
 BQEBB0Awv8kHI2PaEgViDqzbnoe8B9KMHoBZLS92HdC7ZPh8HQMBCAeIfgQYFggAJhYhBMM5
 /lbU970GBS2bZB62lxu92I8YBQJeinHzAhsMBQkJZgGAAAoJEB62lxu92I8YZwUBAJw/74rF
 IyaSsGI7ewCdCy88Lce/kdwX7zGwid+f8NZ3AQC/ezTFFi5obXnyMxZJN464nPXiggtT9gN5
 RSyTY8X+AQ==
Organization: Systematic Software
Message-ID: <e9c82235-862a-c34d-675d-c1e786de9a82@SystematicSw.ab.ca>
Date: Mon, 21 Sep 2020 10:23:21 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
 Thunderbird/68.12.0
MIME-Version: 1.0
In-Reply-To: <0f041eee-0219-eb16-8c8b-1eb02c57a469@cornell.edu>
Content-Language: en-CA
X-CMAE-Envelope: MS4xfH6nLayY4kaq5efL4z2n3F2SeB5XatRAKBi7VU+oVlpNdCMrDpVSPFUWJKHpVme7ixf+Z7DEBLZq2uRDSb7DCYUMeoq0HRGOfw80894NdSnQPpnnK+oy
 CCZT1xeJGRAsP+DzezVcMN+mzuL0URF2uHHA6qYSX8+6Q2ZFpVFCLwsv0jjsO4glArH/TmflIcrM2ahtKjU0pzT1E5Ab3VcJuf0=
X-Spam-Status: No, score=-5.1 required=5.0 tests=BAYES_00, BODY_8BITS,
 KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_DNSWL_LOW,
 SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Reply-To: cygwin AT cygwin DOT com
Content-Type: text/plain; charset="utf-8"
Errors-To: cygwin-bounces AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 08LGO96C018086

On 2020-09-21 09:20, Ken Brown via Cygwin wrote:
> On 9/21/2020 10:53 AM, Praveen Kumar Chandrakar via Cygwin wrote:
>> Hello,
>>
>> We are experiencing some issues with the Cygwin commands (ie. from bash
>> -login) taking several minutes on three of our Windows 2016 servers.
>>
>>      ls -al -- 1min 32sec
>>
>>      find  -- 4min 46sec
>>
>> When you search for a specific file, such as ls -al ./ABC/XYZ.txt, it is
>> very quick.
>>
>> We ran strace on the machines where it is quick and where it slow.
>>
>> Here is some annotation for not working and working excerpted from the
>> files:
>>
>> *=====> Not working Scenario*
>>
>> Command run: strace -m all -o does-not-work.txt find . -name VNM.OUT
>>
>> ===> Initially it hangs here for 15-25 seconds: samlib.dll
>>
>>    214  995796 [main] find 7768 symlink_info::check: 0 =
>> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
>> --- Process 7768 loaded C:\Windows\SysWOW64\samcli.dll at 73080000
>> --- Process 7768 loaded C:\Windows\SysWOW64\samlib.dll at 72c90000
>>   5435  423059 [main] find 7768 pwdgrp::fetch_account_from_windows: line:
>> <WMW2CAOC01+TempUser Users:S-1-5-21-858922594-1104986591-3574882633-1
>> 000:197608:>
>>
>> ===> Then the rest of the almost 4 minutes, it hangs at FWPUCLNT.DLL
>>
>>    214  995796 [main] find 7768 symlink_info::check: 0 =
>> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
>> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
>> --- Process 7768 thread 4596 exited with status 0x0
>> --- Process 7768 thread 6568 exited with status 0x0
>> --- Process 7768 thread 7040 exited with status 0x0
>> 82877524 83873956 [main] find 7768 pwdgrp::fetch_account_from_windows:
>> line: <sa-ca:*:1050640:1049089:U-TAS\sa-ca,S-1-5-21-3577821319-10804827
>> 17-3061384290-2064:/home/sa-ca:/bin/bash>
>>
>>
>> ===> You see the process threads created, and created again, then they
>> close after FWPUCLNT.DLL unfreezes (see above)
>>
>> drwx------    1 CKimball UsersGrp         0 Jul 28 23:13 ..
>>    303  995341 [main] find 7768 symlink_info::check: 0x0 = NtCreateFile
>> (\??\D:\Abc\Xyz\conf.txt)
>>    241  995582 [main] find 7768 symlink_info::check: not a symlink
>>    214  995796 [main] find 7768 symlink_info::check: 0 =
>> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
>> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
>> --- Process 7768 thread 4596 created
>> --- Process 7768 thread 6568 created
>> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
>> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
>> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
>>      6       6 [main] find (7768)
>> **********************************************
>>    505     511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
>> (windows pid 7768)
>>    203     714 [main] find (7768) OS version:   Windows NT-10.0
>>    183     897 [main] find (7768)
>> **********************************************
>> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
>> --- Process 7768 loaded C:\Windows\SysWOW64\msvcrt.dll at 753d0000
>> --- Process 7768 loaded C:\Windows\SysWOW64\sechost.dll at 75490000
>>    212  996008 [main] find 7768 path_conv::check:
>> this->path(D:\Abc\Xyz\conf.txt), has_acls(1)
>> --- Process 7768 thread 7040 created
>> --- Process 7768 thread 4596 created
>> --- Process 7768 loaded D:\Cygwin\bin\cyggcc_s-1.dll at 6c100000
>> --- Process 7768 thread 6568 created
>> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
>> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
>> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
>>      6       6 [main] find (7768)
>> **********************************************
>>    505     511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
>> (windows pid 7768)
>>    203     714 [main] find (7768) OS version:   Windows NT-10.0
>>    183     897 [main] find (7768)
>> **********************************************
>> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
>>
>>
>> *=====> Working Scenario*
>>
>> ===> Runs straight through to completion, no pauses/freezes
>>
>>    215  696807 [main] find 9456 build_fh_pc: fh 0x612AFFD0, dev 000000C3
>>    235  697042 [main] find 9456 stat_worker: (\??\F:\Abc\Xyz\conf.txt,
>> 0x200749F8, 0x612AFFD0), file_attributes 16
>> --- Process 9456 loaded C:\Windows\SysWOW64\rasadhlp.dll at 73190000
>> --- Process 9456 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 72fa0000
>> 161759  858801 [main] find 9456 pwdgrp::fetch_account_from_windows: line:
>> <_svcCASpecLAN:*:1319973:1049089:U-TSYS\_svcCASpecLAN,S-1-5-21-21232
>> 42984-354184167-320618023-271397:/home/_svcCASpecLAN:/bin/bash>
>>   1034  859835 [main] find 9456 fhandler_base::fstat_helper: 0 = fstat
>> (\??\F:\Abc\Xyz\conf.txt, 0x200749F8) st_size=0, st_mod
>> e=040775, st_ino=1688849860295204st_atim=5F210CE0.25975E6C
>> st_ctim=5F210CE0.25975E6C st_mtim=5F210CE0.25975E6C
>> st_birthtim=5F210CE0.16B102A4
>>    195  860030 [main] find 9456 stat_worker: 0 =
>> (\??\F:\Abc\Xyz\conf.txt,0x200749F8)
>>
>> All our other Windows 2016 servers do not experience any issues.
>>
>> Do you have any suggestions on finding out what is causing this?

May not be anything to do with Cygwin.
This DLL is part of the Windows Filtering Platform:

https://docs.microsoft.com/en-us/windows/win32/fwp/windows-filtering-platform-start-page

If your org is doing network content or traffic filtering, have those folks look
into what is happening here, what filters are being applied where, whether this
is hooked into your AV.
Also maybe check how your network interface(s) and subnet(s) hardware are
performing for these servers.

Cygwin may be able to help caching domain information to reduce network and
process load by installing and running cygserver with only default settings and
options:

https://cygwin.com/cygwin-ug-net/using-cygserver.html

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple