DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 512FA398600A
Subject: Re: Slowness of Cygwin commands on Windows Server 2016
To: Praveen Kumar Chandrakar <praveenkumar DOT chandrakar AT broadcom DOT com>,
        cygwin AT cygwin DOT com
References: <CAH8Ga39ji+3puQi1LBJevsyWTW5j=k9jW+yeNv3mnTTmOiU6Gw AT mail DOT gmail DOT com>
Message-ID: <0f041eee-0219-eb16-8c8b-1eb02c57a469@cornell.edu>
Date: Mon, 21 Sep 2020 11:20:54 -0400
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
 Thunderbird/68.12.0
In-Reply-To: <CAH8Ga39ji+3puQi1LBJevsyWTW5j=k9jW+yeNv3mnTTmOiU6Gw@mail.gmail.com>
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from [192.168.0.17] (68.175.129.7) by
 CH2PR14CA0038.namprd14.prod.outlook.com (2603:10b6:610:56::18) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.3391.11 via Frontend Transport; Mon, 21 Sep 2020 15:20:56 +0000
X-Originating-IP: [68.175.129.7]
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 5527fa20-d627-45da-ba37-08d85e41f015
X-MS-TrafficTypeDiagnostic: MN2PR04MB5773:
X-Microsoft-Antispam-PRVS: <MN2PR04MB5773F794FCF353671034A50FD83A0 AT MN2PR04MB5773 DOT namprd04 DOT prod DOT outlook DOT com>
X-MS-Oob-TLC-OOBClassifiers: OLM:7691;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: 90ddxg32/qd7Ib0DBgAW/b5RCVSNCl1Hzs/oE7LO1hcn5GRXxfDx12lAu+nVgbhiJHt+VwvrU78Kwtm+HNxuHRmxxdGy+C54IG6Vt6mCkhib8+kVYqJFAshkpcX/tvWo0wjK6xSW/kuVCs4zVLcvBKZpU9PMW1c5fxBYQn7MuVL1JanOZJIWHGZMPmq8P3MwKMiXWn+MpxsNyb69bgw69j5TiflbfoAOT0KB9dkGuqmssILYlROGPwElZWNf1HizSlyRnWv6OHOOB6fRKDxkw6qEZDVl7kTCWWFzhT+avrye9VaUbR/p/VLDC2VxyvA5W2uVFS427zOoFMMjw3oO1PhTkmTSV0n2MIxwWxAT2Cn2Dx8kiy3yPJWe8yympVqrM6yhHELAe0dh54+p9DqRA5PMxQk5rdQlSkNMmIN7eu/3UZC8LVfCvhsIiMMGhlTYbJFdY64HzIl5Qc5bwmFZEfNI1vRrdE1wzIzBbHbYBxc3P0vv7gU4+ZxNRSZn/7mHnFOY+A23140vYHZU/lUZZkvFAsGTEyzMIdTs3+2g/EY=
X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:MN2PR04MB6176.namprd04.prod.outlook.com; PTR:; CAT:NONE;
 SFS:(4636009)(366004)(376002)(346002)(396003)(39860400002)(136003)(966005)(86362001)(66946007)(66556008)(2906002)(66476007)(786003)(53546011)(6486002)(31696002)(75432002)(956004)(16576012)(4326008)(52116002)(26005)(316002)(2616005)(83380400001)(5660300002)(54906003)(186003)(478600001)(16526019)(31686004)(36756003)(8936002)(8676002)(43740500002)(460985005);
 DIR:OUT; SFP:1102; 
X-MS-Exchange-AntiSpam-MessageData: TQA4UDr4t8wYrIuRXDabcjzlOniGtbLv+U6P1M9aOVAdWGU1g80JwoLWnQI2jbEq0QMLfRsdTP/c4ANaM4cGbkitMQgvvtA92gL1zvPiu2ppP+rJBjoSriUOnn0CzE5Po8qFijtp0tAhBxWo3eCAOaa6VCJ2mKRj0xn2oz4ZVNuEQ/8Sy7Oy2NRVdSGkSI1xyAs2d7OZaYa9bsUobStA+Lt8OBHGWQHb6Ftp0IMpfCqrg21cG4QRc04Cv8e3k1t5CpLNQLH0CBo1uGWBelU9RrhLri6c/PwvNJn5OR1k51Wwx5bYX6HjaO7lRqZVMM16I/6gznbl8p8srQ4uV5To1JwtIuEG50Eb7D/RPmYehihfAxdM4q/Fvvn38sIYohTl2mZgUdZ6SzW9qg8szRZiRg+RfiU8tud6UwSNZb3ccJi7dPfJafhY31NIXw00brymbVfxZ7VKjgN2uKb/jOziDRu59acI5Kjry6ZG9yZi/gqL+wNqUtN1trbsS+3jaSiSZ9/jFPa+edZnnes96HaG3QZAHLxWH7upk7bdQ4Zkn1agNZrYLLZTB+bLk3n4isz1xvdpSJqCwIRFkgB8wOzBOt9zaQeHvoenv1UOUAXs4zTONy41+oLqcpAljlx0ElW1EAuavrQP2bfoMAUwzTSGRw==
X-OriginatorOrg: cornell.edu
X-MS-Exchange-CrossTenant-Network-Message-Id: 5527fa20-d627-45da-ba37-08d85e41f015
X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB6176.namprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Sep 2020 15:20:56.7815 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5qLGU3iz1x+JLD6xojegrcDTMsvlbML/GX0UaQcLrrZu8oNFdCX9hTwaKb5X11GO5rLgjntmixdHvkbx7mSgIg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5773
X-Spam-Status: No, score=-4.0 required=5.0 tests=BAYES_00, DKIM_SIGNED,
 DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, KAM_ASCII_DIVIDERS,
 MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,
 SPF_HELO_PASS, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Ken Brown via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Ken Brown <kbrown AT cornell DOT edu>
Cc: Jayakrishna Karicharla <jayakrishna DOT karicharla AT broadcom DOT com>,
        Narayanarao Yenduri <narayanarao DOT yenduri AT broadcom DOT com>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: cygwin-bounces AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces AT cygwin DOT com>

On 9/21/2020 10:53 AM, Praveen Kumar Chandrakar via Cygwin wrote:
> Hello,
> 
> We are experiencing some issues with the Cygwin commands (ie. from bash
> -login) taking several minutes on three of our Windows 2016 servers.
> 
>      ls -al -- 1min 32sec
> 
>      find  -- 4min 46sec
> 
> When you search for a specific file, such as ls -al ./ABC/XYZ.txt, it is
> very quick.
> 
> We ran strace on the machines where it is quick and where it slow.
> 
> Here is some annotation for not working and working excerpted from the
> files:
> 
> *=====> Not working Scenario*
> 
> Command run: strace -m all -o does-not-work.txt find . -name VNM.OUT
> 
> ===> Initially it hangs here for 15-25 seconds: samlib.dll
> 
>    214  995796 [main] find 7768 symlink_info::check: 0 =
> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
> --- Process 7768 loaded C:\Windows\SysWOW64\samcli.dll at 73080000
> --- Process 7768 loaded C:\Windows\SysWOW64\samlib.dll at 72c90000
>   5435  423059 [main] find 7768 pwdgrp::fetch_account_from_windows: line:
> <WMW2CAOC01+TempUser Users:S-1-5-21-858922594-1104986591-3574882633-1
> 000:197608:>
> 
> ===> Then the rest of the almost 4 minutes, it hangs at FWPUCLNT.DLL
> 
>    214  995796 [main] find 7768 symlink_info::check: 0 =
> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
> --- Process 7768 thread 4596 exited with status 0x0
> --- Process 7768 thread 6568 exited with status 0x0
> --- Process 7768 thread 7040 exited with status 0x0
> 82877524 83873956 [main] find 7768 pwdgrp::fetch_account_from_windows:
> line: <sa-ca:*:1050640:1049089:U-TAS\sa-ca,S-1-5-21-3577821319-10804827
> 17-3061384290-2064:/home/sa-ca:/bin/bash>
> 
> 
> ===> You see the process threads created, and created again, then they
> close after FWPUCLNT.DLL unfreezes (see above)
> 
> drwx------    1 CKimball UsersGrp         0 Jul 28 23:13 ..
>    303  995341 [main] find 7768 symlink_info::check: 0x0 = NtCreateFile
> (\??\D:\Abc\Xyz\conf.txt)
>    241  995582 [main] find 7768 symlink_info::check: not a symlink
>    214  995796 [main] find 7768 symlink_info::check: 0 =
> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
> --- Process 7768 thread 4596 created
> --- Process 7768 thread 6568 created
> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
>      6       6 [main] find (7768)
> **********************************************
>    505     511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
> (windows pid 7768)
>    203     714 [main] find (7768) OS version:   Windows NT-10.0
>    183     897 [main] find (7768)
> **********************************************
> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
> --- Process 7768 loaded C:\Windows\SysWOW64\msvcrt.dll at 753d0000
> --- Process 7768 loaded C:\Windows\SysWOW64\sechost.dll at 75490000
>    212  996008 [main] find 7768 path_conv::check:
> this->path(D:\Abc\Xyz\conf.txt), has_acls(1)
> --- Process 7768 thread 7040 created
> --- Process 7768 thread 4596 created
> --- Process 7768 loaded D:\Cygwin\bin\cyggcc_s-1.dll at 6c100000
> --- Process 7768 thread 6568 created
> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
>      6       6 [main] find (7768)
> **********************************************
>    505     511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
> (windows pid 7768)
>    203     714 [main] find (7768) OS version:   Windows NT-10.0
>    183     897 [main] find (7768)
> **********************************************
> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
> 
> 
> *=====> Working Scenario*
> 
> ===> Runs straight through to completion, no pauses/freezes
> 
>    215  696807 [main] find 9456 build_fh_pc: fh 0x612AFFD0, dev 000000C3
>    235  697042 [main] find 9456 stat_worker: (\??\F:\Abc\Xyz\conf.txt,
> 0x200749F8, 0x612AFFD0), file_attributes 16
> --- Process 9456 loaded C:\Windows\SysWOW64\rasadhlp.dll at 73190000
> --- Process 9456 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 72fa0000
> 161759  858801 [main] find 9456 pwdgrp::fetch_account_from_windows: line:
> <_svcCASpecLAN:*:1319973:1049089:U-TSYS\_svcCASpecLAN,S-1-5-21-21232
> 42984-354184167-320618023-271397:/home/_svcCASpecLAN:/bin/bash>
>   1034  859835 [main] find 9456 fhandler_base::fstat_helper: 0 = fstat
> (\??\F:\Abc\Xyz\conf.txt, 0x200749F8) st_size=0, st_mod
> e=040775, st_ino=1688849860295204st_atim=5F210CE0.25975E6C
> st_ctim=5F210CE0.25975E6C st_mtim=5F210CE0.25975E6C
> st_birthtim=5F210CE0.16B102A4
>    195  860030 [main] find 9456 stat_worker: 0 =
> (\??\F:\Abc\Xyz\conf.txt,0x200749F8)
> 
> All our other Windows 2016 servers do not experience any issues.
> 
> Do you have any suggestions on finding out what is causing this?

I can't give you a definitive answer, but it looks like you're running 32-bit 
Cygwin.  I suggest switching to 64-bit Cygwin.  Here's a quote from 
https://cygwin.com/ that might be relevant:

-----------------------------------------------------------------------------
32 bit Cygwin

Address space is a very limiting factor for Cygwin. These days, a full 32 bit 
Cygwin distro is not feasible anymore, and will in all likelihood fail in random 
places due to an issue with the fork(2) system call.

Therefore we recommend using 32 bit Cygwin only in limited scenarios, with only 
a minimum of necessary packages installed, and only if there's no way to run 64 
bit Cygwin instead.
-----------------------------------------------------------------------------

Ken
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple