X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 53A103857C6D
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1630016399;
	bh=ti3seqEJjiLOlCrallzuroqc/K+WatuPu1xyaO6qwxc=;
	h=Subject:To:References:Date:In-Reply-To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	From;
	b=yvQZKu8L319pJjut/ChMW660HewVVM4kv2eHpLXSlkmFnNY/5JMB15XqAJTe3BoPv
	LMLC4W8N4Ru5x73p3P+3fF/zUMM7eJC15mBHiPc5TG08q2mOD9/OVRFqkcgOgs062L
	7+pFbP4MlGy6yTdRAi5D0mNhIgQ9jAsHrHSpRBbA=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.1 sourceware.org 976443858D3C
ARC-Seal:	i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
	b=fOMtFDZeUSmAUvCDpUbv7lB/Mee8BSAVpY1B1c9W5DN4j2hqj2b3QItkXqIk/2+K8dgJqPUAs2hGheYZzNL4iSuccl7vg+ky2XEuUXh8k07n5hNxLw6UaME6O+CqMGGyRiVjdMeBSjOqBGXoDHJnbSu9MWPkokargCy2iCqG/FPAlf65uMPBDx4Iq62Ob1SVHtJZLis2XUZt1/xpWk6Fhm9RA+4YWEZINeCmDEdtmR4d1cFx9w0LQI0Ury165Achk/5EDXELKv0QsrDk8Svpu/YpgHYM76kbwRz52LR/k4vEdVj3SqIvk1aRm6mN9fVcnE7yOvBVMxRMaF8rp37HmQ==
ARC-Message-Signature:	i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
	s=arcselector9901;
	h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
	bh=ahkQIsMt3Vv7JBBBJvGjR95YlxeG5cpB3YAjkoHO3F4=;
	b=T5wp4qq/XC5YAaEsPWyjRU+vVR2310EyI+yhR7YnP9VemlrbM9eddKX9VUO1IsoaDOhhlQko0FYRWaV8d2L7AtAOVYK1rS+2vrnsNMVUNaG10HR7wB8G9Qj8aJKoFSsvlXrBK/zEiht1j/TWuH7VCBfCfVGVZRJc1Ju2x7qpgIkK/MU+uB0XWMWsgZd3AjmT9qS5zhnwg253R3T3FC/BjpPgF/5NWH2bENjoE0U8djk08D+p1Hd0vEPDHIvAnGLC9UiWbX3VpWhFW/AWHZ3/z3FgPi6heFEIfeyzpu5l8qyScWqXABWCXWha75fcOoxaLr/WVjnkOcps04YVgfKq+A==
ARC-Authentication-Results:	i=1; mx.microsoft.com 1; spf=pass
	smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu;
	dkim=pass header.d=cornell.edu; arc=none
Subject:	Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
To:	cygwin AT cygwin DOT com
References:	<41A583E1-C8E7-42AB-9F24-EEC33A41EC60 AT house DOT org>
	<20210825201845 DOT 07b6400b79dc5558a7761efe AT nifty DOT ne DOT jp>
	<f8106fe7-a2b8-d106-3061-4d888124f4b0 AT cornell DOT edu>
	<20210826062934 DOT 54f2f2216021c095bb7ba13b AT nifty DOT ne DOT jp>
	<d0a8c57d-1ed1-6b4f-c6e7-cbe0e2ec8a1c AT cornell DOT edu>
Message-ID:	<3b560051-ab27-f392-ca4b-d1fd9b5733b0@cornell.edu>
Date:	Thu, 26 Aug 2021 18:18:29 -0400
User-Agent:	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
	Thunderbird/78.13.0
In-Reply-To:	<d0a8c57d-1ed1-6b4f-c6e7-cbe0e2ec8a1c@cornell.edu>
X-ClientProxiedBy:	CH2PR03CA0029.namprd03.prod.outlook.com
	(2603:10b6:610:59::39) To BN7PR04MB4388.namprd04.prod.outlook.com
	(2603:10b6:406:f8::19)
MIME-Version:	1.0
X-MS-Exchange-MessageSentRepresentingType:	1
X-MS-PublicTrafficType:	Email
X-MS-Office365-Filtering-Correlation-Id:	3a006ab3-f0d5-4d40-3ea8-08d968df6fc6
X-MS-TrafficTypeDiagnostic:	BN8PR04MB5617:
X-Microsoft-Antispam-PRVS:	<BN8PR04MB5617F32498A901C3B404F9BAD8C79 AT BN8PR04MB5617 DOT namprd04 DOT prod DOT outlook DOT com>
X-MS-Oob-TLC-OOBClassifiers:	OLM:3968;
X-MS-Exchange-SenderADCheck:	1
X-MS-Exchange-AntiSpam-Relay:	0
X-Microsoft-Antispam:	BCL:0;
X-Microsoft-Antispam-Message-Info:	u7yIbNemN8zGvKV6xiV0CJ2gIeTurE01NUW8LOFbgUbIfsnIWaVqlq9AXN+avcDCTLZcUfNY/KnHaqMKvDeR0sMihEar+Nu60g2tXWOPTcVky6VA3jzP9amWbLLDs92Z+PcemZ0POSQa1hsRKn2B0nmZT5g957+39xnlKMwVY5+8laxu1rdVhT0RRDxtvbDlnPATMMQ7TG6AIVb2dIMb8vJiU9XQnswjnkhs+0SDeLnS9CmC18GIWnF0aNnDEXSs9Z2JFfrBtIie/UQyB4VRj3UWQJTySN6qvDbTX4Nq01Wdz3cbrwpGhu2uP9grVM4iDaPfgOuCBeQj0PMoBk96bQiej0dtiMk4YSUYHWb629/moMJ1ME758fBzdC53AbfxCJu1of37eEWljeE/A31WPtssqQD70mRqXhO5DqtIVNKNmFkpVVuySIlxbpuyJvSAs7PBQ3gLjFibmpMn0mA79aFrz3OuzoLNy2w0+fTANGhf5EPw+Ecy6aRTUSiRRZOgdbD3NGGTGXCsC4wPOQN50PxuSqoAhQDs7qyp7RHFVVGwd3uOUc1tpiocfJDAryGMZI8MoFI5YXn97Uh6tqgB63/y+itC/zr+m5jK+MMTWXNyYWOSqldGQbbdivd5B1W4CA2NQGzmYpIxoRrVJA94TKxgPiGR2ZgaU4AfFcVdlVVO1AtxpVOktm8FXT4vileONF8Q+9fXHWKymHxApJIYt/TRDLNJts0uakDH3lqLNwDvLWcbELPPqSYOj2yVHsnUBXKflnnpBoctG1IrGYiK0qYNJ2vcogqQvc+d85I8scyVcTjjyYrdRdEClo+8+75gLeMCKpFgUAK/e+HBm7uMfA==
X-Forefront-Antispam-Report:	CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
	IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE;
	SFS:(4636009)(136003)(396003)(366004)(39860400002)(346002)(376002)(86362001)(66946007)(31696002)(2906002)(6486002)(966005)(75432002)(5660300002)(31686004)(66476007)(53546011)(2616005)(38100700002)(66556008)(8676002)(478600001)(786003)(36756003)(316002)(8936002)(186003)(83380400001)(6916009)(45980500001)(43740500002)(460985005);
	DIR:OUT; SFP:1102;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount:	1
X-MS-Exchange-AntiSpam-MessageData-0:	=?Windows-1252?Q?np2NEFoHUGghnABBhkHPLM5ThEmIfK9TurQgC2wvE+7UDbZavp7ywc2B?=
	=?Windows-1252?Q?1HDlMoFJX7+bzCZK4aOtdcify/5YJoNqFc8nuI9UwsZm4uCf9XBhAepZ?=
	=?Windows-1252?Q?+9fzl5qYA1W4t3n811C99o2NQvHzrCoqjuwaaalKTTS94tWAgQMDihXv?=
	=?Windows-1252?Q?HEdnCkwxsyztVjzcv0FCjGYrQFL7fTfFiWdFuQOuBRoTKSRdkzBiV88D?=
	=?Windows-1252?Q?3BbCkxo/Eb1w6BplFvTz4etN7otKksBFnZDm9Ukw2NKMOBclW1BxRPr6?=
	=?Windows-1252?Q?nciiqI1INZAtkrbWpMakrCuTr2pVay5WgNQsYl0jiaGANIRnbwL8wQpV?=
	=?Windows-1252?Q?j7FnmXXKkRF53L4+x0bx8M5SBa64+eQTzEY087lw7XlmaTJODUqxn89j?=
	=?Windows-1252?Q?RzRGAXZIBmGDoo1x12R4RV7Ry/J1HFSBIcQFeIGtEx4NSzjSBvbpiNqd?=
	=?Windows-1252?Q?OAztgSB8a4ncTuXiYbpY55gfgR3rF0TIDr8l4pBU8mMo4bVPPHqDr3su?=
	=?Windows-1252?Q?nBaKxmoCFJADYBrQ0pspsiENTCLl1F/MT/a8MUFQKzM6SzInZXdpWFd4?=
	=?Windows-1252?Q?1ZK1e6x6/WKsZHPtRoXe3EYOmlMB8f7LmewfssLye5BVSc8xspJ9mfmu?=
	=?Windows-1252?Q?BdDKeGw9JWuprcL6lnha4Lr/TXdiMGZa+5klEDoKIY6E1I2Zrzy9IdVC?=
	=?Windows-1252?Q?w4AZ4apOOXLTzM7edj3CC0opOPsXb1gduVxZTJ6wQwug0bJt0yAu0rha?=
	=?Windows-1252?Q?tBqGQBpw7+onhbJyM79lgQZNGNt8Cwj/RdgejNYf/sPukPFpSaMNuAze?=
	=?Windows-1252?Q?eefLVj/Ef397qz04qBWg6sE4VtQ2bjwkPEzFpXjEV5YlnCbXy8QMJXTj?=
	=?Windows-1252?Q?TRgldL4HkqEy0DS8OldCq9kom4CyfLOwXGhE1Au1ApK9wrh4ssshNDYk?=
	=?Windows-1252?Q?mucWi6WbdLp0CeAxFFuthBkcs7F0dFWobfFSfCorOdgQzH8NoH40nNeN?=
	=?Windows-1252?Q?HKky2FSS7urhz2bEzMTqb4YLw9s1touZPad7QekSMGl5ICH7AimY33xu?=
	=?Windows-1252?Q?MVFR2HVAcBXzi4xzygEzYuLgnh8SBg25S9nMfZ5ADSDLkSuR5KCr92DW?=
	=?Windows-1252?Q?YWO2Qx8gYS8v2BqGEBOxEn/D4iPyhpg7HoxWkD9kJXm5HYCIFiI1Hbzy?=
	=?Windows-1252?Q?7YkO2QmzdCwWOZZP/2WZHcVZr/r74rJR4j0qtP10DvbBjPlO5+KElhYV?=
	=?Windows-1252?Q?YGMp5zyECjSWmwIns82fBg7i1AIOlIbEfZ2kPym44v5A8S6tEswOdvry?=
	=?Windows-1252?Q?JXyWtlSU5GMlGu5ze/ltQwEmgwuW7aRbzwiQfmqEYjUOpeBN3rznXtNF?=
	=?Windows-1252?Q?mq1LxkUkKPpkl8+/8aw32D0myHgqwEdshHYYvgDMLM5NbE57qwwQahY1?=
	=?Windows-1252?Q?V997vuALk65y1QfhejP4BJr7ahBWMiRfz+3ZJT09wpI4CG3EaFND9ykF?=
	=?Windows-1252?Q?yNnlQ0rE?=
X-OriginatorOrg:	cornell.edu
X-MS-Exchange-CrossTenant-Network-Message-Id:	3a006ab3-f0d5-4d40-3ea8-08d968df6fc6
X-MS-Exchange-CrossTenant-AuthSource:	BN7PR04MB4388.namprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs:	Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime:	26 Aug 2021 22:18:31.5422 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader:	Hosted
X-MS-Exchange-CrossTenant-Id:	5d7e4366-1b9b-45cf-8e79-b14b27df46e1
X-MS-Exchange-CrossTenant-MailboxType:	HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName:	bXtXT684aynTyBl+Io2hqGD0b3jVcJIQ1xO4oLyCER6nWhfYvqEdI+1FsDN8ZJkkS9YA8cptrXPnF8J75tgtCg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped:	BN8PR04MB5617
X-Spam-Status:	No, score=0.2 required=5.0 tests=BAYES_00, BODY_8BITS,
	DKIM_INVALID, DKIM_SIGNED, KAM_DMARC_STATUS, MSGID_FROM_MTA_HEADER,
	NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS,
	TXREP autolearn=no autolearn_force=no version=3.4.4
X-Spam-Checker-Version:	SpamAssassin 3.4.4 (2020-01-24) on
	server2.sourceware.org
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.29
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe:	<https://cygwin.com/mailman/options/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	Ken Brown via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	Ken Brown <kbrown AT cornell DOT edu>
Errors-To:	cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com
Sender:	"Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
X-MIME-Autoconverted:	from quoted-printable to 8bit by delorie.com id 17QMK19P009776

On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote:
> On 8/25/2021 5:29 PM, Takashi Yano wrote:
>> On Wed, 25 Aug 2021 13:52:19 -0400
>> Ken Brown wrote:
>>> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
>>>> On Tue, 24 Aug 2021 12:49:52 -0700
>>>> Chris Roehrig wrote:
>>>>> I have a network of Windows, Linux and Mac machines and I use rsync to 
>>>>> synchronize various directories between them.
>>>>>
>>>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only 
>>>>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or 
>>>>> Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as 
>>>>> expected from gigabit ethernet.  This has been an ongoing problem for me 
>>>>> for a couple of years over several Windows and Cygwin versions, and I'd 
>>>>> like to try to fix it.
>>>>>
>>>>> If I run rsync --daemon --no-detach under mintty in the foreground on the 
>>>>> remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems 
>>>>> like it has something to do with rsync.exe running in the background under 
>>>>> the cygrunsrv+sshd service (which was installed normally using 
>>>>> ssh-host-config).
>>>>>
>>>>> If I do:
>>>>>     pv /dev/zero | ssh $WINHOST "cat > /dev/null"
>>>>> or even
>>>>>     pv /dev/urandom | ssh $WINHOST md5sum
>>>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself 
>>>>> is being throttled by bandwidth or CPU.
>>>>>
>>>>> The machines have less than 15% CPU utilization while transferring, with 
>>>>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
>>>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using 
>>>>> only few percent CPU, and show Power Throttling=Disabled, 
>>>>> Priority=Normal.   Setting their Priority to High doesn't seem to change 
>>>>> things.
>>>>>
>>>>> Looking in Resource Monitor on the remote endpoint, the network usage is 
>>>>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure 
>>>>> looks to me as if rsync is somehow being bandwidth-throttled when run in 
>>>>> the background under cygsshd.
>>>>>
>>>>> It's almost as if rsync has an implicit --bwlimit override when it is run 
>>>>> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no 
>>>>> difference).
>>>>>
>>>>>
>>>>> Any ideas?    Not sure where to go from here.
>>>>
>>>> In cygwin, just scp is very slow.
>>>>
>>>> The transfer speed in my environment is as follows.
>>>> The tests were done with 100MB of test.dat file.
>>>>
>>>> (1-1) From cygwin-PC,
>>>> [yano AT cygwin-PC ~]$ scp test.dat yano AT linux-server:.
>>>> yano AT linux-server's password:
>>>> test.dat                                      100%  100MB   4.0MB/s   00:24
>>>> [yano AT cygwin-PC ~]$ scp yano AT linux-server:test.dat .
>>>> yano AT linux-server's password:
>>>> test.dat                                      100%  100MB   8.0MB/s   00:12
>>>>
>>>> (1-2) From linux-server,
>>>> yano AT linux-server:~$ scp yano AT cygwin-PC:test.dat .
>>>> yano AT cygwin-PC's password:
>>>> test.dat                                      100%  100MB   4.0MB/s   00:24
>>>> yano AT linux-server:~$ scp test.dat yano AT cygwin-PC:.
>>>> yano AT cygwin-PC's password:
>>>> test.dat                                      100%  100MB   4.1MB/s   00:24
>>>>
>>>>
>>>> I looked into this problem, and noticed that this is caused
>>>> by cygwin pipe implementation. Pipe in cygwin is configured
>>>> with FILE_FLAG_OVERLAPPED.
>>>>
>>>> If the pipe is configured without FILE_FLAG_OVERLAPPED,
>>>> the transfer speed is much improved as follows.
>>>>
>>>>
>>>> (2-1) From cygwin-PC,
>>>> [yano AT cygwin-PC ~]$ scp test.dat yano AT linux-server:.
>>>> yano AT linux-server's password:
>>>> test.dat                                      100%  100MB  85.5MB/s   00:01
>>>> [yano AT cygwin-PC ~]$ scp yano AT linux-server:test.dat .
>>>> yano AT linux-server's password:
>>>> test.dat                                      100%  100MB  69.7MB/s   00:01
>>>>
>>>> (2-2) From linux-server,
>>>> yano AT linux-server:~$ scp yano AT cygwin-PC:test.dat .
>>>> yano AT cygwin-PC's password:
>>>> test.dat                                      100%  100MB  80.1MB/s   00:01
>>>> yano AT linux-server:~$ scp test.dat yano AT cygwin-PC:.
>>>> yano AT cygwin-PC's password:
>>>> test.dat                                      100%  100MB  57.7MB/s   00:01
>>>>
>>>> I am not sure why this happens and how to fix this.
>>>
>>> A couple years ago I had an idea for changing the pipe implementation to avoid
>>> overlapped I/O:
>>>
>>>     https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
>>>     https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
>>>
>>> I never followed up on it.  But if you think it might help with this problem, I
>>> could dust it off and try to finish it.
>>
>> Interesting.
>>
>> It will be also helpfull for:
>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
>> which seems to be the same issue with
>> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app 
>>
>>
>> However, I wonder why scp dislikes overlapped I/O.
> 
> I agree that it would be good to understand this.  When I first proposed the 
> change, I was thinking in terms of code simplification.  If it also improves 
> performance (which we don't know yet), it becomes a higher priority, but in that 
> case it would be nice to understand why it improves performance.

Hi Takashi,

In case you want to try out my proposed change, I've just rebased the patches to 
the current master and pushed them to a new topic/pipe branch.

Ken

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -