X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:in-reply-to:references:from:date
	:message-id:subject:to:content-type; q=dns; s=default; b=n8RSBia
	7ICW4jT0KrAWNx4aMJd4QHDQiegEiXFsLrE3CnSNHyoXGsCshu6YgAYXFdxNx5Bq
	cVtSK4IcdMJuXgMImSA8Z6OjWDvjzJDKsTg65N1B7uqTu5/LCb6T5yEYJdFdMlnU
	TVXOhBER3k1B3B52CWzWfzA25rjX+tzRmYuA=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:in-reply-to:references:from:date
	:message-id:subject:to:content-type; s=default; bh=VUnRKTCFFJNJR
	tPfVnKIJOLn1FA=; b=nXu+HMTSg0vknBsz5cmNpP4AYrVlnCtrxTSp7j7x7dkPJ
	smLgV3eigcehIa+9Snig0/nDDVogKnGjZEI1PvGp+BbyhJgCaKOkdnAw3QRyhiFs
	zZvVqTdkHFGXk1SItyqcwnU4K/Dspb5J8YS8yCZ32jcoVJ6iD1Ukd1s2wtDVd0=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Virus-Found:	No
X-Spam-SWARE-Status:	No, score=-1.4 required=5.0 tests=AWL,BAYES_00,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 spammy=Resource, gregfreemyergmailcom, greg DOT freemyer AT gmail DOT com, reboot
X-HELO:	mail-lf0-f41.google.com
X-Google-DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=CG75gJb8gvRpX45VKtHPyyDDRJxYcMJag9V88GzoXsw=; b=Gz9NTeXjZRoLVKkZX2znpTxSiGsD/UeFS4WaPtroQ0r+YxaPpA5LnPryzlIX3glA4q LHGW/937vVEnm3k1V/oksCFfb1Eiba00h5quiSCGFSTNWqDdfLoVf9fdxdH99X1uBg6o KMiTTF8o/IrjsG6pw6WRNNMgwMaR2zjS7DIz38vVhs9O+6tFEpFkdKeHeeQuH/fMamXg Bc7lQNdT/fFwcoLm8LXSrbiWb4Ka0psN2x+fIA/j+BRYGpGJzL0Q4h1szWOlKkcIu8iy p8WUTmZiqYg2YOK5uWrURqiYHwIZVwnCQqL3exVq0POLDZfSRa00rnp7ha4Fgi37Xewi yn3w==
X-Gm-Message-State:	AOPr4FVnZjGFBZp+EvH6XkWa2T4zyxZTKhWxCwUrhlEOvi6hqbmLAg4G84nI3E6Z72D121YXCHTPAAETaCm9gQ==
X-Received:	by 10.112.154.106 with SMTP id vn10mr1461275lbb.123.1460398875136; Mon, 11 Apr 2016 11:21:15 -0700 (PDT)
MIME-Version:	1.0
In-Reply-To:	<570BE03D.7030709@gmail.com>
References:	<CAGpXXZJV-nsB=xyO7=EBUb-JCk5KqMPS=bvg2aWgw_jm4OZFsQ AT mail DOT gmail DOT com> <CAGpXXZLJ=O7y-6YEcODv4r32X96zGSU5PX9KwzwF3QedSV4iHA AT mail DOT gmail DOT com> <570BE03D DOT 7030709 AT gmail DOT com>
From:	Greg Freemyer <greg DOT freemyer AT gmail DOT com>
Date:	Mon, 11 Apr 2016 14:20:35 -0400
Message-ID:	<CAGpXXZL9ZAJF_YSUOR=9WCvu+_K5Uub-zwWdpUV-yu7+OkYzFg@mail.gmail.com>
Subject:	Re: rsync performance degradation? Could be a windows issue?
To:	cygwin AT cygwin DOT com
X-IsSubscribed:	yes

On Mon, Apr 11, 2016 at 1:34 PM, Marco Atzeri <marco DOT atzeri AT gmail DOT com> wrote:
>
>
> On 11/04/2016 18:57, Greg Freemyer wrote:
>>
>> On Thu, Mar 31, 2016 at 4:48 PM, Greg Freemyer <greg DOT freemyer AT gmail DOT com>
>> wrote:
>>>
>>> All,
>>>
>>> I'm not sure this is on-topic, but at least I'm in cygwin when I
>>> notice the below:
>>>
>>> I do a lot of large data transfers between USB drives.  Often I get
>>> great speeds (70MB/sec or more).
>>>
>>> Sometimes it falls way off to closer to 20MB/sec with the same class
>>> of hardware.
>>>
>>> I experienced the "slow" transfer speed today, so I thought I'd ask if
>>> anyone knew a way to resolve it?
>>
>>
>> I'm trying rsync of a bunch of large files again today.
>>
>> Initially I saw the same slow copy speed (about 20 MB/sec).  My
>> destination was connected to a USB 3.1 port, but it seemed to be the
>> bottleneck so I moved it to a USB 3 port (3.1 should be 2x the speed
>> of 3.0, so it should not have been the bottleneck).
>>
>> After getting my throughput up above 50 MB/sec I notice in the windows
>> "performance monitor" that my source disk is hitting 100% utilization,
>> then a few seconds later my destination disk is.  And back and forth.
>> It seems I'm only reading or writing for a few seconds, then
>> alternating.
>>
>> I assume the issue is that too much data is being read / cached by
>> rsync prior to it being written out so I'm getting no advantage of
>> reading and writing in parallel.
>>
>> (I gather iostat isn't available for cygwin?)
>>
>> Is there some rsync (or cygwin) option / feature that would encourage
>> parallel reading/writing?
>>
>> fyi: I did some linux testing with "dd" over the weekend and I hit 140
>> MB/sec if I used a 100MB blocksize.  That was to / from the raw disk
>> (/dev/sda => /dev/sdb).
>>
>> If I bumped my blocksize to 1GB for dd in linux, my throughput dropped
>> to 70 MB/sec just as I see right now with rsync in cygwin.
>>
>> Thanks
>> Greg
>>
>
>
> Is windows robocopy faster ?

As I test, I just copied 30 GB of 1.5 GB files via robocopy.  None of
the files had been accessed since a reboot, so none should have been
in cache.

According to Resource Monitor, yes.

I'm getting about 105 MB/sec for read and 105 MB/sec for write vs  70
MB/sec for rsync.

Significantly, with robocopy Resource Monitor is showing a consistent
read/write speed.  With rsync read and write fluctuate back and forth.

As I stated before, I believe I should be able to get 140 MB/sec, but
I wouldn't complain about 105 MB/sec.  70 MB/sec just seems to
highlight a flaw in how rsync manages the data flow since it is 50% of
theoretical max.

ie.  this seems to be what rsync in cygwin is doing:

while (files)  {
    read 1.5 GB file to ram
    write 1.5 GB file from ram
    fsync()  ensure 1.5 GB file is on disk
} endwhile

I haven't tested in linux.  Maybe rsync just isn't as efficient as I expect?

There is also the problem that rsync often slows down to below 50
MB/sec   I've tried it with no other activity on the PC.  Today I was
only getting 20 MB/sec at first.  (I moved the USB cable and it came
up to 70 MB/sec).

Thanks
Greg

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -