X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:message-id:date:from:mime-version:to:subject
	:references:in-reply-to:content-type:content-transfer-encoding;
	q=dns; s=default; b=fH8QFZctc/gX190jMk99YmvCFKDrV1rZamf1i/R0AMW
	upQJt1zJvAY9x9nvamztDtqwY4stl4QWQmGz68EyTdVLErdrOYJHL9g/3Ev0fAsj
	vtfFNujPIrfwKJHOIsa2zciwW7tWYxGDKWCPEt1bNkjNiUfHFV4wjejben0XOTPs
	=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:message-id:date:from:mime-version:to:subject
	:references:in-reply-to:content-type:content-transfer-encoding;
	s=default; bh=EuJj+iW81amiY+pJ9rfVPSZs9ag=; b=sZhlqVvYru8XeHAew
	CVrI/FfSZozUJWIl3Pieok+QCHDuNS9Pw+vCuLHKUJM4g0mk1rjyz38upF5V2M12
	GNGDY2KNipRCeZ5MMtQluxesoRA2IC5GGNwGr27wDQeMiyFW2fBs+BDvblQCaVGS
	+WJ2e6L6dr5ba6fkWgK06hJ5E4=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Spam-SWARE-Status:	No, score=-2.6 required=5.0 tests=AWL,BAYES_00,FOREIGN_BODY,GIT_PATCH_2,SPF_PASS autolearn=no version=3.3.1 spammy=von, diese, signing, wurde
X-HELO:	Ishtar.sc.tlinx.org
Message-ID:	<5E3252E2.7070104@tlinx.org>
Date:	Wed, 29 Jan 2020 19:52:02 -0800
From:	L A Walsh <cygwin AT tlinx DOT org>
User-Agent:	Thunderbird
MIME-Version:	1.0
To:	cygwin AT cygwin DOT com, Frank-Ulrich Sommer <f-u DOT s AT gmx DOT net>
Subject:	Re: rsync and ls -lR slow for directories with many files
References:	<8582CD6F-C872-41FB-9425-2CBD1126AE33 AT plutonium24 DOT de> <CANnLRdjZbTrfjf5roqwZBcBUfLzb6MJfsyJRyfmWCFivWrJzaA AT mail DOT gmail DOT com> <30EE28DF-7A24-4665-9C62-A36C16F9D285 AT gmx DOT net>
In-Reply-To:	<30EE28DF-7A24-4665-9C62-A36C16F9D285@gmx.net>
X-IsSubscribed:	yes

On 2020/01/08 08:43, Frank-Ulrich Sommer wrote:
> but rsync did not get faster.
>
> I'm sorry to admit that the ultimate solution does not use Cygwin any more. I'm now using a Windows share and connect to that share from my Linux server with cifs and autofs. rsync then runs on the linux machine and accesses that share (due to --whole-file this should not cause problems).
>
> rsync without any changed files directly after booting both systems (both caches are empty) now takes 91s instead of 42m.
>   
---
    I won't go into the many reasons I know of (which are very 
incomplete compared
to those who actually created cygwin), but in situations like you describe,
I almost never use rsync unless I don't care about time and desperately 
need a
feature it has.  Instead I'll find it's faster to create an uncompressed tar
on windows, copy it to linux and expand it there to work locally with it.

    Any compression/signing/encrypting of data will slow down data 
transfer on
my home network where max CIFS transfer speeds are in the 300-700MB/s range.
>
>
>
>
> Am 5. Januar 2020 22:22:35 MEZ schrieb Stephen John Smoogen <smooge AT gmail DOT com>:
>   
>> On Sat, 4 Jan 2020 at 17:16, <muell AT plutonium24 DOT de> wrote:
>>     
>>> I am running rsync on a small linux server to synchronize files in
>>>       
>> one directory and its subdirectories from Windows (using sshd from
>> Cygwin) to this server for backup purposes. The directory contains
>> almost 1 TB of images and videos in about 160k files on a slow disk
>> (Seagate Archive 8TB with SMR) with NTFS.
>>
>> I am not sure if the Linux box has the slow disk or the Windows box
>> has the slow disk.
>>
>>     
>>> Even if there are no changes and whith whole file transfers rsync
>>>       
>> takes about 45 minutes to come to this conclusion.
>>     
>>> I am using the following command line on the linux server:
>>>
>>> rsync -avx --stats --whole-file --no-perms --no-owner --no-group
>>>       
>> <user>@<server>:<source directory> <local destination directory>
>>     
>>> As rsync was only transferring a small number of bytes and gave no
>>>       
>> clue to the cause for being so slow and as rsync should only need
>> filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the
>> linux server this took about 1 minute (only slightly faster magnetic
>> disk, empty read cache at start) and doing the same on cygwin took
>> almost as long as rsync (over 40 minutes). Using Windows Explorer
>> (after a reboot to guarantee that the cache is empty) to get the total
>> number of files and the total size took only a few seconds. Reading all
>> file sizes with Treesize also took less than one minute. As ls -lR
>> needs the same information I would have expected it to take the same
>> time.
>>
>>
>> I would add a bunch of verbose to the rsync to see what it is doing.
>> (I don't recommend sending that to the list as it will be a lot of
>> data.. but maybe an excerpt) I am expecting it is spending a lot of
>> time getting the metadata off of one of the disks and mapping it to
>> Unix permissions then comparing if those items are the same on the
>> other side. Each one of those is going to be a separate action which
>> on a slow drive may be a spinup/get-data/spindown cycle to make it
>> even slower.
>>
>> I would then check to see if perms and metadata on that directory
>> 'look sane' (this is highly dependent on your environment.. if you
>> have an AD server giving out perms it will look different from other
>> things.) If the lookups for mapping metadata permissions is having to
>> ping an AD server or some sort of other network lookup that is going
>> to also slow down things.
>>
>> Sorry I don't have any 'fixes'. I have always found large rsync
>> between Windows and Unix to be slow.
>>
>>     
>>> Runnin "ls -lR" a second time on Cygwin is fast as lightning as it
>>>       
>> only takes less than 30s.
>>     
>>> Is there any way to get ls -lR or better rsync as fast as listing the
>>>       
>> directory with Windows tools?
>>     
>>> Frank
>>>
>>> --
>>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>>>       
>> gesendet.
>>     
>>> --
>>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>>>       
>> gesendet.
>>     
>>> --
>>> Problem reports:       http://cygwin.com/problems.html
>>> FAQ:                   http://cygwin.com/faq/
>>> Documentation:         http://cygwin.com/docs.html
>>> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>>>
>>>       
>> -- 
>> Stephen J Smoogen.
>>
>> --
>> Problem reports:       http://cygwin.com/problems.html
>> FAQ:                   http://cygwin.com/faq/
>> Documentation:         http://cygwin.com/docs.html
>> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>>     
>
>   

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -