X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:mime-version:subject:to:from:message-id
	:content-type:content-transfer-encoding; q=dns; s=default; b=iNN
	zDGr4UnKgTsY2aiOk6Mu3GRDJKnoHdV0C3AUaNYJVPwhw0KyTTuawvPWG0/JH2mI
	/w3UpDc0L/wCRY262fgO4kKAvs+Yr3xP3yEfgBa6wOgUVgiuBGyZXIBi5tQhdd4O
	R4m/Dw8C11nKu/9SbC54VerzccQSAqEr/HjGb+WY=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:mime-version:subject:to:from:message-id
	:content-type:content-transfer-encoding; s=default; bh=pUL+b2QJ6
	+g4h8WTMQLPrlZrlrE=; b=Ub+IsJpz4ZVCocQzP1/nArZl3/qKv+/B8M/hF2wgz
	saYSZBrYGyPZBjc4+ZAQfMZ4sAbKJCncAs43Jd2YzGLZIt9IHAnAz67UcO/NcjI2
	1h3Hqa1biXl1RUy46NpXtoA1IpvvXqR/8UXgWXH561MsIZqAzKAIAxihEMvGIRz/
	3M=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Spam-SWARE-Status:	No, score=1.6 required=5.0 tests=AWL,BAYES_20,FOREIGN_BODY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=no version=3.3.1 spammy=videos, diese, wurde, mit
X-HELO:	ms94429.hosts.iptox.net
Date:	Sat, 04 Jan 2020 23:15:55 +0100
User-Agent:	K-9 Mail for Android
MIME-Version:	1.0
Subject:	rsync and ls -lR slow for directories with many files
To:	cygwin AT cygwin DOT com
From:	muell AT plutonium24 DOT de
Message-ID:	<8582CD6F-C872-41FB-9425-2CBD1126AE33@plutonium24.de>
X-IsSubscribed:	yes
X-MIME-Autoconverted:	from quoted-printable to 8bit by delorie.com id 004MGWhl030122

I am running rsync on a small linux server to synchronize files in one directory and its subdirectories from Windows (using sshd from Cygwin) to this server for backup purposes. The directory contains almost 1 TB of images and videos in about 160k files on a slow disk (Seagate Archive 8TB with SMR) with NTFS.

Even if there are no changes and whith whole file transfers rsync takes about 45 minutes to come to this conclusion.
I am using the following command line on the linux server:

rsync -avx --stats --whole-file --no-perms --no-owner --no-group <user>@<server>:<source directory> <local destination directory>

As rsync was only transferring a small number of bytes and gave no clue to the cause for being so slow and as rsync should only need filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the linux server this took about 1 minute (only slightly faster magnetic disk, empty read cache at start) and doing the same on cygwin took almost as long as rsync (over 40 minutes). Using Windows Explorer (after a reboot to guarantee that the cache is empty) to get the total number of files and the total size took only a few seconds. Reading all file sizes with Treesize also took less than one minute. As ls -lR needs the same information I would have expected it to take the same time.

Runnin "ls -lR" a second time on Cygwin is fast as lightning as it only takes less than 30s.

Is there any way to get ls -lR or better rsync as fast as listing the directory with Windows tools?

Frank

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -