DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:mime-version:subject:to:from:message-id
	:content-type:content-transfer-encoding; q=dns; s=default; b=iNN
	zDGr4UnKgTsY2aiOk6Mu3GRDJKnoHdV0C3AUaNYJVPwhw0KyTTuawvPWG0/JH2mI
	/w3UpDc0L/wCRY262fgO4kKAvs+Yr3xP3yEfgBa6wOgUVgiuBGyZXIBi5tQhdd4O
	R4m/Dw8C11nKu/9SbC54VerzccQSAqEr/HjGb+WY=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Date: Sat, 04 Jan 2020 23:15:55 +0100
User-Agent: K-9 Mail for Android
MIME-Version: 1.0
Subject: rsync and ls -lR slow for directories with many files
To: cygwin AT cygwin DOT com
From: muell AT plutonium24 DOT de
Message-ID: <8582CD6F-C872-41FB-9425-2CBD1126AE33@plutonium24.de>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

I am running rsync on a small linux server to synchronize files in one directory and its subdirectories from Windows (using sshd from Cygwin) to this server for backup purposes. The directory contains almost 1 TB of images and videos in about 160k files on a slow disk (Seagate Archive 8TB with SMR) with NTFS.

Even if there are no changes and whith whole file transfers rsync takes about 45 minutes to come to this conclusion.
I am using the following command line on the linux server:

rsync -avx --stats --whole-file --no-perms --no-owner --no-group <user>@<server>:<source directory> <local destination directory>

As rsync was only transferring a small number of bytes and gave no clue to the cause for being so slow and as rsync should only need filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the linux server this took about 1 minute (only slightly faster magnetic disk, empty read cache at start) and doing the same on cygwin took almost as long as rsync (over 40 minutes). Using Windows Explorer (after a reboot to guarantee that the cache is empty) to get the total number of files and the total size took only a few seconds. Reading all file sizes with Treesize also took less than one minute. As ls -lR needs the same information I would have expected it to take the same time.

Runnin "ls -lR" a second time on Cygwin is fast as lightning as it only takes less than 30s.

Is there any way to get ls -lR or better rsync as fast as listing the directory with Windows tools?

Frank

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple