X-Recipient: archive-cygwin AT delorie DOT com
X-Spam-Check-By: sourceware.org
Message-ID: <84dde90811110922j64d00edet3b571b4d93603459@mail.gmail.com>
Date: Tue, 11 Nov 2008 12:22:42 -0500
From: "Linus Hicks" <linush100 AT gmail DOT com>
To: cygwin AT cygwin DOT com
Subject: Re: cygwin 1.5.25-15: cwrsync 2.1.3 "hangs" when 80 mb of 100 gb data network folder is to be transmitted
In-Reply-To: <E1Kza6t-000Dlt-00._oiler_-mail-ru@f169.mail.ru>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <E1Kza6t-000Dlt-00 DOT _oiler_-mail-ru AT f169 DOT mail DOT ru>
X-IsSubscribed: yes
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Precedence: bulk
List-Id: <cygwin.cygwin.com>
List-Unsubscribe: <mailto:cygwin-unsubscribe-archive-cygwin=delorie DOT com AT cygwin DOT com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com

2008/11/10 Evgeniy:
> Hello,
>
> At main office we have a data storage server with a ~100 Gb folder of data which is daily updated on about 80 Mb. These updated 80 Mb of data should be daily synchronized to a remote data storage at branch office via rsync. rsync client is installed on a workstation at the branch office and cygwin 1.5.25-15 sshd service is installed on a workstation at the main office. The synchronized data storage folders are available to these workstations as network drives. These ~100 Gb folders of data contain a lot of 0,5 - 2 Mb sized files which are pdfs and picture scans. rsync works perfectly on smaller volumes of data, when we transmit, for example, 100 Mb of these files. But when we try to transmit all the 100 Gb folder in which we have only 80 Mb of new files and the rest of the files are identical by name, size and modification time to destination folder files, we got rsync somewhat "hung".
>
-- snip --
>
> So the question is what is wrong? I can't beleive that rsync can't handle with such a big volume of data when it is necessary to transmit only a small part of it.
>
> Any ideas and suggestions would be very much appreciated.

This is probably not Cygwin specific, so I suggest you also check on
the rsync mailing list.

You seem focused on the fact that there is only about 80Mb of
different data, but there are other things that affect rsync
performance.

The performance of rsync can be greatly affected by the combination of
the number of files in the file list and the amount of physical memory
available to it. From the information you have provided, I can only
guess at a few things, but in your example that performs well you say
the total file size is 100Mb while in the case that seems to hang, the
total file size is about 1,000 times greater, or 100Gb.

If that size difference also means you have 1,000 times as many files,
then the size of the file list is probably important with regard to
performance. Your system(s) may have entered a state of heavy use of
the paging file. Have you monitored system activity while running
rsync to find where resources are being used? If not, I suggest you
try running the 100Gb case again while monitoring memory usage to see
what happens.

Linus

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/