Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
Message-ID: <42F9FC86.8060602@byu.net>
Date: Wed, 10 Aug 2005 07:09:26 -0600
From: Eric Blake <ebb9 AT byu DOT net>
User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317)
MIME-Version: 1.0
To: Bob Brown <bob AT smsolutions DOT co DOT nz>
CC: cygwin AT cygwin DOT com
Subject: Re: Rsync over SSH dies before it is able to send with lots of files
References: <42FA081C DOT 25957 DOT 24F4EADF AT localhost> <42FA2A6A DOT 29370 DOT 257AEDF0 AT localhost>
In-Reply-To: <42FA2A6A.29370.257AEDF0@localhost>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-IsSubscribed: yes

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Bob Brown on 8/9/2005 10:25 PM:
> 
>>Hmm, you're making bash expand a wildcard into 45000 files.  This
>>shouldn't crash bash, but I'm guessing there is some of memory
>>corruption or malloc failure when consuming so much memory all before
>>forking(). I'll have to see if I can try and reproduce the failure.
> 
> Do you think the win32 path length will be included for every file?  If this is 
> the case you can probably safely assume 30-40 bytes per filename.  If this 
> were the case it would quickly add up to about 1.7MB in just passed 
> filenames alone.  Considering that this will ultimately process a directory 
> with 500,000 files in it it's not hard to work out that things are going to go 
> pear shaped in a hurry.

I still want to figure out why you were getting a crash.  I tried the
following:

$ cd /tmp
$ mount -m
...
mount -f -s -b -X "c:/cygwin/bin" "/bin"
$ mkdir lots
$ # 45000 invocations of touch is slow, so do 10 files at a time...
$ for i in `seq 4500` ; do echo $i ; touch longfilename${i}0
longfilename${i}1 longfilename${i}2 longfilename${i}3 longfilename${i}4
longfilename${i}5 longfilename${i}6 longfilename${i}7 longfilename${i}8
longfilename${i}9 ; done
$ echo /cygdrive/c/cygwin/tmp/lots/*
... # Works, echo is a builtin
$ /bin/echo /cygdrive/c/cygwin/tmp/lots/*
... # Works for me, since my /bin is mounted executable
$ /cygdrive/c/cygwin/bin/echo /cygdrive/c/cygwin/tmp/lots/*
bash: /cygdrive/c/cygwin/bin/echo: Argument list too long

Hmm, I got a reasonable error message instead of a crash.  I'll keep
playing with it, to throw more data at the command line, to see if the
amount of data contributes to a crash.  Since echo is a built-in, you
should be able to do "echo rsync --progress -rtv
192.168.0.1::mydirectory/* c:\localcopy | wc" as an indication of how many
bytes were being attempted on the command line.

- --
Life is short - so eat dessert first!

Eric Blake             ebb9 AT byu DOT net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFC+fyD84KuGfSFAYARAowfAJ4oyt+h4zZyxBzg5DhVJ+6E6vHNagCfZ1Yw
ysUikKj057gZkNKgQrmQ0w4=
=3Ubk
-----END PGP SIGNATURE-----

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/