Message-ID: <4F68B020.8090404@molconn.com>
Date: Tue, 20 Mar 2012 12:28:16 -0400
From: LMH <lmh_users-groups AT molconn DOT com>
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20120216 Firefox/10.0.2 SeaMonkey/2.7.2
MIME-Version: 1.0
To: cygwin AT cygwin DOT com
Subject: Re: rsync ERROR: out of memory in flist_expand [sender]
References: <CAE=UkuqfuAJMVO1OJ6ou7UPVA-z0-VAHB=KmYcf-4EtP=6tfVQ AT mail DOT gmail DOT com> <jk9ir4$rdj$1 AT dough DOT gmane DOT org>
In-Reply-To: <jk9ir4$rdj$1@dough.gmane.org>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com

If you are continually copying large numbers of files with rsync, maybe 
you would be better off just using cp. Do you need to keep older 
versions with the same file name, meaning an incremental backup?

I generally start by just doing a cp of the src directory to the backup 
location. When the number of files is large, I just do,

cp -Rfp /src_loc_path/  /backup_loc_path/  &> copylog

This gives me a record of anything that might not have been copied due 
to permission issues, etc. It's best to do this from a windows admin 
account.

Then I run rsync on subsequent days so that rsync is just dealing with 
changes made on any one given day and not trying to copy very large 
directories. I always use a log file so I can check to make sure it has 
finished and got everything it was supposed to.

#!/bin/bash

function backup {
# write some stuff to the log file
   echo " " >> $1
   echo " " >> $1
   echo "***** new log entry *****" >> $1
   date >> $logfile_path

# run the backup
   rsync -v -a -r -p -b --suffix=.back -E -t --log-file=$1  $2  $3

# convert the log file format to windows EOL
   unix2dos $1
}

# give a location for the log file, src, and backup loc
logfile_path="/logfile_path/log.txt"
backup_src_path="/backup_src_path/"
backup_destination_path="/backup_destination_path/"
backup $logfile_path  $backup_src_path  $backup_destination_path

I just have this script attached to a windows shortcut. That way I can 
run it with a double click, or add the shortcut to the task manager if I 
want to schedule backups. I guess I should be using cron to get the full 
quantity of linux points, but I haven't got around to it.

If you wanted to, you could make a script that would check if the backup 
destination directory exists, and if it doesn't, just do cp. If it does 
exist, then run rsync.

I don't know how much volume it rsync designed to manage, especially if 
it needs to keep a connection over a net, but if you are running out of 
resources you may want to look at a different tool.

There are other options, like rdiff and unison. I have used unison, but 
it is more a mirroring tool and I found the incremental backup options 
gave me errors. I you are just trying to mirror directories or machines, 
then it is a good tool.

LMH


Richard Ivarson wrote:
> Bruno, is this rsync problem maybe related to my rsync problem which I
> mentioned recently in the thread "Rsync stops inmid of synchronisation" ?
>
> (My latest post there was: http://article.gmane.org/gmane.os.cygwin/132270
> ... but I don't know how to link to the entire thread.)
>
> -Richard
>
>
>
> Am 19.03.2012 11:51, schrieb Bruno Galindro da Costa:
>> Hi!
>>
>> I'm try to copy some files from windows to Linux using rsync but,
>> after some short of time, an error was showed. Here is the log:
>>
>> ERROR: out of memory in flist_expand [sender]
>> rsync error: error allocating core memory buffers (code 22) at
>> /home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/util.c(117)
>> [sender=3.0.9]
>> rsync: connection unexpectedly closed (35795723020 bytes received so
>> far) [receiver]
>> rsync error: error in rsync protocol data stream (code 12) at
>> io.c(601) [receiver=3.0.7]
>> rsync: connection unexpectedly closed (102 bytes received so far)
>> [generator]
>> rsync error: error allocating core memory buffers (code 22) at
>> io.c(601) [generator=3.0.7]
>>
>> Bellow is the command that I used:
>>
>> /usr/bin/rsync
>> --exclude-from=/opt/backup_ti/tmp/FILESERVER_exclude_from.tmp
>> -avzR --delete-excluded --rsh="ssh -o UserKnownHostsFile=/dev/null -o
>> StrictHostKeyChecking=no -p 31060"
>> root AT 127 DOT 0 DOT 0 DOT 1:"/cygdrive/c/fileserver"
>> /backup/fileserver/bkp_files_20120318225522
>>
>> Bellow is the /opt/backup_ti/tmp/FILESERVER_exclude_from.tmp content:
>>
>> - TEMP
>> - RECYCLE.BIN
>> - System
>>
>> Is there any workaround that I can use to "solve" this problem?
>> Maybe using an option or change a configuration file on cygwin...
>>
>> Search in google, I've found this solution
>> (http://blog.rudnyi.ru/2011/04/problem-with-rsync.html ), but, before
>> apply that, I like to see If there is another option to use.
>>
>> --
>> Att.
>> Bruno Galindro da Costa
>
>
> --
> Problem reports: http://cygwin.com/problems.html
> FAQ: http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
>
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple