Message-Id: <200712051533.lB5FXApr020147@delorie.com> X-Recipient: archive-cygwin AT delorie DOT com X-Spam-Check-By: sourceware.org From: "Remy Cote" To: Subject: 1.5.24: rsync slow when env variable cygwin=noNTsec Date: Wed, 5 Dec 2007 10:29:58 -0500 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_002F_01C83729.C9D0A4A0" X-Mailer: Microsoft Office Outlook, Build 11.0.5510 Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner AT cygwin DOT com Mail-Followup-To: cygwin AT cygwin DOT com Delivered-To: mailing list cygwin AT cygwin DOT com ------=_NextPart_000_002F_01C83729.C9D0A4A0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, I use rsync intensively to backup data from server to server over WAN and LAN link. The "Backup servers" are always windows 2003 server. The source can be both Windows and Linux. In most of the cases the source has the daemon running and the backup server run the client as scheduled task. I also do local to local. I use the --linkdest=3D option (NTFS Hardlink) b= ut the problem occurs even if I don't use the linkdest option. I tried many different version of rsync (2.6.9, 3.0.2, 2.6.8) as well as many different version of cygwin1.dll (1.5.22, 23, 24) and I always have the same results. For instance to sync 51 Gb of data on a 100Mb LAN link with 3GB of data that has change. It will take 9 minutes if Cygwin=3Dntsec and 49 minu= tes if cygwin=3DnoNTsec. Here is an example of the command I'm using: RSYNC.EXE -r -t -W --delete-during -v --stats --human-readable --link-dest=3D"/CYGDRIVE/E/KLB/KLBFIL01/MON/" "/CYGDRIVE/D/SHARE/" "/CYGDRIVE/E/KLB/KLBFIL01/TUE/" 1>log.txt 2> err.txt For Now I run my job with ntsec and do a ACL reset when the sync in completed but it takes some additional backup times (I have between 50 000 to 1 700 000 files per server to sync) To give more information about the environment we have AD 2003, Windows servers are always 2003, linux server are Red Hat. The problem is also present on Windows XP. Here is the log file I produce from rSync. First with noNtSec then same job with ntSec 11/2/2007 9:00:01 PM: CMD =3D Omni_Friday.BAT 10.71.101.59::OmniDay /CYGDRIVE/E/KLB/OMNI/Friday/ -v -r -z -t --stats --delete-after --human-readable --log-file=3DOmni_Friday_LOG.TXT 11/2/2007 9:48:58 PM: 10.71.101.59::OmniDay =3D 0 - Success=20=20 Size =3D 3.33G bytes/51.64G bytes Time List/DL =3D 4.568 seconds/0.000 seconds Qte files =3D 991/89911 files. 10/26/2007 9:00:02 PM: CMD =3D Omni_Friday.BAT 10.71.101.59::OmniDay /CYGDRIVE/E/KLB/OMNI/Friday/ -v -r -z -t --stats --delete-after --human-readable --log-file=3DOmni_Friday_LOG.TXT 10/26/2007 9:09:52 PM: 10.71.101.59::OmniDay =3D 0 - Success=20=20 Size =3D 3.30G bytes/51.58G bytes Time List/DL =3D 7.748 seconds/0.000 seconds Qte files =3D 833/89317 files. Note that if the destination folder is empty there is no big difference between ntsec and noNTsec, but if most of the files are there it will take longer for rsync to start downloading the changed file if cygwin=3Dnontsec. Thanks for all the help you can bring and have a great day! R=E9my C=F4t=E9 ------=_NextPart_000_002F_01C83729.C9D0A4A0 Content-Type: application/octet-stream; name="cygcheck.out" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="cygcheck.out" Cygwin Configuration Diagnostics Current System Time: Wed Dec 05 09:09:26 2007 Windows XP Professional Ver 5.1 Build 2600 Service Pack 2 Path: C:\PVSW\BIN D:\Program Files\Windows Resource Kits\Tools\ C:\WINDOWS\system32 C:\WINDOWS C:\WINDOWS\System32\Wbem C:\Program Files\Java\jdk1.5.0_07\bin C:\Program Files\Windows Imaging\ D:\scripts\FastSum C:\Program Files\Common Files\GTK\2.0\bin Output from C:\cygwin\bin\id.exe (nontsec) UID: 11111(rcote) GID: 10545(mkgroup_l_d) 0(root) 544(Administrators) 545(Users) 10545(mkgroup_l_d) Output from C:\cygwin\bin\id.exe (ntsec) UID: 11111(rcote) GID: 10545(mkgroup_l_d) 0(root) 544(Administrators) 545(Users) 10545(mkgroup_l_d) SysDir: C:\WINDOWS\system32 WinDir: C:\WINDOWS Path =3D 'C:\PVSW\BIN;D:\Program Files\Windows Resource Kits\Tools\;C:\WIND= OWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Java\jdk1= .5.0_07\bin;C:\Program Files\Windows Imaging\;D:\scripts\FastSum;C:\Program= Files\Common Files\GTK\2.0\bin' ALLUSERSPROFILE =3D 'C:\Documents and Settings\All Users' APPDATA =3D 'C:\Documents and Settings\rcote\Application Data' CLASSPATH =3D 'C:\PVSW\BIN\PVJDBC2X.JAR;C:\PVSW\BIN\PVJDBC2.JAR' CommonProgramFiles =3D 'C:\Program Files\Common Files' COMPUTERNAME =3D 'KLB-46113851W' ComSpec =3D 'C:\WINDOWS\system32\cmd.exe' FP_NO_HOST_CHECK =3D 'NO' HOMEDRIVE =3D 'U:' HOMEPATH =3D '\' HOMESHARE =3D '\\KLBFIL01\HOME$\rcote' LOGONSERVER =3D '\\KLBADC01' NUMBER_OF_PROCESSORS =3D '1' OS =3D 'Windows_NT' PATHEXT =3D '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH' PROCESSOR_ARCHITECTURE =3D 'x86' PROCESSOR_IDENTIFIER =3D 'x86 Family 6 Model 14 Stepping 8, GenuineIntel' PROCESSOR_LEVEL =3D '6' PROCESSOR_REVISION =3D '0e08' ProgramFiles =3D 'C:\Program Files' PROMPT =3D '$m$p$g' SESSIONNAME =3D 'Console' SystemDrive =3D 'C:' SystemRoot =3D 'C:\WINDOWS' TEMP =3D 'C:\DOCUME~1\rcote\LOCALS~1\Temp' TMP =3D 'C:\DOCUME~1\rcote\LOCALS~1\Temp' USERDNSDOMAIN =3D 'KOTT.CORP' USERDOMAIN =3D 'KOTT' USERNAME =3D 'rcote' USERPROFILE =3D 'C:\Documents and Settings\rcote' VSL =3D 'C:\PVSW\BIN' windir =3D 'C:\WINDOWS' POSIXLY_CORRECT =3D '1' HKEY_CURRENT_USER\Software\Cygnus Solutions HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\mounts v2 (default) =3D '/cygdrive' cygdrive flags =3D 0x00000022 HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\mounts v2\/ (default) =3D 'C:\cygwin' flags =3D 0x00000002 HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\mounts v2\/usr/bin (default) =3D 'C:\cygwin/bin' flags =3D 0x00000002 HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\mounts v2\/usr/lib (default) =3D 'C:\cygwin/lib' flags =3D 0x00000002 HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\Program Options HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2 HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\Program Options c: hd NTFS 20481Mb 63% CP CS UN PA FC=20=20=20=20=20 d: hd NTFS 74912Mb 58% CP CS UN PA FC Data m: net NTFS 34993Mb 15% CP CS UN PA FC=20=20=20=20=20 s: net NTFS 406370Mb 33% CP CS UN PA FC DATA t: net NTFS 406370Mb 33% CP CS UN PA FC DATA u: net NTFS 406370Mb 33% CP CS UN PA FC DATA w: net NTFS 406370Mb 33% CP CS UN PA FC DATA z: cd CDFS 582Mb 100% CS UN AstusFMSv13Setup C:\cygwin / user binmode C:\cygwin/bin /usr/bin user binmode C:\cygwin/lib /usr/lib user binmode . /cygdrive user binmode,cygdrive Not Found: awk Not Found: bash Not Found: cat Not Found: cp Not Found: cpp (good!) Not Found: crontab Not Found: find Not Found: gcc Not Found: gdb Not Found: grep Not Found: kill Not Found: ld Not Found: ls Not Found: make Not Found: mv Not Found: patch Not Found: perl Not Found: rm Not Found: sed Not Found: ssh Not Found: sh Not Found: tar Not Found: test Not Found: vi Not Found: vim 61k 2006/11/10 C:\cygwin\bin\cygbz2-1.dll - os=3D4.0 img=3D1.0 sys=3D4.0 "cygbz2-1.dll" v0.0 ts=3D2006/11/10 18:42 7k 2003/10/19 C:\cygwin\bin\cygcrypt-0.dll - os=3D4.0 img=3D1.0 sys=3D4= .0 "cygcrypt-0.dll" v0.0 ts=3D2003/10/19 3:57 40k 2006/11/15 C:\cygwin\bin\cygform-8.dll - os=3D4.0 img=3D1.0 sys=3D4.0 "cygform-8.dll" v0.0 ts=3D2006/11/15 2:06 25k 2007/08/23 C:\cygwin\bin\cyghistory6.dll - os=3D4.0 img=3D1.0 sys=3D= 4.0 "cyghistory6.dll" v0.0 ts=3D2007/8/23 18:01 271k 2007/08/24 C:\cygwin\bin\cygicons-0.dll - os=3D4.0 img=3D1.0 sys=3D4= .0 "cygicons-0.dll" v0.0 ts=3D2007/8/24 3:24 976k 2006/10/22 C:\cygwin\bin\cygiconv-2.dll - os=3D4.0 img=3D1.0 sys=3D4= .0 "cygiconv-2.dll" v0.0 ts=3D2006/10/22 19:44 37k 2003/08/10 C:\cygwin\bin\cygintl-2.dll - os=3D4.0 img=3D1.0 sys=3D4.0 "cygintl-2.dll" v0.0 ts=3D2003/8/10 17:50 31k 2005/11/20 C:\cygwin\bin\cygintl-3.dll - os=3D4.0 img=3D1.0 sys=3D4.0 "cygintl-3.dll" v0.0 ts=3D2005/11/19 21:04 31k 2006/10/23 C:\cygwin\bin\cygintl-8.dll - os=3D4.0 img=3D1.0 sys=3D4.0 "cygintl-8.dll" v0.0 ts=3D2006/10/22 20:12 21k 2006/11/15 C:\cygwin\bin\cygmenu-8.dll - os=3D4.0 img=3D1.0 sys=3D4.0 "cygmenu-8.dll" v0.0 ts=3D2006/11/15 2:05 67k 2006/11/15 C:\cygwin\bin\cygncurses++-8.dll - os=3D4.0 img=3D1.0 sys= =3D4.0 "cygncurses++-8.dll" v0.0 ts=3D2006/11/15 2:13 237k 2006/11/15 C:\cygwin\bin\cygncurses-8.dll - os=3D4.0 img=3D1.0 sys= =3D4.0 "cygncurses-8.dll" v0.0 ts=3D2006/11/15 2:02 12k 2006/11/15 C:\cygwin\bin\cygpanel-8.dll - os=3D4.0 img=3D1.0 sys=3D4= .0 "cygpanel-8.dll" v0.0 ts=3D2006/11/15 2:04 133k 2007/06/27 C:\cygwin\bin\cygpcre-0.dll - os=3D4.0 img=3D1.0 sys=3D4.0 "cygpcre-0.dll" v0.0 ts=3D2007/6/26 23:49 303k 2007/06/27 C:\cygwin\bin\cygpcrecpp-0.dll - os=3D4.0 img=3D1.0 sys= =3D4.0 "cygpcrecpp-0.dll" v0.0 ts=3D2007/6/26 23:50 7k 2007/06/27 C:\cygwin\bin\cygpcreposix-0.dll - os=3D4.0 img=3D1.0 sys= =3D4.0 "cygpcreposix-0.dll" v0.0 ts=3D2007/6/26 23:50 22k 2002/06/09 C:\cygwin\bin\cygpopt-0.dll - os=3D4.0 img=3D1.0 sys=3D4.0 "cygpopt-0.dll" v0.0 ts=3D2002/6/9 1:45 154k 2007/08/23 C:\cygwin\bin\cygreadline6.dll - os=3D4.0 img=3D1.0 sys= =3D4.0 "cygreadline6.dll" v0.0 ts=3D2007/8/23 18:01 1830k 2007/01/31 C:\cygwin\bin\cygwin1_old.dll - os=3D4.0 img=3D1.0 sys=3D= 4.0 "cygwin1.dll" v0.0 ts=3D2007/1/31 4:58 1835k 2007/02/02 C:\cygwin\bin\cygwin1.dll - os=3D4.0 img=3D1.0 sys=3D4.0 "cygwin1.dll" v0.0 ts=3D2007/2/1 19:50 Cygwin DLL version info: DLL version: 1.5.24 DLL epoch: 19 DLL bad signal mask: 19005 DLL old termios: 5 DLL malloc env: 28 API major: 0 API minor: 156 Shared data: 4 DLL identifier: cygwin1 Mount registry: 2 Cygnus registry name: Cygnus Solutions Cygwin registry name: Cygwin Program options name: Program Options Cygwin mount registry name: mounts v2 Cygdrive flags: cygdrive flags Cygdrive prefix: cygdrive prefix Cygdrive default prefix:=20 Build date: Fri Feb 2 09:50:35 JST 2007 Shared id: cygwin1S4 Can't find the cygrunsrv utility, skipping services check. Cygwin Package Information Last downloaded files to: D:\download\CygWin Last downloaded files from: http://cygwin.osuosl.org Package Version _update-info-dir 00553-1 alternatives 1.3.29a-1 ash 20040127-3 base-files 3.7-1 base-passwd 2.2-1 bash 3.2.25-17 bzip2 1.0.3-2 coreutils 6.9-5 crypt 1.1-1 cygutils 1.3.2-1 cygwin 1.5.24-2 cygwin-doc 1.4-4 editrights 1.01-1 findutils 4.3.8-1 gawk 3.1.6-1 grep 2.5.1a-4 groff 1.18.1-2 gzip 1.3.12-2 less 382-1 libbz2_1 1.0.3-2 libiconv2 1.11-1 libintl2 0.12.1-3 libintl3 0.14.5-1 libintl8 0.15-1 libncurses8 5.5-3 libpcre0 7.2-1 libpopt0 1.6.4-4 libreadline6 5.2.7-9 login 1.9-7 man 1.6e-1 mktemp 1.5-4 rebase 2.4.3-1 rsync 2.6.9-2 run 1.1.10-1 sed 4.1.5-2 tar 1.19-1 termcap 20050421-1 terminfo 5.5_20061104-1 texinfo 4.8a-1 which 1.7-1 Use -h to see help about each section ------=_NextPart_000_002F_01C83729.C9D0A4A0 Content-Type: text/plain; charset=us-ascii -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ------=_NextPart_000_002F_01C83729.C9D0A4A0--