X-Spam-Check-By: sourceware.org Message-Id: <7.0.1.0.0.20060321073317.01dda4f8@weasel.com> Date: Tue, 21 Mar 2006 07:55:04 -0800 To: cygwin AT cygwin DOT com From: jdeifik Subject: pthreads don't scale on windows xp, but does scale on linux, cygwin 1.5.19 In-Reply-To: <25082fe70603210714h23ec44d8v7f1cc9c0f2d4ad30@mail.gmail.co m> References: <25082fe70603210232uc7e017ft8848c336a649c7dc AT mail DOT gmail DOT com> <44200AD3 DOT 7000303 AT byu DOT net> <25082fe70603210714h23ec44d8v7f1cc9c0f2d4ad30 AT mail DOT gmail DOT com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=====================_68951171==_"; x-avg-checked=avg-ok-3D876841 X-IsSubscribed: yes Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm Precedence: bulk List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner AT cygwin DOT com Mail-Followup-To: cygwin AT cygwin DOT com Delivered-To: mailing list cygwin AT cygwin DOT com --=====================_68951171==_ Content-Type: text/plain; charset=us-ascii; format=flowed; x-avg-checked=avg-ok-3D876841 I have a dual xeon 2.4ghz machine with hypertreading enabled. This gives me 4 logical processors. The machine dual boots to windows xp sp2, and linux. I have a highly parallelizable program I wrote, and I tested it running 1 to 8 threads, running with no source changes on windows and linux. Here is the performance on linux using gcc-3.4.3 threads 1 1436.41user 0.10system 7:16.37elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k 2 436.00user 0.02system 3:38.15elapsed 199%CPU (0avgtext+0avgdata 0maxresident)k 3 369.15user 0.05system 2:03.48elapsed 298%CPU (0avgtext+0avgdata 0maxresident)k 4 359.77user 0.08system 1:42.95elapsed 349%CPU (0avgtext+0avgdata 0maxresident)k 6 357.83user 0.09system 1:40.94elapsed 354%CPU (0avgtext+0avgdata 0maxresident)k 8 358.79user 0.06system 1:41.80elapsed 352%CPU (0avgtext+0avgdata 0maxresident)k To compute efficiency, take the single thread elapsed time/(# threads * threaded elapsed time) There is virtually perfect scaling. 4 processors scale with an efficiency of about 103%. For 6 and 8 threads, efficiency goes up a small amount. Here is the performance on windows xp using cygwin pthreads and gcc-3.4.4 1 434.60user 0.20system 7:16.47elapsed 99%CPU (0avgtext+0avgdata 509696maxresident)k 2 441.78user 0.24system 3:42.06elapsed 199%CPU (0avgtext+0avgdata 510208maxresident)k 3 579.68user 0.15system 3:14.50elapsed 298%CPU (0avgtext+0avgdata 511232maxresident)k 4 675.39user 0.15system 2:51.50elapsed 393%CPU (0avgtext+0avgdata 512000maxresident)k 6 711.70user 0.18system 3:01.20elapsed 392%CPU (0avgtext+0avgdata 511488maxresident)k 8 683.35user 0.21system 2:56.05elapsed 388%CPU (0avgtext+0avgdata 512000maxresident)k Things are fine for 2 threads, scaling with an efficiency of 96% For 3 threads, scaling efficiency is 73% For 4 threads, scaling efficiency is 62% For 6 threads, scaling efficiency is 39% For 8 threads, scaling efficiency is 30% So clearly something is broken with pthreads on windows using cygwin. I can make the source code and test data available, but it is several mbytes and involves a lot of source files to compile. Thanks, Jeff Deifik --=====================_68951171==_ Content-Type: text/plain; charset=us-ascii; x-avg-checked=avg-ok-3D876841 Content-Disposition: attachment; filename="cygcheck.out" Cygwin Configuration Diagnostics Current System Time: Tue Mar 21 06:44:35 2006 Windows XP Professional Ver 5.1 Build 2600 Service Pack 2 Path: C:\WINDOWS\system32 C:\WINDOWS C:\WINDOWS\System32\Wbem l:\cygwin\bin l:\cygwin\usr\local\bin u:\j\bin C:\Program Files\Executive Software\Diskeeper\ C:\Program Files\Common Files\Sonic Shared C:\Program Files\QuickTime\QTSystem\ Output from l:\cygwin\bin\id.exe (nontsec) UID: 1003(jdeifik) GID: 513(None) 513(None) 545(Users) Output from l:\cygwin\bin\id.exe (ntsec) UID: 1003(jdeifik) GID: 513(None) 513(None) 545(Users) SysDir: C:\WINDOWS\system32 WinDir: C:\WINDOWS HOME = 'u:\j' Path = 'C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;l:\cygwin\bin;l:\cygwin\usr\local\bin;u:\j\bin;C:\Program Files\Executive Software\Diskeeper\;C:\Program Files\Common Files\Sonic Shared;C:\Program Files\QuickTime\QTSystem\' ALLUSERSPROFILE = 'C:\Documents and Settings\All Users' APPDATA = 'C:\Documents and Settings\jdeifik\Application Data' CLASSPATH = 'C:\Program Files\Java\jre1.5.0_04\lib\ext\QTJava.zip' CommonProgramFiles = 'C:\Program Files\Common Files' COMPUTERNAME = 'DUAL_XEON' ComSpec = 'C:\WINDOWS\system32\cmd.exe' CVSROOT = '/cygdrive/u/CVSROOT' FP_NO_HOST_CHECK = 'NO' HOMEDRIVE = 'C:' HOMEPATH = '\Documents and Settings\jdeifik' LOGONSERVER = '\\DUAL_XEON' NUMBER_OF_PROCESSORS = '4' OS = 'Windows_NT' PATHEXT = '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH' PROCESSOR_ARCHITECTURE = 'x86' PROCESSOR_IDENTIFIER = 'x86 Family 15 Model 2 Stepping 7, GenuineIntel' PROCESSOR_LEVEL = '15' PROCESSOR_REVISION = '0207' ProgramFiles = 'C:\Program Files' PROMPT = '$P$G' QTJAVA = 'C:\Program Files\Java\jre1.5.0_04\lib\ext\QTJava.zip' SESSIONNAME = 'Console' SystemDrive = 'C:' SystemRoot = 'C:\WINDOWS' TEMP = 'C:\DOCUME~1\jdeifik\LOCALS~1\Temp' TERM = 'cygwin' TMP = 'C:\DOCUME~1\jdeifik\LOCALS~1\Temp' USERDOMAIN = 'DUAL_XEON' USERNAME = 'jdeifik' USERPROFILE = 'C:\Documents and Settings\jdeifik' windir = 'C:\WINDOWS' POSIXLY_CORRECT = '1' HKEY_CURRENT_USER\Software\Cygnus Solutions HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\mounts v2 HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\Program Options HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2 (default) = '/cygdrive' cygdrive flags = 0x00000022 HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/ (default) = 'l:\cygwin' flags = 0x0000000a HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/bin (default) = 'l:\cygwin/bin' flags = 0x0000000a HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/lib (default) = 'l:\cygwin/lib' flags = 0x0000000a HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\Program Options a: fd N/A N/A c: hd NTFS 8738Mb 71% CP CS UN PA FC Win_Xp d: cd N/A N/A j: hd NTFS 256169Mb 89% CP CS UN PA FC Seagate_300_mp3 k: hd NTFS 230000Mb 84% CP CS UN PA FC WD_250_av l: hd NTFS 238472Mb 76% CP CS UN PA FC Seagate_250 t: hd FAT32 8463Mb 1% CP UN WD_250_TRAN u: hd NTFS 29996Mb 75% CP CS UN PA FC Seagate_300_Usr v: cd N/A N/A x: fd N/A N/A y: fd N/A N/A l:\cygwin / system binmode l:\cygwin/bin /usr/bin system binmode l:\cygwin/lib /usr/lib system binmode . /cygdrive system binmode,cygdrive Found: l:\cygwin\bin\awk.exe Found: l:\cygwin\bin\bash.exe Found: l:\cygwin\bin\cat.exe Found: l:\cygwin\bin\cp.exe Found: l:\cygwin\bin\cpp.exe Not Found: crontab Found: l:\cygwin\bin\find.exe Found: l:\cygwin\bin\gcc.exe Found: l:\cygwin\bin\gdb.exe Found: l:\cygwin\bin\grep.exe Found: l:\cygwin\bin\kill.exe Found: l:\cygwin\bin\ld.exe Found: l:\cygwin\bin\ls.exe Found: l:\cygwin\bin\make.exe Found: l:\cygwin\bin\mv.exe Found: l:\cygwin\bin\patch.exe Not Found: perl Found: l:\cygwin\bin\rm.exe Found: l:\cygwin\bin\sed.exe Found: l:\cygwin\bin\ssh.exe Found: l:\cygwin\bin\sh.exe Found: l:\cygwin\bin\tar.exe Found: l:\cygwin\bin\test.exe Not Found: vi Found: l:\cygwin\bin\vim.exe 802k 2003/09/15 l:\cygwin\bin\cygaspell-15.dll - os=4.0 img=1.0 sys=4.0 "cygaspell-15.dll" v0.0 ts=2003/9/15 5:32 56k 2005/07/09 l:\cygwin\bin\cygbz2-1.dll - os=4.0 img=1.0 sys=4.0 "cygbz2-1.dll" v0.0 ts=2005/7/8 22:09 7k 2005/11/20 l:\cygwin\bin\cygcharset-1.dll - os=4.0 img=1.0 sys=4.0 "cygcharset-1.dll" v0.0 ts=2005/11/19 18:24 7k 2003/10/19 l:\cygwin\bin\cygcrypt-0.dll - os=4.0 img=1.0 sys=4.0 "cygcrypt-0.dll" v0.0 ts=2003/10/19 0:57 1108k 2005/10/17 l:\cygwin\bin\cygcrypto-0.9.7.dll - os=4.0 img=1.0 sys=4.0 "cygcrypto-0.9.7.dll" v0.0 ts=2005/10/17 2:16 1047k 2005/10/11 l:\cygwin\bin\cygcrypto-0.9.8.dll - os=4.0 img=1.0 sys=4.0 "cygcrypto-0.9.8.dll" v0.0 ts=2005/10/11 5:47 895k 2004/04/28 l:\cygwin\bin\cygdb-4.2.dll - os=4.0 img=1.0 sys=4.0 "cygdb-4.2.dll" v0.0 ts=2004/4/27 8:31 1156k 2004/04/28 l:\cygwin\bin\cygdb_cxx-4.2.dll - os=4.0 img=1.0 sys=4.0 "cygdb_cxx-4.2.dll" v0.0 ts=2004/4/27 8:35 174k 2004/10/14 l:\cygwin\bin\cygexpat-0.dll - os=4.0 img=1.0 sys=4.0 "cygexpat-0.dll" v0.0 ts=2004/10/14 1:34 40k 2005/09/29 l:\cygwin\bin\cygform-8.dll - os=4.0 img=1.0 sys=4.0 "cygform-8.dll" v0.0 ts=2005/9/28 19:15 45k 2001/04/25 l:\cygwin\bin\cygform5.dll - os=4.0 img=1.0 sys=4.0 "cygform5.dll" v0.0 ts=2001/4/24 22:28 35k 2002/01/09 l:\cygwin\bin\cygform6.dll - os=4.0 img=1.0 sys=4.0 "cygform6.dll" v0.0 ts=2002/1/8 22:03 48k 2003/08/09 l:\cygwin\bin\cygform7.dll - os=4.0 img=1.0 sys=4.0 "cygform7.dll" v0.0 ts=2003/8/9 2:25 28k 2003/07/20 l:\cygwin\bin\cyggdbm-3.dll - os=4.0 img=1.0 sys=4.0 "cyggdbm-3.dll" v0.0 ts=2003/7/20 0:58 30k 2003/08/11 l:\cygwin\bin\cyggdbm-4.dll - os=4.0 img=1.0 sys=4.0 "cyggdbm-4.dll" v0.0 ts=2003/8/10 19:12 19k 2003/03/22 l:\cygwin\bin\cyggdbm.dll - os=4.0 img=1.0 sys=4.0 "cyggdbm.dll" v0.0 ts=2002/2/19 19:05 15k 2003/07/20 l:\cygwin\bin\cyggdbm_compat-3.dll - os=4.0 img=1.0 sys=4.0 "cyggdbm_compat-3.dll" v0.0 ts=2003/7/20 1:00 15k 2003/08/11 l:\cygwin\bin\cyggdbm_compat-4.dll - os=4.0 img=1.0 sys=4.0 "cyggdbm_compat-4.dll" v0.0 ts=2003/8/10 19:13 17k 2001/06/28 l:\cygwin\bin\cyghistory4.dll - os=4.0 img=1.0 sys=4.0 "cyghistory4.dll" v0.0 ts=2001/1/6 20:34 29k 2003/08/10 l:\cygwin\bin\cyghistory5.dll - os=4.0 img=1.0 sys=4.0 "cyghistory5.dll" v0.0 ts=2003/8/10 16:16 24k 2006/02/18 l:\cygwin\bin\cyghistory6.dll - os=4.0 img=1.0 sys=4.0 "cyghistory6.dll" v0.0 ts=2006/2/17 22:03 947k 2005/11/20 l:\cygwin\bin\cygiconv-2.dll - os=4.0 img=1.0 sys=4.0 "cygiconv-2.dll" v0.0 ts=2005/11/19 18:24 22k 2001/12/13 l:\cygwin\bin\cygintl-1.dll - os=4.0 img=1.0 sys=4.0 "cygintl-1.dll" v0.0 ts=2001/12/13 1:28 37k 2003/08/10 l:\cygwin\bin\cygintl-2.dll - os=4.0 img=1.0 sys=4.0 "cygintl-2.dll" v0.0 ts=2003/8/10 14:50 31k 2005/11/20 l:\cygwin\bin\cygintl-3.dll - os=4.0 img=1.0 sys=4.0 "cygintl-3.dll" v0.0 ts=2005/11/19 18:04 21k 2001/06/20 l:\cygwin\bin\cygintl.dll - os=4.0 img=1.0 sys=4.0 "cygintl.dll" v0.0 ts=2001/6/20 10:09 48k 2005/11/19 l:\cygwin\bin\cygmagic-1.dll - os=4.0 img=1.0 sys=4.0 "cygmagic-1.dll" v0.0 ts=2005/11/19 3:22 21k 2005/09/29 l:\cygwin\bin\cygmenu-8.dll - os=4.0 img=1.0 sys=4.0 "cygmenu-8.dll" v0.0 ts=2005/9/28 19:15 26k 2001/04/25 l:\cygwin\bin\cygmenu5.dll - os=4.0 img=1.0 sys=4.0 "cygmenu5.dll" v0.0 ts=2001/4/24 22:27 20k 2002/01/09 l:\cygwin\bin\cygmenu6.dll - os=4.0 img=1.0 sys=4.0 "cygmenu6.dll" v0.0 ts=2002/1/8 22:03 29k 2003/08/09 l:\cygwin\bin\cygmenu7.dll - os=4.0 img=1.0 sys=4.0 "cygmenu7.dll" v0.0 ts=2003/8/9 2:25 21k 2004/10/22 l:\cygwin\bin\cygminires.dll - os=4.0 img=1.0 sys=4.0 "cygminires.dll" v0.0 ts=2004/10/22 13:28 67k 2005/09/29 l:\cygwin\bin\cygncurses++-8.dll - os=4.0 img=1.0 sys=4.0 "cygncurses++-8.dll" v0.0 ts=2005/9/28 19:16 156k 2001/04/25 l:\cygwin\bin\cygncurses++5.dll - os=4.0 img=1.0 sys=4.0 "cygncurses++5.dll" v0.0 ts=2001/4/24 22:29 175k 2002/01/09 l:\cygwin\bin\cygncurses++6.dll - os=4.0 img=1.0 sys=4.0 "cygncurses++6.dll" v0.0 ts=2002/1/8 22:03 226k 2005/09/29 l:\cygwin\bin\cygncurses-8.dll - os=4.0 img=1.0 sys=4.0 "cygncurses-8.dll" v0.0 ts=2005/9/28 19:02 226k 2001/04/25 l:\cygwin\bin\cygncurses5.dll - os=4.0 img=1.0 sys=4.0 "cygncurses5.dll" v0.0 ts=2001/4/24 22:17 202k 2002/01/09 l:\cygwin\bin\cygncurses6.dll - os=4.0 img=1.0 sys=4.0 "cygncurses6.dll" v0.0 ts=2002/1/8 22:03 224k 2003/08/09 l:\cygwin\bin\cygncurses7.dll - os=4.0 img=1.0 sys=4.0 "cygncurses7.dll" v0.0 ts=2003/8/9 2:24 11k 2005/09/29 l:\cygwin\bin\cygpanel-8.dll - os=4.0 img=1.0 sys=4.0 "cygpanel-8.dll" v0.0 ts=2005/9/28 19:15 15k 2001/04/25 l:\cygwin\bin\cygpanel5.dll - os=4.0 img=1.0 sys=4.0 "cygpanel5.dll" v0.0 ts=2001/4/24 22:27 12k 2002/01/09 l:\cygwin\bin\cygpanel6.dll - os=4.0 img=1.0 sys=4.0 "cygpanel6.dll" v0.0 ts=2002/1/8 22:03 19k 2003/08/09 l:\cygwin\bin\cygpanel7.dll - os=4.0 img=1.0 sys=4.0 "cygpanel7.dll" v0.0 ts=2003/8/9 2:24 176k 2005/09/06 l:\cygwin\bin\cygpcre-0.dll - os=4.0 img=1.0 sys=4.0 "cygpcre-0.dll" v0.0 ts=2005/9/6 13:49 299k 2005/09/06 l:\cygwin\bin\cygpcrecpp-0.dll - os=4.0 img=1.0 sys=4.0 "cygpcrecpp-0.dll" v0.0 ts=2005/9/6 14:26 6k 2005/09/06 l:\cygwin\bin\cygpcreposix-0.dll - os=4.0 img=1.0 sys=4.0 "cygpcreposix-0.dll" v0.0 ts=2005/9/6 14:26 22k 2002/06/09 l:\cygwin\bin\cygpopt-0.dll - os=4.0 img=1.0 sys=4.0 "cygpopt-0.dll" v0.0 ts=2002/6/8 22:45 108k 2001/06/28 l:\cygwin\bin\cygreadline4.dll - os=4.0 img=1.0 sys=4.0 "cygreadline4.dll" v0.0 ts=2001/1/6 20:34 148k 2003/08/10 l:\cygwin\bin\cygreadline5.dll - os=4.0 img=1.0 sys=4.0 "cygreadline5.dll" v0.0 ts=2003/8/10 16:16 152k 2006/02/18 l:\cygwin\bin\cygreadline6.dll - os=4.0 img=1.0 sys=4.0 "cygreadline6.dll" v0.0 ts=2006/2/17 22:03 231k 2005/10/17 l:\cygwin\bin\cygssl-0.9.7.dll - os=4.0 img=1.0 sys=4.0 "cygssl-0.9.7.dll" v0.0 ts=2005/10/17 2:16 215k 2005/10/11 l:\cygwin\bin\cygssl-0.9.8.dll - os=4.0 img=1.0 sys=4.0 "cygssl-0.9.8.dll" v0.0 ts=2005/10/11 5:47 247k 2004/12/07 l:\cygwin\bin\cygtidy-0-99-0.dll - os=4.0 img=1.0 sys=4.0 "cygtidy-0-99-0.dll" v0.0 ts=2004/12/7 6:34 65k 2005/08/23 l:\cygwin\bin\cygz.dll - os=4.0 img=1.0 sys=4.0 "cygz.dll" v0.0 ts=2005/8/22 19:03 1763k 2006/01/20 l:\cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0 "cygwin1.dll" v0.0 ts=2006/1/20 10:28 Cygwin DLL version info: DLL version: 1.5.19 DLL epoch: 19 DLL bad signal mask: 19005 DLL old termios: 5 DLL malloc env: 28 API major: 0 API minor: 150 Shared data: 4 DLL identifier: cygwin1 Mount registry: 2 Cygnus registry name: Cygnus Solutions Cygwin registry name: Cygwin Program options name: Program Options Cygwin mount registry name: mounts v2 Cygdrive flags: cygdrive flags Cygdrive prefix: cygdrive prefix Cygdrive default prefix: Build date: Fri Jan 20 13:28:43 EST 2006 CVS tag: cr-0x5ef Shared id: cygwin1S4 No Cygwin services found. Cygwin Package Information Last downloaded files to: U:\j\win-general\gnu\cygwin Last downloaded files from: http://mirrors.xmission.com/cygwin Package Version _update-info-dir 00381-1 alternatives 1.3.20a-2 ash 20040127-3 aspell 0.50.3-1 aspell-en 0.51.0-1 base-files 3.7-1 base-passwd 2.2-1 bash 3.0-14 bc 1.06-2 binutils 20050610-1 bison 2.1-1 bzip2 1.0.3-1 coreutils 5.94-1 crypt 1.1-1 cvs 1.11.17-1 cygrunsrv 1.12-1 cygutils 1.2.10-1 cygwin 1.5.19-4 cygwin-doc 1.4-3 diffutils 2.8.7-1 editrights 1.01-1 expat 1.95.8-1 file 4.16-1 findutils 4.2.27-1 fortune 1.99.1-1 gawk 3.1.5-3 gcc 3.4.4-1 gcc-core 3.4.4-1 gcc-g++ 3.4.4-1 gcc-gdc 3.4.4-1 gcc-mingw-core 20050522-1 gcc-mingw-g++ 20050522-1 gcc-mingw-gdc 20050522-1 gdb 20041228-3 gdbm 1.8.3-7 grep 2.5.1a-2 groff 1.18.1-2 gzip 1.3.5-1 indent 2.2.9-1 less 381-1 libaspell15 0.50.3-1 libbz2_1 1.0.3-1 libcharset1 1.9.2-2 libdb4.2 4.2.52-1 libgdbm 1.8.0-5 libgdbm-devel 1.8.3-7 libgdbm3 1.8.3-3 libgdbm4 1.8.3-7 libiconv 1.9.2-2 libiconv2 1.9.2-2 libintl 0.10.38-3 libintl1 0.10.40-1 libintl2 0.12.1-3 libintl3 0.14.5-1 libncurses5 5.2-1 libncurses6 5.2-8 libncurses7 5.3-4 libncurses8 5.4-4 libpcre0 6.3-1 libpopt0 1.6.4-4 libreadline4 4.1-2 libreadline5 4.3-5 libreadline6 5.1-4 login 1.9-7 m4 1.4.4-1 make 3.80-1 man 1.5p-1 mingw-runtime 3.9-2 minires 1.00-1 mktemp 1.5-3 ncurses 5.4-4 openssh 4.3p2-2 openssl 0.9.8a-1 openssl097 0.9.7i-1 patch 2.5.8-8 patchutils 0.2.31-1 python 2.4.1-1 run 1.1.7-1 rxvt 2.7.10-6 sed 4.1.5-1 smartmontools 5.33cvs20051205-1 tar 1.15.1-4 tcltk 20030901-1 tcsh 6.14.00-5 termcap 20050421-1 terminfo 5.4_20041009-1 texinfo 4.8-1 tidy 041206-1 time 1.7-1 typespeed 0.4.4-2 vim 6.4-4 w32api 3.6-1 which 1.7-1 whois 4.7.11-1 zlib 1.2.3-1 Use -h to see help about each section --=====================_68951171==_ Content-Type: text/plain; charset=us-ascii -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ --=====================_68951171==_--