Mail Archives: cygwin/2024/04/22/08:51:54

Johannes Khoshnazar-Thoma via Cygwin <cygwin AT cygwin DOT com> · Mon, 22 Apr 2024 14:50:51 +0200

Hi cygwin team :)

I have found something what may be a cygwin bug. Sometimes
(1 out of 1000 times) a drbdadm.exe process (which is part of WinDRBD's
user mode programs originally written for Linux) hangs for several
days on exiting (closing the console). I got a minidump and analyzed
the dump with gdb (the minidump enabled version at https://github.com/ssbssa/gdb)

I have to note that the application (drbdadm.exe) is run from a
Windows Service. Furthermore it is not a full cygwin installation
however the cygwin1.dll (3.4.10-1) is in the $PATH.

here is the gdb output:

(gdb) info thread
   Id   Target Id         Frame
* 1    Thread 0x14c8     0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
   2    Thread 0x2e0      0x00007ffc6a405ee4 in ntdll!ZwReadFile () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
   3    Thread 0x1060     0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
   4    Thread 0x858      0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
   5    Thread 0xe40      0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
(gdb) thread 1
[Switching to thread 1 (Thread 0x14c8)]
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
(gdb) bt
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#1  0x00007ffc66826d1f in WaitForSingleObjectEx () from C:\src\dlls-syms\KERNELBASE.dll\6502815021e000\KERNELBASE.dll
#2  0x00007ffc492ebb42 in fhandler_console::close (this=0x800007f20) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/fhandler/console.cc:1769
#3  0x00007ffc492dc365 in fhandler_base::close_with_arch (this=0x800007f20) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/fhandler/base.cc:1202
#4  0x00007ffc49353640 in init_cygheap::close_ctty (this=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/mm/cygheap.cc:133
#5  0x00007ffc492aee9f in close_all_files (norelease=norelease AT entry=false) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/syscalls.cc:81
#6  0x00007ffc49226763 in do_exit (status=0) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/dcrt0.cc:1125
#7  0x00007ffc4922693f in _exit (n=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/dcrt0.cc:1240
#8  0x00007ffc49390899 in exit (code=0) at /usr/src/debug/cygwin-3.4.10-1/newlib/libc/stdlib/exit.c:65
#9  0x00007ffc49226923 in cygwin_exit (n=568) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/dcrt0.cc:1234
#10 0x00007ffc492280aa in dll_crt0_1 () at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/dcrt0.cc:999
#11 0x00007ffc49225c86 in _cygtls::call2 (this=0x7ffffce00, func=0x7ffc49226f80 <dll_crt0_1(void*)>, arg=0x0, buf=buf AT entry=0x7ffffcdf0) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:41
#12 0x00007ffc49225d34 in _cygtls::call (func=<optimized out>, arg=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:28
#13 0x0000000000000000 in ?? ()
(gdb) thread 1
[Switching to thread 1 (Thread 0x14c8)]
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
(gdb) info thread
   Id   Target Id         Frame
* 1    Thread 0x14c8     0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
   2    Thread 0x2e0      0x00007ffc6a405ee4 in ntdll!ZwReadFile () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
   3    Thread 0x1060     0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
   4    Thread 0x858      0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
   5    Thread 0xe40      0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
(gdb) bt
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#1  0x00007ffc66826d1f in WaitForSingleObjectEx () from C:\src\dlls-syms\KERNELBASE.dll\6502815021e000\KERNELBASE.dll
#2  0x00007ffc492ebb42 in fhandler_console::close (this=0x800007f20) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/fhandler/console.cc:1769
#3  0x00007ffc492dc365 in fhandler_base::close_with_arch (this=0x800007f20) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/fhandler/base.cc:1202
#4  0x00007ffc49353640 in init_cygheap::close_ctty (this=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/mm/cygheap.cc:133
#5  0x00007ffc492aee9f in close_all_files (norelease=norelease AT entry=false) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/syscalls.cc:81
#6  0x00007ffc49226763 in do_exit (status=0) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/dcrt0.cc:1125
#7  0x00007ffc4922693f in _exit (n=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/dcrt0.cc:1240
#8  0x00007ffc49390899 in exit (code=0) at /usr/src/debug/cygwin-3.4.10-1/newlib/libc/stdlib/exit.c:65
#9  0x00007ffc49226923 in cygwin_exit (n=568) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/dcrt0.cc:1234
#10 0x00007ffc492280aa in dll_crt0_1 () at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/dcrt0.cc:999
#11 0x00007ffc49225c86 in _cygtls::call2 (this=0x7ffffce00, func=0x7ffc49226f80 <dll_crt0_1(void*)>, arg=0x0, buf=buf AT entry=0x7ffffcdf0) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:41
#12 0x00007ffc49225d34 in _cygtls::call (func=<optimized out>, arg=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:28
#13 0x0000000000000000 in ?? ()
(gdb) thread 2
[Switching to thread 2 (Thread 0x2e0)]
#0  0x00007ffc6a405ee4 in ntdll!ZwReadFile () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
(gdb) bt
#0  0x00007ffc6a405ee4 in ntdll!ZwReadFile () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#1  0x00007ffc6682e704 in ReadFile () from C:\src\dlls-syms\KERNELBASE.dll\6502815021e000\KERNELBASE.dll
#2  0x00007ffc4929f541 in wait_sig () at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/sigproc.cc:1331
#3  0x00007ffc49224aa3 in cygthread::callfunc (this=this AT entry=0x7ffc49416880 <threads>, issimplestub=issimplestub AT entry=false) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygthread.cc:48
#4  0x00007ffc49225066 in cygthread::stub (arg=arg AT entry=0x7ffc49416880 <threads>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygthread.cc:91
#5  0x00007ffc49225c86 in _cygtls::call2 (this=0x10ece00, func=0x7ffc49224fe0 <cygthread::stub(void*)>, arg=0x7ffc49416880 <threads>, buf=buf AT entry=0x10ecd50) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:41
#6  0x00007ffc49225d34 in _cygtls::call (func=<optimized out>, arg=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:28
#7  0x00007ffc691584d4 in KERNEL32!BaseThreadInitThunk () from C:\src\dlls-syms\kernel32.dll\640993B0ad000\kernel32.dll
#8  0x00007ffc6a3b1791 in ntdll!RtlUserThreadStart () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#9  0x0000000000000000 in ?? ()
(gdb) thread 3
[Switching to thread 3 (Thread 0x1060)]
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
(gdb) bt
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#1  0x00007ffc66826d1f in WaitForSingleObjectEx () from C:\src\dlls-syms\KERNELBASE.dll\6502815021e000\KERNELBASE.dll
#2  0x00007ffc492250a1 in cygthread::stub (arg=arg AT entry=0x7ffc494168d8 <threads+88>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygthread.cc:112
#3  0x00007ffc49225c86 in _cygtls::call2 (this=0x19bce00, func=0x7ffc49224fe0 <cygthread::stub(void*)>, arg=0x7ffc494168d8 <threads+88>, buf=buf AT entry=0x19bcd50) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:41
#4  0x00007ffc49225d34 in _cygtls::call (func=<optimized out>, arg=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:28
#5  0x00007ffc691584d4 in KERNEL32!BaseThreadInitThunk () from C:\src\dlls-syms\kernel32.dll\640993B0ad000\kernel32.dll
#6  0x00007ffc6a3b1791 in ntdll!RtlUserThreadStart () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#7  0x0000000000000000 in ?? ()
(gdb) thread 4
[Switching to thread 4 (Thread 0x858)]
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
(gdb) bt
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#1  0x00007ffc66826d1f in WaitForSingleObjectEx () from C:\src\dlls-syms\KERNELBASE.dll\6502815021e000\KERNELBASE.dll
#2  0x00007ffc492250a1 in cygthread::stub (arg=arg AT entry=0x7ffc49416930 <threads+176>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygthread.cc:112
#3  0x00007ffc49225c86 in _cygtls::call2 (this=0x22fce00, func=0x7ffc49224fe0 <cygthread::stub(void*)>, arg=0x7ffc49416930 <threads+176>, buf=buf AT entry=0x22fcd50) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:41
#4  0x00007ffc49225d34 in _cygtls::call (func=<optimized out>, arg=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:28
#5  0x00007ffc691584d4 in KERNEL32!BaseThreadInitThunk () from C:\src\dlls-syms\kernel32.dll\640993B0ad000\kernel32.dll
#6  0x00007ffc6a3b1791 in ntdll!RtlUserThreadStart () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#7  0x0000000000000000 in ?? ()
(gdb) thread 5
[Switching to thread 5 (Thread 0xe40)]
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
(gdb) bt
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#1  0x00007ffc66826d1f in WaitForSingleObjectEx () from C:\src\dlls-syms\KERNELBASE.dll\6502815021e000\KERNELBASE.dll
#2  0x00007ffc492250a1 in cygthread::stub (arg=arg AT entry=0x7ffc49416988 <threads+264>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygthread.cc:112
#3  0x00007ffc49225c86 in _cygtls::call2 (this=0x24fce00, func=0x7ffc49224fe0 <cygthread::stub(void*)>, arg=0x7ffc49416988 <threads+264>, buf=buf AT entry=0x24fcd50) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:41
#4  0x00007ffc49225d34 in _cygtls::call (func=<optimized out>, arg=<optimized out>) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/cygtls.cc:28
#5  0x00007ffc691584d4 in KERNEL32!BaseThreadInitThunk () from C:\src\dlls-syms\kernel32.dll\640993B0ad000\kernel32.dll
#6  0x00007ffc6a3b1791 in ntdll!RtlUserThreadStart () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
#7  0x0000000000000000 in ?? ()
(gdb) p master_thread_started
$7 = true
(gdb) thread 1
[Switching to thread 1 (Thread 0x14c8)]
#0  0x00007ffc6a405ea4 in ntdll!ZwWaitForSingleObject () from C:\src\dlls-syms\ntdll.dll\6502806A1cf000\ntdll.dll
(gdb) up
#1  0x00007ffc66826d1f in WaitForSingleObjectEx () from C:\src\dlls-syms\KERNELBASE.dll\6502815021e000\KERNELBASE.dll
(gdb)
#2  0x00007ffc492ebb42 in fhandler_console::close (this=0x800007f20) at /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/fhandler/console.cc:1769
1769    in /usr/src/debug/cygwin-3.4.10-1/winsup/cygwin/fhandler/console.cc
(gdb) p name
$8 = "cygcons.thread_sync.0", '\000' <repeats 19 times>, "r", '\000' <repeats 151 times>, "^8\025\r!‘\000\000"...
(gdb) p myself
$9 = {<pinfo_minimal> = {h = 0x118, hProcess = 0x0, rd_proc_pipe = 0x0}, destroy = false, winpid_hdl = 0x11c, procinfo = 0x1a2000000, waiter_ready = false, wait_thread = 0x0}

It looks like the cons_master_thread already exited, maybe clearing the master_thread_started
flag in that case helps? but this is only a wild guess ...

If you need more information, please let me know.

Thanks a lot and Best regards,

- Johannes

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org CA0513858C78
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1713790311;
	bh=bAKrKqQVkMK/ZoVNRC/J5oK4h077BtVsWF35zOeHyso=;
	h=Date:To:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post:
	List-Help:List-Subscribe:From:Reply-To:From;
	b=xbAmei856FTj/Ne/b6Gix8Dt6+4CFdk/RQJkyAkyauiFx5WrgaekghrwxC2SZvSac
	MZ+9mruR79jpqD6HOx3GgbGqS/VXEg6x6ppraea1DB+eD03kSEEEjsItTf9syni8mx
	wyhVhozYGWg/30JzjZA0lOvHAWf34YpvysU0n11U=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.2 sourceware.org E29623858D35
ARC-Filter:	OpenARC Filter v1.0.0 sourceware.org E29623858D35
ARC-Seal:	i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1713790257; cv=none;
	b=pYsqcWBpthgS/F7Gw2JL5K0vwl0vW5adg0B5dJ9Ot4adc1KWb6sQn8HU/F50BMoQ9fimWc/L/tEncT386N0Q7ltBPzp4fqgsP/F+pCs/em9HPngoy7iyyXRaSoEX4zwZEIwzkzITorSnIrQtwNjpuxBtxblIoyrzeLUmFvMpH5c=
ARC-Message-Signature:	i=1; a=rsa-sha256; d=sourceware.org; s=key;
	t=1713790257; c=relaxed/simple;
	bh=tYGThGdCOng5TDZ9dBuFDl5qFGWXi/zfSeUr4ajXECM=;
	h=DKIM-Signature:Message-ID:Date:MIME-Version:To:From:Subject;
	b=aroSU5LTZZQMWibsR2DdAbEILrcnosjNowezCg1O8FGgUqoToklOCmTX46MR3Rc5NzUjKsJvs8mkQJjX4pkpBBFXcPRnJhInKudvDL4vhPARTCzarXzxJTD6ZMa9mIjQIVLQ4wD3WEmfrz1C5t3N0oI7IwGyc540LSiWE/vbfsc=
ARC-Authentication-Results:	i=1; server2.sourceware.org
X-UI-Sender-Class:	55c96926-9e95-11ee-ae09-1f7a4046a0f6
Message-ID:	<d268ca90-3c5b-4de6-9dd5-320ee713283c@johannesthoma.com>
Date:	Mon, 22 Apr 2024 14:50:51 +0200
MIME-Version:	1.0
User-Agent:	Mozilla Thunderbird
To:	The Cygwin Mailing List <cygwin AT cygwin DOT com>
Subject:	cygwin application hangs on closing console
X-Provags-ID:	V03:K1:fn3awpP7pCerRoCaiouKdijCUn9PwwgnnsKuTqs6CzV+uaSftcB
	wxn3x9gI0ii8JIssAd2AYDanNgSHHz6M98FSQ3WMBCLqARHEvs2KY69VCkZv3KMHAqIpmcz
	vuDwxlteYtCBa0Nrn2vHzxb0zQREaoZxPbBoDlmww8opoRvM0uKmGamQ/Me24MSBHeX5PAM
	f7xOM9Z0zpovVVm6qdYEA==
UI-OutboundReport:	notjunk:1;M01:P0:VcQxxBnwABU=;XXBScN6mb2YTNFZPd01CdipK9dv
	ClH15+CGQ1aXDRtm74BTteIeWgp1Y9nZ1DUcPWghsO5xb4V39YQvDrvH/PZhkJL5Cl8eZC/Rg
	rExq6TJ6/feyw4ApcGUYx9RM+4XcVqu1UTDrFov3cav5p9Z5Ko3FLT3UvVnTAeHNUXsgANUwn
	9WTf+N8M5U3BH/2d3KwnLtnTGzsl+sS4n7SX/vR+P0cDxVLakTsGErUfh2gq+63TZtQstUJrM
	CY09XMi5pHSFFoirrgLP1PycEbz0ZYo2typS1f6saI0YXxRKQQd6aTb95b7cDNN58OV3H83Ig
	0aY4h8a06WvEiD/CGIU5taHok55dHmHjd1dyDL4efANg+cbAoX+8IrkR7kHG8kAWTF3bkgxE/
	XqKz4zEwXkZ6hUnKrZyBi6fgL4Ou8a/AFE7dSxe4rm5jokzqSFOIePUi/6BoRYrgM8yKr5dFk
	8rB8hDpepLcfc2Ks0SWW8WUxe88UcON8tgM2HNbYaewBwydY9TrkU4TUhhHvNZB2yNb4Aga8H
	mD15HIx4XiZ5rZyBE7tV5nPduPXmmmvU7UOjCkjTkh/CFHGucL+eTXregVVroawyigNRCo+v4
	186fycTy89wJOD+vlVibAjs+PPwbScIzQLRMm3N6VR9U7j7stGMgpN8P8/IQzvFmCjGwOG0la
	kFoD1J9mzV+wwu14nKFwB8ET1FYfcoWIuxK0BOoJ2MJajTKZgRlW9TTiMK1RUgHW4qgnsafrE
	6fXQ/8JvA/Z4dtpAPmcKNkNeQKcYV7MW+1wnF+o4ciY1aPz9dy9kDs=
X-Spam-Status:	No, score=-2.0 required=5.0 tests=BAYES_00, DKIM_SIGNED,
	DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE,
	RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_PASS, TXREP, URIBL_SBL_A,
	WEIRD_PORT autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version:	SpamAssassin 3.4.6 (2021-04-09) on
	server2.sourceware.org
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.30
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	Johannes Khoshnazar-Thoma via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	Johannes Khoshnazar-Thoma <johannes AT johannesthoma DOT com>
Sender:	"Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
X-MIME-Autoconverted:	from base64 to 8bit by delorie.com id 43MCpssA695975

webmaster	delorie software privacy
Copyright � 2019 by DJ Delorie	Updated Jul 2019