Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner AT cygwin DOT com Mail-Followup-To: cygwin AT cygwin DOT com Delivered-To: mailing list cygwin AT cygwin DOT com From: "Dave Korn" To: Subject: Bug identified [was RE: perl - segfault on "free unused scalar"] Date: Thu, 28 Jul 2005 14:45:53 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit In-Reply-To: Message-ID: ----Original Message---- >From: Krzysztof Duleba >Sent: 28 July 2005 08:00 > Igor Pechtchanski wrote: >> On Thu, 28 Jul 2005, Krzysztof Duleba wrote: >> >>> I am not. I understand that this is how it should work theoretically, >>> but I've _checked_ that on a couple of Cygwin boxes with different >>> versions of cygwin1.dll and gcc. All of them didn't really care that >>> heap_chunk_in_mb was undefined in the registry. Perl, on the other hand, >>> do care. >> >> Actually, you're right. Perhaps it depends on what kind of malloc the >> program uses (i.e., whether it uses the Cygwin builtin malloc, or >> something else). > > That's strange. How could Perl use something essentially different than > malloc? I thought it would all come down to brk or sbrk. > > Krzysztof Duleba There's a real bug here. I think it could be a consequence of the recent cygload changes, but OTOH it could have already been there: I haven't looked at the details of that patch, or the cvs history of that part of the code, but we're in the same area. Here's how it goes: Regardless of their malloc implementations, Perl and C both rely on Cygwin's sbrk(...) implementation. The code that grows the heap looks like this: if ((VirtualAlloc (cygheap->user_heap.top, newbrksize, MEM_RESERVE, PAGE_NOACCESS) || VirtualAlloc (cygheap->user_heap.top, newbrksize = commitbytes, MEM_RESERVE, PAGE_NOACCESS)) && VirtualAlloc (cygheap->user_heap.top, commitbytes, MEM_COMMIT, PAGE_READWRITE) != NULL) { cygheap->user_heap.max = (char *) cygheap->user_heap.max + pround (newbrksize); goto good; } Here, cygwin is trying to extend the heap by allocating the maximum of (requested size, heap_chunk_in_mb) bytes of memory contiguously to the end of the current heap (or the minimum of the two if that fails), and it is this that is failing in the perl case, and succeeding in the case of compiled C code; I stepped through it in insight and watched the calls fail. This must be because there is something allocated by perl that fragments the process' memory map that isn't allocated when running a C program. I've been on the track of it. By running that perl example ------------------------------------------------ perl -e '$a="a"x(200 * 1024 * 1024);' ------------------------------------------------ under windbg, and breakpointing VirtualAlloc, I was able to intercept the call that tries to extend the heap: here's the stack, showing the args to the function call ------------------------------------------------ 0:000> dd esp 0022eba0 204d6000 10001000 00002000 00000001 0022ebb0 101255f0 10243bdc 0022ec78 0022eb90 0022ebc0 00000000 100017ff 304d7000 00000003 ------------------------------------------------ and here's what the output from !vadump showed in that area: ------------------------------------------------ BaseAddress: 204d6000 RegionSize: 07c6a000 State: 00002000 MEM_RESERVE Type: 00020000 MEM_PRIVATE BaseAddress: 28140000 RegionSize: 00001000 State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE BaseAddress: 28141000 RegionSize: 37ebf000 State: 00010000 MEM_FREE Protect: 00000001 PAGE_NOACCESS ------------------------------------------------ clearly showing that there's ~900meg of contiguous free space after the heap except for a single page allocated at 0x28140000, thus preventing it from growing any larger than (0x28140000 - 0x204d6000) ~= 125meg. So, rerunning the example under insight again, I breakpointed VirtualAlloc, and looked for a call that was allocating a small amount of memory at some address beginning 0x2???????. Here's what I found: ------------------------------------------------ (gdb) c Continuing. Breakpoint 7, 0x77e7abc5 in VirtualAlloc () from /win/c/WINDOWS/system32/kernel32.dll (gdb) x/32xw $esp+4 0x22f82c: 0x28140000 0x0000013c 0x00002000 0x00000004 0x22f83c: 0x00240178 0x002429e0 0x77fa88f0 0x00010000 0x22f84c: 0xffffffff 0x7ffde000 0x0000001c 0x610f2350 0x22f85c: 0x28140000 0x00000000 0x00000000 0x37ec0000 0x22f86c: 0x00010000 0x00000001 0x00000000 0x00401088 0x22f87c: 0x00000000 0x00001000 0x00010000 0x7ffeffff 0x22f88c: 0x00000001 0x00000001 0x0000024a 0x00010000 0x22f89c: 0x0209000f 0x00000000 0x002429d8 0x00240000 (gdb) bt #0 0x77e7abc5 in VirtualAlloc () from /win/c/WINDOWS/system32/kernel32.dll #1 0x61010b29 in dll_list::alloc (this=0x610f2350, h=0x10000000, p=0x10125520, type=DLL_LINK) at /usr/build/src/winsup/cygwin/dll_init.cc:147 #2 0x6101110a in dll_dllcrt0 (h=0x10000000, p=0x10125520) at /usr/build/src/winsup/cygwin/dll_init.cc:382 #3 0x10108e78 in init_os_extras () from /usr/bin/cygperl5_8.dll #4 0x77f5b42c in ntdll!LdrSetAppCompatDllRedirectionCallback () from ntdll.dll ------------------------------------------------ So it's something to do with dll_list::alloc. This function tries to allocate space immediately after a dll for some kind of control struct. It does so by walking the process space after the dll's bss section looking for an unallocated chunk. In this particular case, it's trying to process cygperl5_8.dll, which has a default load address of 0x10000000: ------------------------------------------------ 10000000 1013f000 cygperl5_8 (deferred) ------------------------------------------------ So, it's a consequence of having a dll with such a low preferred image base. The rough order of events is: that dll gets loaded down at that low address; then the initial heap_chunk_in_mb size heap gets allocated and ends up being after the dll; then dll_list::alloc gets in, starts walking after the end of the dll, skips over the cygwin heap to the first bit of unallocated memory and allocates itself a single page there. This means that much later, when cygwin wants to extend the heap, there's already something there, so it can't. I don't have a patch yet: I need to understand what's going on with this code a bit better first, but ISTM you should be able to work around this problem and get lots more memory available for perl by rebasing the cygperl5_8.dll. This post has been brought to you by the numbers 3 and 0x28140000 and the letters `perl -e 'print "a"x(200 * 1024 * 1024)'`! cheers, DaveK -- Can't think of a witty .sigline today.... -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/