Message-ID: <3DB5D1E6.648F14BF@yahoo.com> Date: Tue, 22 Oct 2002 18:32:07 -0400 From: CBFalconer Organization: Ched Research X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: djgpp-workers AT delorie DOT com Subject: Re: CBFalconer's malloc References: <3DB3D5D9 DOT F1DA8C95 AT yahoo DOT com> <9003-Tue22Oct2002212734+0200-eliz AT is DOT elta DOT co DOT il> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Reply-To: djgpp-workers AT delorie DOT com I am trying to make sure that this exchange contains a list of things needed to be done, so I have unsnipped a portion. Please keep it a massive whole, so I can eventually use it for review. I still have to go for the .h files, nor have I looked for malloc.txh. Another useful thing would be an example of a debugging package using all this. I have limited time, my wife seems to be amazingly able to find things other than programming for me to do :-) Eli Zaretskii wrote: > CBFalconer wrote: > > Eli Zaretskii wrote: > > > On Mon, 21 Oct 2002, CBFalconer wrote: > > > > > > > > The following external interfaces are missing: > > > > > mallinfo > > > > > malloc_debug > > > > > malloc_verify > > > > > mallocmap > > > > > __libc_malloc_hook > > > > > __libc_malloc_fail_hook > > > > > __libc_free_hook > > > > > __libc_free_null_hook > > > > > __libc_realloc_hook > > > > > __malloc_get_freelist > > > > > __malloc_get_slop > > > > > __malloc_get_bytes_in_use > > > > > __malloc_get_chunks_in_use > > > > > __malloc_get_sbrked > > > > > > > > > > These will need to be added, or alternate debugging routines written and > > > > > documented. It appears retrofitting CBF's code with the above routines > > > > > would be the right thing to do (but is fairly substantial work). > > > > > > > > I made provisions for getting at the internal structure, and used > > > > an example of it in the dumps from the testing code. If you will > > > > give me the details of what those interfaces have to do I will see > > > > (no promises) how they can be generated. > > > > > > IIRC, all those interfaces, or at least those which are intended to be > > > for public use, are documented in the library reference (see malloc.txh > > > in CVS). > > > > I still have not gotten CVS installed here, much less learned to > > use it properly, especially from a remote source. > > > > > > The point to me is that > > > > any such interfaces are done through header files, and leave the > > > > malloc code itself free to do what it needs. > > > > > > I'm not sure I understand. The debugging hooks are called by malloc at > > > strategic places, provided that they are non-NULL (each hook is a pointer > > > to a function, by default set to NULL). malloc is free to do what it > > > needs, but it must call the hooks while it's doing that. > > > > Why do it within malloc? Isn't it easier to define say > > > > void * xmalloc(size_t sz) > > { > > void *p; > > > > xmentryhook(sz); > > p = malloc(sz); > > xmexithook(p); > > return p; > > } > > > > in the users space. > > Because the hooks don't just tell you when malloc is called. They > tell you something about the inner workings of malloc, so a program > which hooked into malloc can gather information about memory > allocation. > > For example, __malloc_get_freelist returns the free list of memory > blocks, an otherwise internal structure. An application can then > examine the free list for whatever it needs, like count the size of > the free memory pool. The _sysquery call in nmalloc does just that - it returns sufficient information to completely scan the free list. Having made the call to get the info, the user is free to scan that free list at any time. In fact he can usually (but not always) scan all allocated blocks also. He can also follow the hidden linkages from any allocated block, including its actual size, physical neighbors, etc. All this involves *NO* run-time overhead. See the code in tnmalloc.c > > Another example is __libc_malloc_fail_hook, which, if points to > something non-NULL, is called by malloc when it is about to fail the > call. An application that sets up such a hook can later print how > many allocations failed. __libc_malloc_hook is called whenever a > block is returned to an application, and the info about that block is > passed as arguments to the hook; this can be used to gather statistics > about memory allocations. Etc., etc. > > malldbg.c uses these hooks to implement a simple malloc debugging > and memory reporting package. How is that different from an external intercept (see the sample xmalloc call above)? It is perfectly free to collect statistics on the count of calls and the count of NULL returns. > > > Incorporating such pointers in the module will mean they > > have to be NULL initialized and tested > > True. I don't think a test against NULL is a significant overhead in > normal usage. > > > and non-standard routines will have to be present to set and reset > > them. > > Well, messing with internals of a library implementation is inherently > non-standard. Those hooks are meant to be used by memory allocation > debuggers and various malloc extensions, so standardization of the > internal names is not the issue. For example, malldbg.c implements > several fairly-standard functions using these facilities. > > > It also means that > > such routines as "free(NULL)" cannot simply do an early return, > > but must use the single return point. > > I don't see why. `free' simply does this: > > if (ptr == 0) > { > if (__libc_free_hook) > __libc_free_hook(); > return; > } > > So you again pay the price of a single comparison. > > > Or is name changing the operational problem? > > It's mainly a back compatibility problem. > > > At any rate, if you can ease my getting the descriptions of the > > interfaces, I can think about things much better, rather than > > making wild guesses. > > I'll happily help you understand teh current implementation. Just > ask questions here if the docs is not enough. > > > Just looking at the names above realloc_fail_hook seems to be > > missing. > > realloc fails if malloc fails, and the latter has a fail hook. Not in nmalloc. realloc does not call malloc, because it goes to lengths to avoid copying the data fields. However it does use the same subroutines that malloc uses. > > But I don't have anything against adding a reall-c-specific fail > hook, if that's what people want. > > > And what about calls that > > are rejected because the memory has been corrupted or free/realloc > > is passed an invalid pointer (some of which I detect and do a > > signal). > > If these are detected, we probably want additional hooks for these > specific failures. On advice here I made them use SIGABRT, which can sometimes be caught. For example, an attempt to free something not mallocd, or already in the free list, is not fatal if converted into a NOP rather than destruction of the arena, so I allowed a return from the signal to continue. On the other hand if the error was something like a fouled arena, a return from the signal will call exit(EXIT_FAILURE). Before signalling each of these errors emits an appropriate message to stderr. There there is no problem at all in adding a hook. The question would be before or after the signal?. If we install all these hooks I would probably modify _sysquery to also return a pointer to an array of hook pointers, and the user can then put whatever they want in there. Which pointer hooks what would be a matter of convention. It would be very vulnerable to poor programming. We could then supply precooked routines ready for inclusion in the user module to use those pointers to set/clear hooks. At least that would be my interface recommendation. There are only three routines in the package. Thus hooks are limited, for each, to entry, exit, failure, success, and abort. Free has no success or failure mechanism, but does have an abort. -- Chuck F (cbfalconer AT yahoo DOT com) (cbfalconer AT worldnet DOT att DOT net) Available for consulting/temporary embedded and systems. USE worldnet address!