Message-Id: <199711072108.IAA01257@mona.lexicon.net.au>
Comments: Authenticated sender is <sjmachin AT mail DOT lexicon DOT net DOT au>
From: "John Machin" <sjmachin AT lexicon DOT net DOT au>
To: Eli Zaretskii <eliz AT is DOT elta DOT co DOT il>, djgpp-workers AT delorie DOT com
Date: Sat, 8 Nov 1997 08:06:32 +1000
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: 7BIT
Subject: Re: malloc()
References: <199710311119 DOT WAA10318 AT mona DOT lexicon DOT net DOT au>
In-reply-to: <Pine.SUN.3.91.971102141823.11098G-100000@is>
Precedence: bulk

On  2 Nov 97 at 14:18, Eli Zaretskii wrote:
> On Fri, 31 Oct 1997, John Machin wrote:
> 
> > > Could you please write a test program along these lines, or find such
> > > a program written by somebody else?  It would be interesting to compare 
> > > the two malloc's with such a program.  Thanks.
> > > 
> > OK, I've been sucked in now :-)
> 
> Welcome aboard ;-).
> 
> > One thing we would need to agree on is how to measure wastage.
> 
> The best measure I can think of is to see when a given amount of
> memory is exhausted and paging begins.  That is, after all, what DJGPP
> users want to avoid at all costs, at least on MS-DOS (Windows is more
> complicated).  The issue of DPMI and virtual memory management is so
> complex that we need a very simple criteria, or we risk making this
> project a Ph.D.  Percentage of wasted memory is IMHO not good enough
> because it assumes too many things about how memory is used (e.g., it
> assumes that all allocated memory is paged in, which might not be
> correct in some cases).
[snip] 
> 
> Linking several programs that use memory extensively (cc1, cc1plus and
> emacs come to mind) with the two allocators and checking when do they
> run out of memory seems also a good idea.

Three ways of testing:

(a) real programs linked with different allocators, but:
(1) how to check when it runs out of real memory & starts swapping? 
(2) I'm not wildly excited about the idea of FTPing emacs source, 
compiling it etc, finding out how to make it actually do something, 
... [I have a 66MHz 486 with 8Mb memory, 420Mb HD, an Iomega ZIP 
drive, and a 28.8Kbps dial-up modem]

(b) simulations using synthetic random distributions, but: not 
realistic

(c) simulations using traces of malloc/free calls from real programs.

This happens in two steps:
(1) you link a special version of (say) emacs by including a file 
called (say) maltrace.o. This provides special versions of malloc etc 
that record in a file what your program actually does. You run the 
special emacs doing various realistic workloads, and save the 
trace files for step 2.
(2) you run a simulator that has been linked with xxmalloc.o 
(whichever allocator you want to test). The simulator reads a trace 
file, calls malloc/free/realloc etc as found in the trace file, 
measures real/virtual memory usage as it goes, and prints statistics 
at the end.

Some advantages: the traces can be zipped & sent around the world. 
E.g. Someone with a big machine & emacs knowledge could prepare some 
traces and send them to me to try on my 8mb machine. Someone 
developing proprietary software could send their traces to a 
developer of the next world's best malloc without having to disclose 
any of their trade secrets.

A disadvantage: Can't simulate exactly the real world e.g. creating 
child processes.

 ... anyhow, I have found such a package on the web, and am in the 
throes of DJGPP-ising it. I'll report progress later.


> 
> Again, I think this thread should move to djgpp-workers list.

OK, but do I subscribe or is it "invitation only" :-) ... I'll try 
subscribing & see what happens.

John Machin
1/27 Auburn Grove
Hawthorn East, VIC 3123, Australia
Phone: +61-3-98130561