Date: Wed, 6 Dec 2000 09:44:32 +0200 (IST)
From: Eli Zaretskii <eliz AT is DOT elta DOT co DOT il>
X-Sender: eliz AT is
To: Greg Holdridge <greg AT holdridge7 DOT freeserve DOT co DOT uk>
cc: djgpp AT delorie DOT com
Subject: Re: realloc causing fault
In-Reply-To: <000501c05f0e$62521020$f347893e@gtdf>
Message-ID: <Pine.SUN.3.91.1001206094236.10098D-100000@is>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Reply-To: djgpp AT delorie DOT com
Errors-To: nobody AT delorie DOT com
X-Mailing-List: djgpp AT delorie DOT com
X-Unsubscribes-To: listserv AT delorie DOT com
Precedence: bulk


[Please don't take this thread off the news group, I think it is
useful to have others read this and contribute if they can.]

On Tue, 5 Dec 2000, Greg Holdridge wrote:

> > Your calculations are mistaken.  You don't take into account the
> > overhead of each allocation (8 bytes per allocated buffer).  So in
> > fact you allocate much more than 150KB.  You also do it very
> 
> even so, with 64MB plus virtual mem it should be ok.

Not necessarily.  There are many subtle quirks in how memory is
managed by Windows and DJGPP programs.

> > inefficiently: allocating memopry 1 byte at a time, then reallocating
> > it for the next byte is about the worst memory usage I could think of.
> 
> not one byte, one string.

One string whose length is 1:

   strtable[numstrings-1].len = str->len;
   strtable[numstrings-1].buffer = malloc(str->len);

and str->len is 1 in the program you posted.

> And besides, I need to allocate in array format
> because strings are accessed by code.  Otherwise access would be very slow,
> if for example I used the binary tree structure I use for compression. How
> else could this be achieved without pre-allocation the stringtable space

Sorry, I don't follow your reasoning, probably because I don't know
enough about the problem you are trying to solve.

In general, allocating memory in lots of small chinks is a bad idea.
Your algorithm starts with very small memory buffer, then grows it one
small chunk at a time.  This will cause a massive memory fragmentation
in most malloc implementations (one sign of this fragmentation is that
the address goes up all the time, that is, memory allocated for the
previous chunk is not reused, but rather abandoned).

The usual way to avoid this fragmentation is to allocate space for a
reasonable number of structures (based on what you expect from an
average case), then grow the storage as needed by _doubling_ it.

> > I could try guessing where your program crashes and why, but I don't
> > want to guess.  Please post the full crash message printed by the
> > program, after running symify on it, and please show on the source
> 
> I cant do this easily since the fault cause is handled by windows under a
> DOS-box and so no frame trace is performed - just a little grey window pops
> up telling me 'the DOS program was naughty and has been...'

First, you can run the program in DOS mode (you will need CWSDPMI to
be installed).  Second, if the Windows GPF dialog shows any details,
please post them in their entirety.  And third, you could instrument
the program with debugging print-outs and find out which line exactly
causes the GPF.

> However the references I get are
> 
> _malloc +545
> _realloc +40

Where did you get these references?  This looks like part of the DJGPP
crash traceback, but you told above that it isn't printed?  If this
_is_ a DJGPP traceback, please post the rest of it.

Anyway, I ran your program on my machine, after making a single
change.  I changed this fragment:

   strtable = (lstring_t*)realloc(strtable, numstrings*sizeof(lstring_t));
   fprintf(log,"\nrealloc succeeded (%#x)",strtable);
   fflush(log);

   if (!strtable)
   {
      printf("\ndecompress error: couldnt allocate stringtable entry");
      exit(1);
   }

to say this instead:

   strtable = (lstring_t*)realloc(strtable, numstrings*sizeof(lstring_t));

   if (!strtable)
   {
      printf("\ndecompress error: couldnt allocate stringtable entry");
      exit(1);
   }
   fprintf(log,"\nrealloc succeeded (%#x)",strtable);
   fflush(log);

That is, I took a precaution of not invoking `fprintf' in a situation
where all virtual memory might be exhausted.  Then the program didn't
crash, but simply announced that it has run out of memory and exited,
after consuming about 22MB (on a 64MB machine with several large
programs running).