X-Spam-Check-By: sourceware.org
Message-ID: <3593.67.40.28.188.1169584799.squirrel@67.40.28.188>
In-Reply-To: <20070122202137.GC20665@calimero.vinschen.de>
References: <20070122181727 DOT GC27843 AT calimero DOT vinschen DOT de>     <1552 DOT 67 DOT 40 DOT 28 DOT 188 DOT 1169493973 DOT squirrel AT 67 DOT 40 DOT 28 DOT 188>     <20070122202137 DOT GC20665 AT calimero DOT vinschen DOT de>
Date: Tue, 23 Jan 2007 12:39:59 -0800 (PST)
Subject: Re: Perl bug?
From: "Yitzchak Scott-Thoennes" <sthoenna AT efn DOT org>
To: cygwin AT cygwin DOT com
User-Agent: SquirrelMail/1.5.0
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-IsSubscribed: yes
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Precedence: bulk
List-Id: <cygwin.cygwin.com>
List-Unsubscribe: <mailto:cygwin-unsubscribe-archive-cygwin=delorie DOT com AT cygwin DOT com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com

Corinna Vinschen wrote:
> On Jan 22 11:26, Yitzchak Scott-Thoennes wrote:
>> Corinna wrote:

> $a = "a" x (100 * 1024 * 1024);
> sleep 5;

>> >  "a"         --> malloc (2 bytes)
>> >  x 100 Megs  --> realloc (100 Megs) + malloc (100 Megs)
>> >
>> > So the result is that each string of 100 Megs requires 200 Megs of
>> > memory.  Doing this once is no problem, but doing it over and over
>> > again will hit the maximum memory available twice as early.
>>
>> This is as I would expect.  Most operators have a target, a temporary
>> lexical, allocated to store their results.  Like all lexicals, these
>> hold on to any memory they have allocated in the hope of saving having
>> to allocate it a second time.
>
> The problem is that it's *not* reused.  If you use strace when running
> this script you see the allocations as I described them.  When the
> 2 Gigs virtual memory size for the process are used up, mmap (which
> is called by malloc for big memory chunks) is called and returns -1,
> MAP_FAILED.  Then malloc tries to get the memory from the heap, when
> that fails, it just prints "Out of memory!", munmaps half of the above
> allocations and then exits.

Right, *each* x operation has it's own target.  If you undef $a, $b, etc.
at the end and wrap the whole thing in a for (1..2) loop, you'll see the
second time round that space is only allocated once per statement.
Or just have one x operation, and only see the "leaked" memory allocated
once:

#!/usr/bin/perl
for ($a, $b, $c, $d, $e, $f, $g, $h, $h, $i, $j, $k, $l) {
	$_ = "a" x (100 * 1024 * 1024);
	sleep 5;
}
sleep 95;

x= doesn't behave the same way. It doesn't use the target, since it knows
where the result should go, so the following doesn't have the problem you
observed:

#!/usr/bin/perl
$a = "a"; $a x= (100 * 1024 * 1024);
sleep 5;
$b = "b"; $b x= (100 * 1024 * 1024);
sleep 5;
$c = "c"; $c x= (100 * 1024 * 1024);
sleep 5;
$d = "d"; $d x= (100 * 1024 * 1024);
sleep 5;
$e = "e"; $e x= (100 * 1024 * 1024);
sleep 5;
$f = "f"; $f x= (100 * 1024 * 1024);
sleep 5;
$g = "g"; $g x= (100 * 1024 * 1024);
sleep 5;
$h = "h"; $h x= (100 * 1024 * 1024);
sleep 5;
$h = "h"; $h x= (100 * 1024 * 1024);
sleep 5;
$i = "i"; $i x= (100 * 1024 * 1024);
sleep 5;
$j = "j"; $j x= (100 * 1024 * 1024);
sleep 5;
$k = "k"; $k x= (100 * 1024 * 1024);
sleep 5;
$l = "l"; $l x= (100 * 1024 * 1024);
sleep 100;

It is possible to free the memory used by targets, but only by putting the
code in an anonymous closure:

#!/usr/bin/perl
my $dummy;
&{sub {

$dummy if 0;
$a = "a" x (1 * 1024 * 1024);
sleep 5;
$b = "b" x (1 * 1024 * 1024);
sleep 5;
$c = "c" x (1 * 1024 * 1024);
sleep 5;
$d = "d" x (1 * 1024 * 1024);
sleep 5;
$e = "e" x (1 * 1024 * 1024);
sleep 5;
$f = "f" x (1 * 1024 * 1024);
sleep 5;
$g = "g" x (1 * 1024 * 1024);
sleep 5;
$h = "h" x (1 * 1024 * 1024);
sleep 5;
$h = "h" x (1 * 1024 * 1024);
sleep 5;
$i = "i" x (1 * 1024 * 1024);
sleep 5;
$j = "j" x (1 * 1024 * 1024);
sleep 5;
$k = "k" x (1 * 1024 * 1024);
sleep 5;
$l = "l" x (1 * 1024 * 1024);
sleep 5;

}};
sleep 95;

$dummy forces the anonymous sub to be a closure, which gets it's own pad
(storage space for lexicals and targets) each time the sub{} is encountered
at runtime.  Without it, the sub{} holds on to it's pad even after the
code reference it returns is freed.

There has been some work towards implementing a "use less 'memory';" pragma
that may make perl less greedy, but there's no timetable for that or clear
plan for what it would do.

-- 
I'm looking for a job: http://perlmonks.org/?node=ysth#looking

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/