Date: Mon, 10 Feb 2003 14:08:25 +0200 (EET)
From: Esa A E Peuha <peuha AT cc DOT helsinki DOT fi>
Sender: peuha AT sirppi DOT helsinki DOT fi
To: djgpp-workers AT delorie DOT com
Subject: Re: Checking for stack overflow
In-Reply-To: <10302071430.AA20844@clio.rice.edu>
Message-ID: <Pine.OSF.4.51.0302101329270.31820@sirppi.helsinki.fi>
References: <10302071430 DOT AA20844 AT clio DOT rice DOT edu>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Reply-To: djgpp-workers AT delorie DOT com
Errors-To: nobody AT delorie DOT com
X-Mailing-List: djgpp-workers AT delorie DOT com
X-Unsubscribes-To: listserv AT delorie DOT com
Precedence: bulk

On Fri, 7 Feb 2003, Charles Sandmann wrote:

> > Won't that cause an exception even now?  But I'll keep that in mind.
>
> If near pointers are enabled, or under Windows where it can do the
> address wrap thing behind your back - no ...

OK.  Here's what I wrote during the weekend.  Assuming that esp has
already decremented to point to the newly allocated space, the possible
overflow is detected thus:

	cmpl	___djgpp_stack_limit, %esp
	jge	0f
	movl	$0f, ___djgpp_stack_overflow_eip
	jmp	___djgpp_stack_overflow_exit
0:

This will fail only if the size of allocation is so ridiculously large
that esp wraps over to a huge positive value (when considered as signed
int), but gcc seems to guard against this already (at least the size of
a variable-sized automatic array is anded so that it's at most
0x20000000 IIRC, and built-in alloca should do something similar).

Then there's the overflow handler (which might be put in
src/libc/go32/overflow.S if there are no better suggestions):

	.data
	.balign	8
	.comm	exception_stack, 8000
	.balign	8
overflow_state_buf:		/* jmp_buf */
	.long	0, 0, 0, 0	/* eax, ebx, ecx, edx */
	.long	0, 0, 0, 0	/* esi, edi, ebp, esp */
	.globl	___djgpp_stack_overflow_eip
___djgpp_stack_overflow_eip:
	.long	0, 0		/* eip, eflags */
	.word	0, 0, 0, 0, 0, 0/* cs, ds, es, fs, gs, ss */
	.long	0, 0, 0		/* sigmask, signum, exception_ptr */
	.text
errstring:
	.ascii	"Out of stack. \0"
	.balign	16,,7
	.globl	___djgpp_stack_overflow_exit
___djgpp_stack_overflow_exit:
	movl	%eax, overflow_state_buf
	movl	$overflow_state_buf, %eax

	/* change stack as soon as possible */
	movl	%esp, 28(%eax)
	movl	$exception_stack + 8000, %esp

	/* eax already stored */
	movl	%ebx, 4(%eax)
	movl	%ecx, 8(%eax)
	movl	%edx, 12(%eax)
	movl	%esi, 16(%eax)
	movl	%edi, 20(%eax)
	movl	%ebp, 24(%eax)
	/* esp already stored */
	/* eip stored before jumping here */
	pushfl
	popl	36(%eax)
	movw	%cs, 40(%eax)
	movw	%ds, 42(%eax)
	movw	%es, 44(%eax)
	movw	%fs, 46(%eax)
	movw	%gs, 48(%eax)
	movw	%ss, 50(%eax)

	/* fake exception number like in __djgpp_traceback_exit,
	   0x7e == 0x7a + 1 + SIGSEGV - SIGABRT */
	movl	$0x7e, 56(%eax)

	movl	$overflow_state_buf, ___djgpp_exception_state_ptr

	/* print error message */
	pushl	$14
	pushl	$errstring
	pushl	$2
	call	__write

	/* 291 == SIGSEGV */
	pushl	$291

	/* this does not return */
	call	___djgpp_traceback_exit

This works in the sense that it exits with a traceback, but I'm not
quite satisfied with the output.  For one thing, if the traceback has
more entries than will fit the screen (very likely in case of infinite
recursion), then the line with "Out of stack. Exiting due to signal
SIGSEGV" will be scrolled out of screen (this seems like a bug in
show_call_frame).  For another, do_faulting_finish_message might print
"Stack overflown" instead of "Exception 7e", but I didn't know what
value to pass in signum for that (clearly 0x7e won't do since it might
be used by __djgpp_traceback_exit in other cases with SIGSEGV).

-- 
Esa Peuha
student of mathematics at the University of Helsinki
http://www.helsinki.fi/~peuha/