X-Recipient: archive-cygwin AT delorie DOT com
DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:message-id:date:from:mime-version:to:cc
	:subject:references:in-reply-to:content-type
	:content-transfer-encoding; q=dns; s=default; b=rTRaeXWtaVIQN5ru
	+rMgeP5Tq+jZa9TJR5V0mrfkLXw/RZiBu4ExPXEVSDmkd9XDa9uNO0Jo9O8t+tHd
	dATltB76rpMZS01/7pi5OjijmbpakM6HkvOLQSut/WHc+GIEvyuyqFa+MS54w0bA
	32AQQCykoXI9RZNTVCdbnPJZOgs=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:message-id:date:from:mime-version:to:cc
	:subject:references:in-reply-to:content-type
	:content-transfer-encoding; s=default; bh=D8DLR5LAinpwsEQ6Um/Emz
	pPHW0=; b=eUakBS0gjgwH0+h2OcfB29fRESDj5ulcBJn8XxA/Ad2t6om2t6Mr7Z
	iumGxOMRQlfnJuA3aiHCZJIzkDpbO2kxBxbxjkbY63EXzlHZdAj9L00pI65ZNaUi
	x3TLmc+bfoVH3lXuO7gr0SoCEh1fv/q6aV0Mo2IWGCXkspotcrgBc=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id: <cygwin.cygwin.com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
X-Spam-SWARE-Status: No, score=-4.4 required=5.0 tests=AWL,BAYES_00,KHOP_THREADED,RCVD_IN_HOSTKARMA_YE,RP_MATCHES_RCVD,SPF_PASS autolearn=ham version=3.3.2
X-CornellRouted: This message has been Routed already.
Message-ID: <520D089A.1020806@cornell.edu>
Date: Thu, 15 Aug 2013 12:58:02 -0400
From: Ken Brown <kbrown AT cornell DOT edu>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2
MIME-Version: 1.0
To: cygwin AT cygwin DOT com
CC: Eli Zaretskii <eliz AT gnu DOT org>
Subject: Re: 64-bit emacs crashes a lot
References: <51F3151D DOT 7040000 AT cs DOT utoronto DOT ca> <51F33565 DOT 1090406 AT cornell DOT edu> <51F33F52 DOT 4060405 AT cs DOT utoronto DOT ca> <51FB1D9E DOT 5090102 AT cs DOT utoronto DOT ca> <20130802080211 DOT GA18054 AT calimero DOT vinschen DOT de> <51FB9228 DOT 2020309 AT cornell DOT edu> <51FBA100 DOT 90005 AT cs DOT utoronto DOT ca> <51FD5462 DOT 5020400 AT cs DOT utoronto DOT ca> <51FFBDFF DOT 7040501 AT cornell DOT edu> <51FFC4F2 DOT 8080909 AT cs DOT utoronto DOT ca> <5203D89E DOT 6030801 AT cornell DOT edu> <5203DCCA DOT 1010105 AT cs DOT utoronto DOT ca> <5205B364 DOT 8090007 AT cs DOT utoronto DOT ca> <52064730 DOT 50404 AT cornell DOT edu> <52065B3C DOT 6060104 AT cs DOT utoronto <520CCA41 DOT 3000107 AT cs DOT utoronto DOT ca>
In-Reply-To: <520CCA41.3000107@cs.utoronto.ca>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 8/15/2013 8:32 AM, Ryan Johnson wrote:
> On 14/08/2013 10:04 AM, Ryan Johnson wrote:
>> On 10/08/2013 2:01 PM, Ken Brown wrote:
>>> I'm not sure that garbage collection is the underlying problem here.
>> Agree it's probably not GC... GC would just tend to trip over any bad
>> pointers that were lurking around...
>>
>> After a rash of crashes where I either forgot to attach gdb or forgot
>> to set appropriate breakpoints, I finally managed to catch the stack
>> trace below. It occurred during M-x compile, while emacs parsed the
>> compilation's rather copious output, which is by far the most common
>> type of crash I've been getting lately.
>>
> This time I really caught one, again during a compile (aside: the crash
> is usually right at the beginning of a compile, not partway through as
> my previous comment suggested). The short version is that I seem to have
> tickled a latent bug in emacs' character handling at character.c:189,
> where processing an 8-bit char could result in a return value larger
> than CHAR_MAX (which later triggers an abort). New stack trace and
> analysis follow.
>
> A quick peek at the backtrace and some source diving suggests something
> went wrong in bidi_fetch_char (bidi.c:928)... which unfortunately has
> about a dozen different ways of coming up with a character. There are
> only two major flavors that look like they can possibly fail, though:
> via STRING_CHAR_AND_LENGTH or UNIBYTE_TO_CHAR. Both are macros in
> character.h. The latter calls BYTE8_TO_CHAR, which simply returns
> byte+0x3fff00 (and can only take values between 0x3ffe80 and 0x3fff7f
> regardless of whether "byte" is signed or unsigned, though I think it's
> unsigned). The former is a pretty large macro (lots of ?:) that usually
> ends up fetching a char from p[], but has a test for sign bit to prevent
> unwanted sign extension (but again, I'm pretty sure they use unsigned
> bytes, so that shouldn't be an issue). However, the macro bottoms out by
> calling string_char (character.c:170), which includes the following
> expression:
>
> ((p)[1] & 0x3F) << 18
>
> That expression can return values significantly larger than CHAR_MAX
> (0x3FFFFF), the largest such being 0xFC0000.
>
> I suspect this is a bug in emacs (typo), and that the mask should really
> be 0xF: that would keep the value in bounds, and would also mirror a
> different branch of the if/else that differs only in the index passed to
> p[] and in using 0xF for the leading byte instead of 0x3F.
>
> On the other hand, the offending code seems to be trying to decode a
> 5-byte unicode sequence. If so, then the bug is not the mask 0x3F, but
> (a) forgetting to actually fetch the most significant two bits of the
> code point from p[0], and (b) having CHAR_MAX smaller than a valid
> 5-byte code point. Interestingly, CHAR_MAX is 0x3FFFFF---five F---while
> the largest valid 5-byte code point is 0x3FFFFFF---six F---and the
> largest 4-byte is 0x1FFFFF, not 0x3FFFFF (though perhaps the latter is
> intentional, to give emacs some out-of-band values to work with?).
> Meanwhile, 5- and 6-byte utf-8 sequences were banished in 2003
> (according to the Wikipedia entry on UTF-8), so it seems like this code
> path shouldn't even exist.
>
> I don't quite grok what sequence of input bytes it would take to tickle
> the above code path. I suspect that the input byte is invalid (= illegal
> unicode), due to the same memory corruption that made GC seg fault in a
> past crash, and just happens to tickle this latent bug. However, this
> charset bug should probably be sorted out and fixed, regardless of
> whether it's the cause of the troubles or merely the symptom of
> something deeper.
>
> Thoughts?
> Ryan

Eli is the expert on bidi.c (he wrote it).  He can probably tell you 
whether you've really bumped into an emacs bug here.

> Breakpoint 2, 0x000000010055d190 in kill ()
> (gdb) bt
> #0  0x000000010055d190 in kill ()
> #1  0x000000010053702e in process_send_signal (process=<optimized out>,
> signo=signo AT entry=1, current_group=<optimized out>, nomsg=nomsg AT entry=1)
> at /usr/src/debug/emacs-24.3-4/src/process.c:5948
> #2  0x0000000100537ae0 in kill_buffer_processes (buffer=4304412722) at
> /usr/src/debug/emacs-24.3-4/src/process.c:7157
> #3  0x0000000100485cb0 in shut_down_emacs (sig=sig AT entry=6,
> stuff=4304412722) at /usr/src/debug/emacs-24.3-4/src/emacs.c:1915
> #4  0x0000000100485ea7 in terminate_due_to_signal (sig=6,
> backtrace_limit=10) at /usr/src/debug/emacs-24.3-4/src/emacs.c:329
> #5  0x00000001004a0363 in emacs_abort () at
> /usr/src/debug/emacs-24.3-4/src/sysdep.c:2152
> #6  0x000000010046d365 in bidi_get_type (ch=<optimized out>,
> override=override AT entry=NEUTRAL_DIR) at
> /usr/src/debug/emacs-24.3-4/src/bidi.c:107
> #7  0x000000010046d527 in bidi_get_type (override=NEUTRAL_DIR,
> ch=<optimized out>) at /usr/src/debug/emacs-24.3-4/src/bidi.c:104
> #8  bidi_resolve_explicit_1 (bidi_it=bidi_it AT entry=0x225d18) at
> /usr/src/debug/emacs-24.3-4/src/bidi.c:1411
> #9  0x000000010046d8b5 in bidi_resolve_explicit
> (bidi_it=bidi_it AT entry=0x225d18) at
> /usr/src/debug/emacs-24.3-4/src/bidi.c:1540
> #10 0x000000010046db0a in bidi_resolve_weak
> (bidi_it=bidi_it AT entry=0x225d18) at
> /usr/src/debug/emacs-24.3-4/src/bidi.c:1625
> #11 0x000000010046e3c1 in bidi_resolve_neutral
> (bidi_it=bidi_it AT entry=0x225d18) at
> /usr/src/debug/emacs-24.3-4/src/bidi.c:1861
> #12 0x000000010046e7a0 in bidi_type_of_next_char (bidi_it=0x225d18) at
> /usr/src/debug/emacs-24.3-4/src/bidi.c:2032
> #13 bidi_level_of_next_char (bidi_it=bidi_it AT entry=0x225d18) at
> /usr/src/debug/emacs-24.3-4/src/bidi.c:2145
> #14 0x000000010046faba in bidi_move_to_visually_next
> (bidi_it=bidi_it AT entry=0x225d18) at
> /usr/src/debug/emacs-24.3-4/src/bidi.c:2355
> #15 0x000000010041c44f in set_iterator_to_next (it=it AT entry=0x225370,
> reseat_p=reseat_p AT entry=1) at /usr/src/debug/emacs-24.3-4/src/xdisp.c:7109
> #16 0x000000010042477a in display_line (it=it AT entry=0x225370) at
> /usr/src/debug/emacs-24.3-4/src/xdisp.c:19864
> #17 0x0000000100426cd8 in try_window (window=window AT entry=25782575869,
> pos=..., flags=flags AT entry=1) at
> /usr/src/debug/emacs-24.3-4/src/xdisp.c:16353
> #18 0x000000010042c45a in redisplay_window
> (window=window AT entry=25782575869,
> just_this_one_p=just_this_one_p AT entry=0) at
> /usr/src/debug/emacs-24.3-4/src/xdisp.c:15879
> #19 0x000000010042e2b6 in redisplay_window_0
> (window=window AT entry=25782575869) at
> /usr/src/debug/emacs-24.3-4/src/xdisp.c:13934
> #20 0x00000001004f5d9c in internal_condition_case_1
> (bfun=bfun AT entry=0x10042e290 <redisplay_window_0>, arg=25782575869,
> handlers=4304384246, hfun=hfun AT entry=0x10040fe90 <redisplay_window_error>)
>      at /usr/src/debug/emacs-24.3-4/src/eval.c:1327
> #21 0x000000010041458a in redisplay_windows (window=25782575869) at
> /usr/src/debug/emacs-24.3-4/src/xdisp.c:13914
> #22 0x00000001004145a8 in redisplay_windows (window=25780199573) at
> /usr/src/debug/emacs-24.3-4/src/xdisp.c:13908
> #23 0x000000010042f449 in redisplay_internal () at
> /usr/src/debug/emacs-24.3-4/src/xdisp.c:13493
> #24 0x0000000100430f95 in redisplay_preserve_echo_area
> (from_where=<optimized out>) at
> /usr/src/debug/emacs-24.3-4/src/xdisp.c:13754
> #25 0x0000000100535faa in wait_reading_process_output
> (time_limit=<optimized out>, nsecs=0, read_kbd=read_kbd AT entry=-1,
> do_display=do_display AT entry=true, wait_for_cell=4304412722,
> wait_proc=wait_proc AT entry=0x0,
>      just_wait_proc=just_wait_proc AT entry=0) at
> /usr/src/debug/emacs-24.3-4/src/process.c:4862
> #26 0x00000001004098da in sit_for (timeout=<optimized out>,
> reading=reading AT entry=true, display_option=display_option AT entry=1) at
> /usr/src/debug/emacs-24.3-4/src/dispnew.c:5978
> #27 0x0000000100491920 in read_char (commandflag=1, nmaps=2,
> maps=0x22a200, prev_event=4304412722, used_mouse_menu=0x22a327,
> end_time=end_time AT entry=0x0) at
> /usr/src/debug/emacs-24.3-4/src/keyboard.c:2669
> #28 0x00000001004926a3 in read_key_sequence
> (keybuf=keybuf AT entry=0x22a470, prompt=<optimized out>,
> dont_downcase_last=dont_downcase_last AT entry=false,
> can_return_switch_frame=can_return_switch_frame AT entry=true,
>      fix_current_buffer=fix_current_buffer AT entry=true, bufsize=30) at
> /usr/src/debug/emacs-24.3-4/src/keyboard.c:9231
> #29 0x000000010049495e in command_loop_1 () at
> /usr/src/debug/emacs-24.3-4/src/keyboard.c:1459
> #30 0x00000001004f5c2e in internal_condition_case
> (bfun=bfun AT entry=0x100494740 <command_loop_1>, handlers=4304470642,
> hfun=hfun AT entry=0x10048ae40 <cmd_error>) at
> /usr/src/debug/emacs-24.3-4/src/eval.c:1289
> #31 0x000000010048630a in command_loop_2
> (ignore=ignore AT entry=4304412722) at
> /usr/src/debug/emacs-24.3-4/src/keyboard.c:1168
> #32 0x00000001004f5aef in internal_catch (tag=<optimized out>,
> func=func AT entry=0x1004862e0 <command_loop_2>, arg=4304412722) at
> /usr/src/debug/emacs-24.3-4/src/eval.c:1060
> #33 0x000000010048a914 in command_loop () at
> /usr/src/debug/emacs-24.3-4/src/keyboard.c:1147
> #34 recursive_edit_1 () at /usr/src/debug/emacs-24.3-4/src/keyboard.c:779
> #35 0x000000010048ac47 in Frecursive_edit () at
> /usr/src/debug/emacs-24.3-4/src/keyboard.c:843
> #36 0x000000010055e8ef in main (argc=<optimized out>, argv=<optimized
> out>) at /usr/src/debug/emacs-24.3-4/src/emacs.c:1537
>
>>
>> Thanks,
>> Ryan
>>
>> Breakpoint 2, 0x000000010055d190 in kill ()
>> (gdb) bt
>> #0  0x000000010055d190 in kill ()
>> #1  0x000000010053702e in process_send_signal
>> (process=process AT entry=25781889629, signo=signo AT entry=2,
>> current_group=<optimized out>, nomsg=nomsg AT entry=0) at
>> /usr/src/debug/emacs-24.3-4/src/process.c:5948
>> #2  0x0000000100537198 in Finterrupt_process (process=25781889629,
>> current_group=<optimized out>) at
>> /usr/src/debug/emacs-24.3-4/src/process.c:5966
>> #3  0x00000001004f7761 in Ffuncall (nargs=<optimized out>,
>> args=<optimized out>) at /usr/src/debug/emacs-24.3-4/src/eval.c:2781
>> #4  0x000000010052b5ed in exec_byte_code (bytestr=4294962344,
>> vector=2268896, maxdepth=2, args_template=4303595040,
>> nargs=4304157760, args=0x100902032 <bss_sbrk_buffer+250194>)
>>     at /usr/src/debug/emacs-24.3-4/src/bytecode.c:900
>> #5  0x00000001004f7293 in funcall_lambda (fun=25778101277,
>> nargs=nargs AT entry=0, arg_vector=arg_vector AT entry=0x22a188) at
>> /usr/src/debug/emacs-24.3-4/src/eval.c:3010
>> #6  0x00000001004f75cb in Ffuncall (nargs=nargs AT entry=1,
>> args=args AT entry=0x22a180) at /usr/src/debug/emacs-24.3-4/src/eval.c:2839
>> #7  0x00000001004f8bef in apply1 (fn=25778613730, fn AT entry=4304161216,
>> arg=arg AT entry=4304412722) at /usr/src/debug/emacs-24.3-4/src/eval.c:2539
>> #8  0x00000001004f3567 in Fcall_interactively (function=4304161216,
>> record_flag=4304412722, keys=4299711881) at
>> /usr/src/debug/emacs-24.3-4/src/callint.c:377
>> #9  0x00000001004f7752 in Ffuncall (nargs=nargs AT entry=4,
>> args=args AT entry=0x22a3b0) at /usr/src/debug/emacs-24.3-4/src/eval.c:2785
>> #10 0x00000001004f91b7 in call3 (fn=<optimized out>, arg1=<optimized
>> out>, arg2=<optimized out>, arg3=<optimized out>) at
>> /usr/src/debug/emacs-24.3-4/src/eval.c:2603
>> #11 0x00000001004883cd in Fcommand_execute (cmd=<optimized out>,
>> record_flag=<optimized out>, keys=<optimized out>, special=<optimized
>> out>) at /usr/src/debug/emacs-24.3-4/src/keyboard.c:10241
>> #12 0x0000000100494ae8 in command_loop_1 () at
>> /usr/src/debug/emacs-24.3-4/src/keyboard.c:1587
>> #13 0x00000001004f5c2e in internal_condition_case
>> (bfun=bfun AT entry=0x100494740 <command_loop_1>, handlers=4304470642,
>> hfun=hfun AT entry=0x10048ae40 <cmd_error>) at
>> /usr/src/debug/emacs-24.3-4/src/eval.c:1289
>> #14 0x000000010048630a in command_loop_2
>> (ignore=ignore AT entry=4304412722) at
>> /usr/src/debug/emacs-24.3-4/src/keyboard.c:1168
>> #15 0x00000001004f5aef in internal_catch (tag=<optimized out>,
>> func=func AT entry=0x1004862e0 <command_loop_2>, arg=4304412722) at
>> /usr/src/debug/emacs-24.3-4/src/eval.c:1060
>> #16 0x000000010048a914 in command_loop () at
>> /usr/src/debug/emacs-24.3-4/src/keyboard.c:1147
>> #17 recursive_edit_1 () at /usr/src/debug/emacs-24.3-4/src/keyboard.c:779
>> #18 0x000000010048ac47 in Frecursive_edit () at
>> /usr/src/debug/emacs-24.3-4/src/keyboard.c:843
>> #19 0x000000010055e8ef in main (argc=<optimized out>, argv=<optimized
>> out>) at /usr/src/debug/emacs-24.3-4/src/emacs.c:1537



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple