Mail Archives: cygwin/2009/07/28/06:36:35
On Jul 28 11:24, Andy Koppe wrote:
> 2009/7/28 Corinna Vinschen:
> > On Jul 27 22:56, Andy Koppe wrote:
> >> I've encountered what looks like a bug in mbrtowc's handling of UTF-8.
> >> Here's an example:
> >>
> >> #include <stdio.h>
> >> #include <locale.h>
> >> #include <stdlib.h>
> >> #include <wchar.h>
> >>
> >> int main(void) {
> >> wchar_t wc;
> >> size_t ret;
> >> mbstate_t s = { 0 };
> >> puts(setlocale(LC_CTYPE, "en_GB.UTF-8"));
> >> printf("%i\n", mbrtowc(&wc, "\xe2", 1, 0));
> >> printf("%i\n", mbrtowc(&wc, "\x94", 1, 0));
> >> printf("%i\n", mbrtowc(&wc, "\x84", 1, 0));
> >> printf("%x\n", wc);
> >> return 0;
> >> }
> >>
> >> The sequence E2 94 84 should translate to U+2514. Instead, the second
> >> and third calls to mbrtowc report encoding errors. It does work
> >> correctly if the three bytes are passed to mbrtowc() in one go:
> >>
> >> printf("%i\n", mbrtowc(&wc, "\xe2\x94\x84", 3, 0));
> >
> > That's a bug in the newlib function __utf8_mbtowc. I'm really surprised
> > that this bug has never been reported before since it's in the code for
> > years, probably since it has been introduced in 2002.
>
> I guess normallly programs just pass whole strings to mbrstowcs?
>
> I've had a look at the code, but didn't grasp it enough to suggest a
> fix. I'd also wondered how mbrtowc() deals with non-BMP characters
> given that wchar_t is only 16 bits wide, and was quite pleased to see
> that it does have a special hack for turning them into UTF-16
> surrogates.
>
> Trouble is, the hack will also only work correctly if the whole UTF-8
> sequence for the non-BMP character is passed at once. If you pass the
> bytes one-by-one instead, and assuming the bug above wasn't there,
> you'd get this:
Yes, I know. The real trouble is, I don't know how that can be fixed
in a still sort-of-POSIXy way. Typical POSIX code doesn't know about
UTF-16 and expects the wchar returned to be complete.
So, for the time being, surrogates work if they are part of a string
given to mbstowcs/mbsrtowcs/mbsnrtowcs, but they need very special
care if the application uses mbrtowc.
If you can come up with a nice patch, send it to the newlib list,
please.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
- Raw text -