Date: Mon, 31 Jul 2023 15:38:11 +0200
To: Bruno Haible <bruno@clisp.org>
Subject: Re: GB18030 locale
Message-ID: <ZMe5Q02S5ap5gBbJ@calimero.vinschen.de>
Mail-Followup-To: Bruno Haible <bruno@clisp.org>, cygwin@cygwin.com
References: <3884636.3uDm00564X@nimes> <4641755.FJ9Bj1ZfmD@nimes>
 <ZMTaqNf4dnbry6BD@calimero.vinschen.de> <5536760.9AX2XiyloC@nimes>
 <ZMeH6yZQkK0exU8H@calimero.vinschen.de>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <ZMeH6yZQkK0exU8H@calimero.vinschen.de>
Precedence: list
From: Corinna Vinschen via Cygwin <cygwin@cygwin.com>
Reply-To: cygwin@cygwin.com
Cc: Corinna Vinschen <corinna-cygwin@cygwin.com>, cygwin@cygwin.com
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: cygwin-bounces+archive-cygwin=delorie.com@cygwin.com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie.com@cygwin.com>

Hi Bruno,

On Jul 31 12:07, Corinna Vinschen via Cygwin wrote:
> On Jul 29 11:53, Bruno Haible via Cygwin wrote:
> > Corinna Vinschen wrote:
> > > However, on debugging this, I see it's totally broken.  Trying to fix
> > > this in the existing functions is futile.  We need dedicated
> > > support functions for GB18030, kind of like the FreeBSD functions,
> > > just with extra support for surrogate pairs, as with our UTF8 stuff.
> > 
> > In case it helps: Find here a test suite for the various multibyte
> > functions with GB18030 specific test cases. (Extracted from gnulib.)
> > https://haible.de/bruno/gnu/testdir-gb18030.tar.gz
> 
> Thank you, I'm already hacking and testing :)

I have a problem with the c32isalpha function.

c32isalpha fails for the character U+FF11 FULLWIDTH DIGIT ONE,
because it expects the character to be an alphabetic character.

The Cygwin unicode information is automatically generated from the
Unicode data file UnicodeData.txt, fresh from their homepage.  iswalpha
in newlib is checking for the Unicode categories, using the expression:

    return cat == CAT_LC || cat == CAT_Lu || cat == CAT_Ll || cat == CAT_Lt
          || cat == CAT_Lm || cat == CAT_Lo
	  || cat == CAT_Nl // Letter_Number
	  ;

with CAT_foo being equivalent to Unicode category foo.

Per UnicodeData.txt, ff11 is of category Nd, so it's a digit, not an
alphabetic character.

I see that Glibc returns 1 from c32isalpha for U+FF11, but I don't see
where it takes that info and why this is correct.  Can you point me to
some info on this?


Thanks,
Corinna

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple