DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 03A4B3858CD1
To: cygwin@cygwin.com
Subject: Re: character class "alpha"
Date: Mon, 31 Jul 2023 16:06:13 +0200
Message-ID: <5176597.IBPj4gxFZX@nimes>
In-Reply-To: <ZMe5Q02S5ap5gBbJ@calimero.vinschen.de>
References: <3884636.3uDm00564X@nimes> <ZMeH6yZQkK0exU8H@calimero.vinschen.de>
 <ZMe5Q02S5ap5gBbJ@calimero.vinschen.de>
MIME-Version: 1.0
Precedence: list
From: Bruno Haible via Cygwin <cygwin@cygwin.com>
Reply-To: Bruno Haible <bruno@clisp.org>
Content-Type: text/plain; charset="iso-8859-1"
Errors-To: cygwin-bounces+archive-cygwin=delorie.com@cygwin.com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie.com@cygwin.com>
Content-Transfer-Encoding: 8bit

Corinna Vinschen wrote:
> I have a problem with the c32isalpha function.
> 
> c32isalpha fails for the character U+FF11 FULLWIDTH DIGIT ONE,
> because it expects the character to be an alphabetic character.

This is not a big problem. You can see in the test-c32isalpha.c file
that this test is disabled for many platforms, in particular glibc.
There's no problem with disabling it on Cygwin as well.

> The Cygwin unicode information is automatically generated from the
> Unicode data file UnicodeData.txt, fresh from their homepage.  iswalpha
> in newlib is checking for the Unicode categories, using the expression:
> 
>     return cat == CAT_LC || cat == CAT_Lu || cat == CAT_Ll || cat == CAT_Lt
>           || cat == CAT_Lm || cat == CAT_Lo
> 	  || cat == CAT_Nl // Letter_Number
> 	  ;
> 
> with CAT_foo being equivalent to Unicode category foo.
> 
> Per UnicodeData.txt, ff11 is of category Nd, so it's a digit, not an
> alphabetic character.

This is not wrong. However, see the comments in the generator of the
gnulib tables:

https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/gen-uni-tables.c;h=0dceedc06cd72f886807fd575a2c4dba99cd147a;hb=HEAD#l5789

   /* Consider all the non-ASCII digits as alphabetic.
      ISO C 99 forbids us to have them in category "digit",
      but we want iswalnum to return true on them.  */

Likewise in the generator of the glibc tables:

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/unicode-gen/unicode_utils.py;h=5af03113a2f1f063769752ea426fcaf6f6ba9e95;hb=HEAD#l274

The original comment (from 2000) was:

  /* SUSV2 gives us some freedom for the "digit" category, but ISO C 99
     takes it away:
     7.25.2.1.5:
        The iswdigit function tests for any wide character that corresponds
        to a decimal-digit character (as defined in 5.2.1).
     5.2.1:
        the 10 decimal digits 0 1 2 3 4 5 6 7 8 9
   */
  return (ch >= 0x0030 && ch <= 0x0039);

The question is: In which category do you put these non-ASCII digits?
"print" and "graph", sure. But other than that? "punct" or "alnum"?
"punct" seems wrong. If you, like me, decide to put them in "alnum",
then you they need to be in "alpha" or "digit" (per POSIX
https://pubs.opengroup.org/onlinepubs/9699919799/functions/iswalnum.html ).
But ISO C 23 § 7.4.1.5 + § 5.2.1 does not allow them in category "digit".

Bruno


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple