DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C954B385843B
Date: Thu, 25 Nov 2021 13:54:42 +0100
To: cygwin@cygwin.com
Subject: Re: raise(-1) has stopped returning an error recently
Message-ID: <YZ+HkgPIwmCuTcJr@calimero.vinschen.de>
Mail-Followup-To: cygwin@cygwin.com
References: <YZsoj6UvpF6pcbtt@slk1.local.net>
 <YZtwMZ1LUbx+b5+s@calimero.vinschen.de>
 <YZuVy5+nbzPtiqdw@calimero.vinschen.de>
 <YZyl69ODRcBVnMed@slk1.local.net>
 <YZy5bRsZuulb6FUV@calimero.vinschen.de>
 <42c9bb90-dd78-edfa-99ff-f65f7e000956@SystematicSw.ab.ca>
 <YZ1tAfzwlW8C84z4@slk1.local.net>
 <YZ4FGpEDDar45HC7@calimero.vinschen.de>
 <643c1cb7-9b18-25cf-62b0-8085c8fab137@Shaw.ca>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <643c1cb7-9b18-25cf-62b0-8085c8fab137@Shaw.ca>
Precedence: list
From: Corinna Vinschen via Cygwin <cygwin@cygwin.com>
Reply-To: cygwin@cygwin.com
Cc: Corinna Vinschen <corinna-cygwin@cygwin.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: cygwin-bounces+archive-cygwin=delorie.com@cygwin.com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie.com@cygwin.com>

On Nov 24 11:01, Brian Inglis via Cygwin wrote:
> On 2021-11-24 02:25, Corinna Vinschen via Cygwin wrote:
> > > On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote:
> > > > Do Cygwin and/or Windows support surrogate pairs in UTF-8?
> > 
> > You mean UTF-16.  UTF-8 doesn't know surrogate pairs, UTF-16 does.
> > Originally there was UCS-2, 16 bits, with only 65536 code points.
> > However, Unicode left the BMP already with version 2.0 in 1996, so
> > UTF-16 and surrogate pairs became necessary.  Windows as well as Cygwin
> > support them.
> 
> How does Cygwin support UTF-16 locales with surrogate pairs?

UTF-16 locales?  There's no such thing.  UTF-16 is just the 16 bit
representation for Unicode, and as such, is independent of the locale.
On the user side, Cygwin only supports UTF-8 as Unicode representation.
Internally you can then convert them to wchar_t which is UTF-16.

> Are they the "native" locales inherited from Windows if others are not
> specified e.g. UTF-8, some OEM SBCS or MBCS?

Just try `locale -av' and you'll see all supported locales and their
respective default codeset.  All of them can be used with .utf8
specifier to use UTF-8 instead of the default codeset.  Some of them
use UTF-8 as default codeset anyway, e. g., fa_IR or yo_NG.

> > > There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess
> > > surrogate pairs in UTF-8 "mostly work".
> > 
> > UTF-16.  The surrogate stuff is evil at times.  Have a look at the
> > __utf8_wctomb function in
> > https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/wctomb_r.c
> > Lone surrogate halfs in an input stream are a problem, for instance.
> 
> Thus the confusion with grep surrogate pair tests which appear to be running
> under a UTF-8 locale: see attached surrogate pair extract from cygport
> --debug grep.cygport check.

An STC in plain C might be helpful.


Corinna

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple