X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C331B3850210
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1679698190;
	bh=RZPJacvJ+JRQP7DdKAgfl+fNpx2Y3dw2vWwMXy5i3tM=;
	h=Date:Subject:To:References:In-Reply-To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:
	 From;
	b=BP98Pb35Ja3iFpIVz0sKVBzjeu/DADah0VJuPzC7hAH7XmsDOwcQGaqUEvGFHD1Ju
	 zcBkt7+yyoTR00yksrImdLQ8B9B3+kI78dxyvhNvE9V1nGBzmlTsPNc8V4grYKsaQH
	 RMaaYrfToWXyBbvsFSkLrB+lE3GsP6qhOP5YZpqI=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D11E73858D35
X-Authority-Analysis: v=2.4 cv=e5oV9Il/ c=1 sm=1 tr=0 ts=641e28e4
 a=DxHlV3/gbUaP7LOF0QAmaA==:117 a=DxHlV3/gbUaP7LOF0QAmaA==:17
 a=IkcTkHD0fZMA:10 a=BqEg4_3jAAAA:8 a=qv0N3cA8AAAA:8 a=n8i27M1mAAAA:8
 a=1XWaLZrsAAAA:8 a=Og4VvuUy-h3zDRD9t38A:9 a=QEXdDO2ut3YA:10
 a=0mFWnFbQd5xWBqmg7tTt:22 a=IkJeoXH5QaFgIRku97pq:22
Message-ID: <bb1eed09-54c2-08a1-7eb9-40e39c7657d9@Shaw.ca>
Date: Fri, 24 Mar 2023 16:49:07 -0600
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Subject: Re: newlocale: Linux incompatibility
Content-Language: en-CA
To: cygwin AT cygwin DOT com
References: <bd7ebebc-cf90-d509-ef15-11d702a6126c AT cornell DOT edu>
 <ZBzBL2jFO7Oltjd1 AT calimero DOT vinschen DOT de>
 <ZB2U/JCFrwSUo1+U AT calimero DOT vinschen DOT de>
Organization: Inglis
In-Reply-To: <ZB2U/JCFrwSUo1+U@calimero.vinschen.de>
X-CMAE-Envelope: MS4xfFrfYEuMgDvs2zCd5AJPfCOyU0ntLVe+/yTCsH/zsfSjO9NV54bPd+6F3F38byYQy4UTCP86DCr3GpXRQ24Z1Av+RQY9f8L6Xyj68IvxP7rEAOKygcIC
 RL/zL9S8swvhiYe14Fd5JLGIrBdDlyE7dlBmuxKl+vuxWqkzDjcH1E9fYLucnH6723zy0FOW5ySVxQ==
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, DKIM_SIGNED,
 DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, KAM_SHORT, NICE_REPLY_A,
 RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_PASS,
 TXREP autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Brian Inglis via Cygwin <cygwin AT cygwin DOT com>
Reply-To: cygwin AT cygwin DOT com
Cc: Brian Inglis <Brian DOT Inglis AT Shaw DOT ca>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 32OMoFvp030518

On 2023-03-24 06:18, Corinna Vinschen via Cygwin wrote:
> On Mar 23 22:14, Corinna Vinschen via Cygwin wrote:
>> On Mar 23 15:48, Ken Brown via Cygwin wrote:
>>> I'm reporting this here rather than the newlib list because the behavior is
>>> compatible with Posix but not Linux, so I think it's a Cygwin issue.
>>
>> Actually, it's a Windows issue :)
>>
>>> Consider the following test case:
>>>
>>> $ cat locale_test.c
>>> #include <stdio.h>
>>> #include <locale.h>
>>>
>>> int main ()
>>> {
>>>    const char *locale = "en_DE.UTF-8";
>>>    locale_t loc = newlocale (LC_COLLATE_MASK | LC_CTYPE_MASK, locale, 0);
>>>    if (!loc)
>>>      perror ("newlocale");
>>>    else
>>>      printf ("newlocale succeeded on invalid locale %s\n", locale);
>>> }
>>>
>>> $ gcc -o locale_test locale_test.c
>>>
>>> $ ./locale_test.exe
>>> newlocale succeeded on invalid locale en_DE.UTF-8
>>>
>>> On Linux, the newlocale call fails with ENOENT, as is documented on the man
>>> page.  Posix doesn't say what should happen on an invalid locale, so this is
>>> not, strictly speaking, a bug.
>>
>> Three bugs in fact.
>>
>> First, it's a bug in the Emacs testsuite.  The test simply assumes that
>> there's no en_DE locale on any system, but that's just not true.
>> Windows support the RFC 5646 locale "en-DE", which is called "English
>> (Germany)" in the "Region" settings.
>>
>> You can also check with `locale -av | less' and search for en_DE.
>>
>> For the reminder of this mail, I assume you're talking about Cygwin 3.5.
>> I won't fix this for 3.4 anymore, given how much locale handling has
>> changed for 3.5.
>>
>> The second bug is that Cygwin blindly trusts the Windows function
>> ResolveLocaleName().  That function blatantly converts even vaguely
>> similar locales into something it supports.  E.g., it converts "en-XY"
>> to "en-US".  I. .e., even if you use "en_XY.utf8" as locale, the above
>> testcase will wrongly succeed.  So I have to rethink how I resolve POSIX
>> locales to Windows locales.

Does Windows even consider https://www.rfc-editor.org/rfc/rfc4647 "Matching of 
Language Tags", part of https://www.rfc-editor.org/info/bcp47 "Language Tags", 
and if POSIX only matches exactly, will LANGUAGE be able to be used for fallback?

I currently define LANGUAGE=en_CA:en_GB:en in case en-CA is unsupported by 
anything.
[I use my own en-CA locale not the glibc default created by https://rap.dk/.]

Will "-" be supported like "_" as a separator in values?

>> And the third bug is that Cygwin fails to set errno if it doesn't
>> support a locale, but that's a minor inconvenience in comparison.
>>
>> Thanks for the report, I totally missed the above problem with
>> ResolveLocaleName.
> 
> I pushed a couple of patches which hopefully clean up the code.  It's
> really frustrating how these Windows locale functions work.  Or, rather,
> not work.  I mean, come on...
> 
> - ResolveLocaleName() resolves "ff-BF" to "ff-Latn-SN", not to
>    "ff-Adlm-BF" or "ff-Latn-BF", even though both exist.
> 
> - There's a locale called "sd-Arab-PK" and a locale "sd-Deva-IN".  If
>    you ask for the script used in "sd-IN", the result is "Arab", not
>    "Deva".
 >
> I had to create a replacement function for ResolveLocaleName which
> doesn't return totally screwy and unexpected results, and special case
> two more locales in /proc/locales output so the output makes sense.

Aha - a nice new 3.5.0 feature - as well as /proc/codesets - is that charsets 
e.g. ISO-10646, etc. rather than encodings e.g. UTF-8, etc.!

FYI Google fixed their English L14N falling back to en-GB except US territories:

https://developer.android.com/guide/topics/resources/multilingual-support#postN
https://issuetracker.google.com/issues/64429534#comment6

and there have been similar issues posted for other languages.

> Oh, and I added error handling to the code so newlocale is now able to
> set errno to ENOENT if the locale is not supported.
> 
> If you want to test this, the changes are in test release
> 3.5.0-0.260.gb5b67a65f87c, which is just building.
-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple