DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 55RF0q3l1477820
Date: Fri, 27 Jun 2025 16:59:51 +0200
To: cygwin@cygwin.com
Subject: Re: readdir() returns inaccessible name if file was created with
 invalid UTF-8
Message-ID: <aF6x55WXIS1t655i@calimero.vinschen.de>
Mail-Followup-To: cygwin@cygwin.com
References: <96f2253b-791b-b8a0-97dd-8d257eefb9b1@t-online.de>
 <03c4fae7-7322-572c-ae72-52e300f0b438@t-online.de>
 <aFxRfI4NdZ8y5IlK@calimero.vinschen.de>
 <f78c615c-aefe-b3d0-aada-5f9d0cf73a0a@t-online.de>
 <aF5y15iQ840LxLYJ@calimero.vinschen.de>
 <3295c8bd-2c09-76c7-8b5f-0106dc39dd96@t-online.de>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <3295c8bd-2c09-76c7-8b5f-0106dc39dd96@t-online.de>
Precedence: list
From: Corinna Vinschen via Cygwin <cygwin@cygwin.com>
Reply-To: cygwin@cygwin.com
Cc: Corinna Vinschen <corinna-cygwin@cygwin.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: cygwin-bounces~archive-cygwin=delorie.com@cygwin.com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie.com@cygwin.com>

On Jun 27 15:32, Christian Franke via Cygwin wrote:
> $ touch $'t-\xef\x80\x80'
> The name mapping is:
> "t-\xEF\x80\x80" -(open, ...)-> L"t-\xDB59" -(readdir)-> "t-"

Did you copy/paste this from the old mail, by any chance?

Using the latest test DLL the mapping is

  "t-\xEF\x80\x80" -(open, ...)-> L"t-\xF000" 

And that's basically correct, albeit it leads to problems.

You know that we defined the area from 0xf000 to 0xf0ff as our private
use area to create filenames with characters invalid in DOS filenames
by transposing these chars into the private use area.  When converting
the filenames back, the 0xf0XX chars are transposed back to 0xXX.

But yeah, I found the bug here.  The problem is that the transpose table
incorrectly contains NUL as transposable character.  So if you create
L"t-\xF000", that's fine.  However, when converting this name back to
UTF-8, the filename becomes L"t-\0".  Oops.

I dropped the ASCII NUL from the list of transposable characters and
now what you get is this:

  $ touch $'t-\xef\x80\x80'
  $ touch $'t-\xef\x80\x81'
  $ ls -l
  total 0
  -rw-r--r-- 1 corinna vinschen 0 Jun 27 16:49 't-'$'\001'
  -rw-r--r-- 1 corinna vinschen 0 Jun 27 16:49 't-'$'\357\200\200'

Apart from the incorrect transposition of ASCII NUL, the transposition 
works transparently:

  $ echo foo > $'t-\xef\x80\x81'
  $ cat $'t-\xef\x80\x81'
  foo
  $ cat $'t-\x01'
  foo

I'll apply the patch shortly.


Thanks,
Corinna

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple