Mail Archives: cygwin/2009/09/22/12:12:35
2009/9/22 Corinna Vinschen:
>> > As you might know, invalid bytes >=3D 0x80 are translated to UTF-16 by
>> > transposing them into the 0xdc00 - 0xdcff range by just or'ing 0xdc00.
>> > The problem now is that readdir() will return the transposed characters
>> > as if they are the original characters.
>>
>> Yep, that's where the bug is. Those 0xDC?? words represent invalid
>> UTF-8 bytes. They do not represent CP1252 or ISO-8859-1 characters.
>>
>> Therefore, when converting a UTF-16 Windows filename to the current
>> charset, 0xDC?? words should be treated like any other UTF-16 word
>> that can't be represented in the current charset: it should be encoded
>> as a ^N sequence.
>
> How? =C2=A0Just like the incoming multibyte character didn't represent a =
valid
> UTF-8 char, a single U+DCxx value does not represent a valid UTF-16 char.
> Therefore, the ^N conversion will fail since U+DCxx can't be converted
> to valid UTF-8.
True, but that's an implementation issue rather than a design issue,
i.e. the ^N conversion needs to do the UTF-8 conversion itself rather
than invoke the __utf8 functions. Shall I look into creating a patch?
>> > So it looks like the current mechanism to handle invalid multibyte
>> > sequences is too complicated for us. =C2=A0As far as I can see, it wou=
ld be
>> > much simpler and less error prone to translate the invalid bytes simply
>> > to the equivalent UTF-16 value. =C2=A0That creates filenames with UTF-=
16
>> > values from the ISO-8859-1 range.
>>
>> This won't work correctly, because different POSIX filenames will map
>> to the same Windows filename. For example, the filenames "\xC3\xA4"
>> (valid UTF-8 for a-umlaut) and "\xC4" (invalid UTF-8 sequence that
>> represents a-umlaut in 8859-1), will both map to Windows filename
>> "U+00C4", i.e a-umlaut in UTF-16. Furthermore, after creating a file
>> called "\xC4", a readdir() would show that file as "\xC3\xA4".
>
> Right, but using your above suggestion will also lead to another filename
> in readdir, it would just be \x0e\xsome\xthing.
I don't think the suggestion above is directly relevant to the problem
I tried to highlight here.
Currently, with UTF-8 filename encodings, "\xC3xA4" turns into U+00C4
on disk, while "\xC4" turns into U+DCC4, and converting back yields
the original separate filenames. If I understand your proposal
correctly, both "\xC3\xA4" and "\xC4" would turn into U+00C4, hence
converting back would yield "\xC3\xA4" for both. This is wrong. Those
filenames shouldn't be clobbering each other, and a filename shouldn't
change between open() and readdir(), certainly not without switching
charset inbetween.
Having said that, if you did switch charset from UTF-8 e.g. to
ISO-8859-1, the on-disk U+DCC4 would indeed turn into
"\x0E\xsome\xthing". However, that issue applies to any UTF-16
character not in the target charset, not just those funny U+DC?? codes
for representing invalid UTF-8 bytes.
The only way to avoid the POSIX filenames changing depending on locale
would be to assume UTF-8 for filenames no matter the locale charset.
That's an entirely different can of worms though, extending the
compatibility problems discussed on the "The C locale" thread to all
non-UTF-8 locales, and putting the onus for converting filenames on
applications.
Andy
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
- Raw text -