Mail Archives: cygwin/2009/09/22/13:07:29
On Sep 22 17:12, Andy Koppe wrote:
> 2009/9/22 Corinna Vinschen:
> >> Therefore, when converting a UTF-16 Windows filename to the current
> >> charset, 0xDC?? words should be treated like any other UTF-16 word
> >> that can't be represented in the current charset: it should be encoded
> >> as a ^N sequence.
> >
> > How? Just like the incoming multibyte character didn't represent a valid
> > UTF-8 char, a single U+DCxx value does not represent a valid UTF-16 char.
> > Therefore, the ^N conversion will fail since U+DCxx can't be converted
> > to valid UTF-8.
>
> True, but that's an implementation issue rather than a design issue,
> i.e. the ^N conversion needs to do the UTF-8 conversion itself rather
> than invoke the __utf8 functions. Shall I look into creating a patch?
Well, sure I'm interested to see that patch (lazy me), but please note
that we need a snail mailed copyright assignment per
http://cygwin.com/assign.txt from you before we can apply any significant
patches. Sorry for the hassle.
Hmm... maybe it's not that complicated. The ^N case checks for a valid
UTF-8 lead byte right now. The U+DCxx case could be handled by
generating (in sys_cp_wcstombs) and recognizing (in sys_cp_mbstowcs) a
non-valid lead byte, like 0xff.
> >> This won't work correctly, because different POSIX filenames will map
> >> to the same Windows filename. For example, the filenames "\xC3\xA4"
> >> (valid UTF-8 for a-umlaut) and "\xC4" (invalid UTF-8 sequence that
> >> represents a-umlaut in 8859-1), will both map to Windows filename
> >> "U+00C4", i.e a-umlaut in UTF-16. Furthermore, after creating a file
> >> called "\xC4", a readdir() would show that file as "\xC3\xA4".
> >
> > Right, but using your above suggestion will also lead to another filename
> > in readdir, it would just be \x0e\xsome\xthing.
>
> I don't think the suggestion above is directly relevant to the problem
> I tried to highlight here.
>
> Currently, with UTF-8 filename encodings, "\xC3xA4" turns into U+00C4
> on disk, while "\xC4" turns into U+DCC4, and converting back yields
> the original separate filenames.
Well, right now it doesn't exactly.
> If I understand your proposal
> correctly, both "\xC3\xA4" and "\xC4" would turn into U+00C4, hence
> converting back would yield "\xC3\xA4" for both. This is wrong. Those
> filenames shouldn't be clobbering each other, and a filename shouldn't
> change between open() and readdir(), certainly not without switching
> charset inbetween.
I see your point. I was more thinking along the lines of how likely
that clobbering is, apart from pathological testcases.
> Having said that, if you did switch charset from UTF-8 e.g. to
> ISO-8859-1, the on-disk U+DCC4 would indeed turn into
> "\x0E\xsome\xthing". However, that issue applies to any UTF-16
You don't have to switch the charset. Assume you're using any
non-singlebyte charset in which \xC4 is the start of a double- or
multibyte sequence. open ("\xC4"); close; readdir(); will return
"\x0E\xsome\xthing" on readdir.
Only singlebyte charsets are off the hook. So, your proposal to switch
to the default ANSI codepage for the C locale would be good for most
western languages, but it would still leave the eastern language users
with double-byte charsets behind.
Note that I'm not as opposed to your proposal to use the ANSI codepage
as before this discussion. But I would like to see that the solution
works for most eastern language users as well.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
- Raw text -