Mail Archives: cygwin/2011/01/29/10:10:33
--------------enigEB77E732F4D2D606FC5E086B
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
On 01/29/2011 05:30 AM, Corinna Vinschen wrote:
>> But when characters outside the basic plane, such as
>> U+12345 (CUNEIFORM SIGN URU TIMES KI), are encoded by 2 consecutive wcha=
r_t
>> values, values of type wchar_t don't correspond to ISO/IEC 10646 charact=
ers.
>> (Or maybe I'm underestimating what "coded representations" means...?)
>=20
> I don't read that from your above quote. The core is that the *type*
> wchar_t is a *coded* *representation* of the characters defined in
> 10646. At no point it says that a single wchar_t value must represent a
> single character from 10646. So I take it that UTF-16 is a valid, coded
> representation of the characters from 10646.
POSIX is clear that wchar_t must be wide enough so that 1 wchar_t is one
character. Which limits a 2-byte wchar_t to just the Unicode basic
plane. There's nothing cygwin can do about this other than break LOTS
of ABI to support a 4-byte wchar_t to supply all of Unicode.
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap06.html#tag=
_06_03
"All wide-character codes in a given process consist of an equal number
of bits. This is in contrast to characters, which can consist of a
variable number of bytes. The byte or byte sequence that represents a
character can also be represented as a wide-character code.
Wide-character codes thus provide a uniform size for manipulating text
data."
So, using UTF-16 surrogate encodings for characters outside the basic
plane violates POSIX, but it's the best we can do for those characters.
> I've put a lot of effort in 2009 and early 2010 to make the wchar_t
> representation in Cygwin and newlib as much Unicode 5.2 compatible as
> possible. Even the wcrtomb and mbrtowc functions in newlib are capable
> of dealing with UTF-16 surrogates.
And I appreciate that effort - even though it means wchar_t is just as
painful as multi-byte char characters in that an array of wchar_t is not
necessarily that many characters long, but only when surrogates are
involved.
>=20
> However, given that Windows XP basically only supports the charset from
> Unicode 4.0, and given that Cygwin's support for east-asian double and
> triple byte codesets (Big5, GBK, eucKR, eucJP, and a SJIS/CP932 bastard)
> still requires the underlying Windows conversion functions, I've set
> __STDC_ISO_10646__ to a value which reflects Unicode 4.0 (200305L) for
> Cygwin 1.7.8.
Someday when gcc has better support for C+1x 16- and 32-bit characters
(regardless of the sizing of wchar_t), then we can add all the new
32-bit character APIs that use Unicode unimpeded, without breaking
existing ones that use wchar_t.
--=20
Eric Blake eblake AT redhat DOT com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
--------------enigEB77E732F4D2D606FC5E086B
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iQEcBAEBCAAGBQJNRC3aAAoJEKeha0olJ0NqQ98IAIE5vfjLBgcVJ7thBoEFZv/W
KwsuOkwMAygUqM/kK3gL5EnHEqK/LSsLkNz/Qbdva+mcpzRCB570ys4hbthnjJTt
PIOAvDqGWa7PYN7DqKayDnIp8tWmgtkgsoCvXhiF+XuKB8zbhlpQo5g+PkKbU7Y3
2FdBOtHIqLPli2Xhv6+Nie+OkvLLaQV3DJuaqVuPtxUWAsliFM0P7sbzk6CZ9Nq0
ZUvsPuZUN3tfcM98vMrRfL0wJLHSprF61nPX8vJrHdRFMTVRPy56acW5L65mfxs3
QQgoE3ht8HK5Ve//olXQOMnkjyqiTqHuN62uTu6ja4FmW8aB6DYDw8RgLe9dwGQ=
=eZ/A
-----END PGP SIGNATURE-----
--------------enigEB77E732F4D2D606FC5E086B--
- Raw text -