Re: Why No Supplemental Characters In Character Literals?
On 02/04/2011 12:10 PM, Mike Schilling wrote:
"Arne Vajh??j" <arne@vajhoej.dk> wrote in message
But since codepoints above U+FFFF was added after the String
class was defined, then the options on how to handle it were
pretty limited.
The sticky issue is, I think, that chars were defined as 16-bit. If that
had been left undefined, they could have been extended to 24 bits, which
would make things nice and regular again.
Well, the real problem is that Unicode swore that 16 bits were enough
for everybody, so people opted for the UTF-16 encoding in Unicode-aware
platforms (e.g., Windows uses 16-bit char values for wchar_t). When they
backtracked and increased the count to 20 bits, every system that did
UTF-16 was now screwed, because UTF-16 "kind of" becomes a
variable-width format like UTF-8... but not really. Instead you get a
mess with surrogate characters, this distinction between UTF-16 and
UCS-2, and, in short, anything not in the Basic Multilingual Plane is a
recipe for disaster.
Extending to 24 bits is problematic because 24 bits opens you up to
unaligned memory access on most, if not all, platforms, so you'd have to
go fully up to 32 bits (this is what the codePoint methods in String et
al. do). But considering the sheer amount of Strings in memory, going to
32-bit memory storage for Strings now doubles the size of that data...
and can increase memory consumption in some cases by 30-40%.
To make a long story short: Unicode made a very, very big mistake, and
everyone who designed their systems to be particularly i18n-aware before
that is now really smarting as a result.
It actually is possible to change the internal storage of String to a
UTF-8 representation (while keeping UTF-16/UTF-32 API access) and still
get good performance--people largely use direct indexes into strings in
largely consistent access patterns (e.g., str.substring(str.indexOf(":")
+ 1) ), so you can cache index lookup tables for a few values. It's ugly
as hell to code properly, taking into account proper multithreading,
etc., but it is not impossible.
--
Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth