Re: Binary or text file
On May 14, 2:43 pm, Gianni Mariani <gi3nos...@mariani.ws> wrote:
On May 14, 5:24 pm, James Kanze <james.ka...@gmail.com>
wrote:> On May 13, 1:40 pm, Gianni Mariani
<gi3nos...@mariani.ws> wrote:
[...]
Seriously, do you really believe that you can judge people by
their government? Even in so-called democracies, like France
and the USA. I've lived in three different countries, and I've
very close contacts with a fourth (my wife is Italian). I've
found people to be pretty much the same everywhere, and in the
vast majority, I've found them to be pretty decent.
Same here. Almost exactly but they are all European.Roman in
heritage. Try spending some serious time in India, Thailand, PRC or
Hong Kong though, and then some more time in the Saudi, or UAE or
Nigeria even. The cultural skew takes some time to come to grips
with.
Yes, but it's really still very superficial. Human nature is
human nature. It does make it more difficult to recognize the
similarities, however.
[...]
I have applications that still generate them. I still
generate them. I see them every day. Accusing me of being
parochial is very arrogant and disingenuous.
So what machine are you using? Posix requires 8 bit characters,
and it doesn't have a function "isascii" anymore---it requires
full support for an eight bit character set. And of course,
correct code will not fail because some file happens to contain
an accented character. You can pretend that your files are
ASCII, but that's just pretending.
You talk about processing technology, I talk about actual files. See,
not so deep and meaningful. It all works back to the context of the
original statement. Way too much energy spent here.
They're related, but my real point was different. Perhaps if I
stated it something along the lines "a correct program cannot
assume that any file it reads contains only characters in the
ASCII character set."
It's a conceptual point of view. When I first started working
on Unix, we pretty much considered that all text files were
ASCII. In some ways, it was false even then; the OS never made
the slightest guarantee, and characters with the 8th bit set did
creep into text files from time to time. But we had a function,
isascii(), which we used to test for such characters, and if
they were present, we rejected the file as being corrupt.
Today, of course, we no longer have that function, and every
editor, on every system, is capable of generating accented
characters. So the files aren't really ASCII, but whatever
encoding the editor was generating (ISO 8859-1 seems very
common). And of course, a correct program will handle them
correctly.
Now, you may say that all, or almost all of the files you have
to deal with actually only contain characters in the subset
common to ASCII, the ISO 8859 encodings and UTF-8. That may be
(although it's not the case where I work, and hasn't been for
well over 10 years). But I insist that that is not an
appropriate way of thinking about it. Those files were created
by an editor, or some other program, which is perfectly capable
of creating characters which are not in ASCII. And considering
them "pure" ASCII will lead to carelessness in programming, and
an increased risk of errors.
In that sense, ASCII files simply do not exist. There is no way
you can open a file, and say, this file is pure ASCII, and
cannot possibly contain anything else. I also suspect that it
is exceedingly rare that you can open a text file saying: this
file should be pure ASCII, and anything else means it is
corrupt. There are doubtlessly exceptions to this, particularly
with regards to machine generated data. But most of the
exceptions I know go even further: if the file contains, say, a
list of floating point values, then it is corrupt if it contains
any alpha character, not just if it contains an accented
character.
[...]
...
Life with Unicode is much easier. Surprising little code really =
needs
to care that it is parsing utf-8.
Are you kidding? What about code which uses e.g. "isalpha()".
Ok, you need to think a little harder at what you're trying to do.
In general. Once you can no longer count on just ASCII, you do
have problems. Regardless of the encoding. On the whole, I
think UTF-8 is the only viable solution for communications, and
it is also the prefered solution for internal coding for a lot
of applications. Other applications will prefer UTF-32. And a
number of applications will still make do with some pure 8 bit
encoding, ISO 8859-1, or such.
At one point I was asked to give a recommendation on
internationalizing and application. It was a web browser. My default
answer was "wide chars", etc etc I examined the code and realized I'd
given the project a death sentence because there was no way the
project would recover so I went back to the team and said - JUST
KIDDING. What you need is utf8 with one of these special string
classes that converts a string transparently between utf-8 and utf-16
whenever it needs to and slowly move more of the application over to
wide char code. The code was migrated when it needed to and much of
the application didn't need touching.
The main point of this was that the codebase never broke uncontainably
and it's i18n support improved incrementally until it was adequate
without needing to interrupt development of other parts of the
product.
I presume you're talking about internal representation here. A
Web browser certainly has to deal with a large number of
different external encodings. If I control the entire chain,
there's no doubt that everything would be Unicode, UTF-8
externally, and either UTF-8 or UTF-32 internally, depending on
what I was doing. But I never do control the entire chain: here
at work, the powers that be haven't installed any Unicode fonts
on the machines, so I'm stuck with ISO 8859-1:-(.
Some code will break because it splits characters or it
compares un-normalized strings, but these problems are far
easier to deal with than the mish-mash of encodings in the
past.
Easier, yes, but not all of the tools are necessarily in place.
Things like "isalpha()" are an obvious problem.
There is a need to standardize on something that handles all these
things - ICU is the only thing I have seen that gets close.
They seem to have done the most work in this direction to date.
On the other hand, they use UTF-16, which doesn't seem a
judicious choice today: UTF-32 or UTF-8 would seem preferable,
depending on what the program is doing.
Yeah, I recall having the same thought now. You should find this one
amusing:
http://mail-archives.apache.org/mod_mbox/xerces-c-dev/200007.mbox/%3c397B=
A2AB.CD58D...@orconet.com%3e
Time for a new ICU.
:-). To be fair to them: when they defined their spec, Unicode
was only 16 bits. Also, any program really treating text
seriously will have to deal with various composite characters
anyway, and handling the surrogates isn't that much more work.
On the other hand, the more I work with such characters, the
more I realize that you can do directly in UTF-8. Multibyte
characters have a reputation for causing all sorts of problems,
but UTF-8 has addressed some of the issues (and of course, a lot
of the problems are just because the code isn't prepared for
multibyte characters). Once you're handling surrogates and
composite characters, is UTF-8 really any more difficult than
UTF-32?
--
James Kanze (GABI Software) email:james.kanze@gmail.com
Conseils en informatique orient=E9e objet/
Beratung in objektorientierter Datenverarbeitung
9 place S=E9mard, 78210 St.-Cyr-l'=C9cole, France, +33 (0)1 30 23 00 34