Re: Mixing text and binary I/O
"Ivan Voras" <ivoras@fer.hr.nije> wrote in message
news:ecq2h0$813$1@news2.carnet.hr...
In case of complex encodings like UTF-8, I'd expect (and will probably
create for my case) its behaviour to be like this:
- Backed by a buffer (the usual way, probably byte[])
In fact, I think you can build it on top of an InputStream, which is more
flexible and more general, since all you need is a source of bytes.
- readByte() reads from the buffer, handles buffering of new data, etc.
Let the underlying stream handle buffering.
- readChar() reads as much bytes as it needs to reconstitute a
character, in case of UTF-8 it could be one or several - it doesn't
matter. If it encounters an invalid byte (by the expectations set by
used encoding), raise proper exception because it's an encoding error in
the stream.
I don't know how to build this in general. It's mostly straightforward to
build for a specific encoding, say UTF-8, but CharsetDecoder has no method
that means "decode exactly one character". (I suppose you could give it one
byte, then two, then three, etc. until it stoips returning a failure status,
but that seems inelegant.) Even in UTF-8, you get oddities where a
codepoint > FFFF returns two characters; returning the first consumes 4
bytes, and returning the second consumes 0 bytes. In other words, you'd
have to be careful with logic like "I know that this set of characters
occupies bytes 3-10, and I've processed all of them, so I'll switch to
reading bytes again."
- Introduce private or protected pushByte() and pushChar() that do the
reverse of readXXX, on the buffer. "Fixup" the fact that one character
can have more bytes by initially making the buffer 4+ bytes longer, but
don't use this extra space when filling the buffer in readByte(). Like
in C, make pushXXX work only for a single byte/character.
- Modify readLine() to use readChar(), reads characters until CR+LF; can
use existing logic that reads one char after CR to see if it's LF and
push it back if it isn't.
More precisely, reads until CR, LF, or CRLF. You're right that pushing back
a non-LF after CR is easy enough.
- Every other readXXX method uses readByte() as usual.
The intended result: freely mix bytes and characters. In the extreme
(but supported!) case, the stream can have a UTF-8 character (encoded by
one or several bytes) followed by a "raw" byte, followed by a UTF-8
character, etc. The programmer is responsible to know how the stream is
formatted.