Re: How to read uncode encoded files?

From:

Ulrich Eckhardt <eckhardt@satorlaser.com>

Newsgroups:

microsoft.public.vc.language

Date:

Thu, 26 Jul 2007 09:29:29 +0200

Message-ID:

<r3vln4-97n.ln1@satorlaser.homedns.org>

Anddy wrote:

I try to read unicode encoded files.

Unicode (capital 'U') is not an encoding but a whole standard that defines
several encodings. Keep that in mind!

File starts with unicode BOM (0xFEFF).

Here's the file content.

FF FE 42 00 45 00 47 00 49 00 4E 00

Okay, this looks like little-endian UTF-16 or UCS2, both defined by the
Unicode standard. If I had the choice, I would prefer UTF-8 though.

And I use following code.

if ((fd = _open(buffer, _O_RDONLY)) != -1)
{
while (_read(fd,&mem, 1) == 1)
;
_close(fd);
}

When I check the contents of 'mem'.

The contents of 'mem' are

42 45 47 49 4E

Why this happen?

How can I read Unicode BOM (0xFEFF)?

It might be the case that you are screwed by a locale-specific conversion
performed by the C implementations of VC8[1]. It sees that there is a BOM
and then transparently transcodes the file to the internally used charset.
In that case, I suggest that you use the 'binary' flag when opening the
file (which you should do anyway) and maybe invoke a 'setlocale("C");' or
something like that to set the locale to neutral.

Uli

[1] I hope I get this point right, I don't exactly remember what and where
these conversions took place.