Re: problem with storing greek chars to a buffer (os linux)
nass wrote:
i am not sure how to tackle this problem or where is originates from so
i am writing here in hope that if you can not help you can at least
point me in a direction.
I am writing some little program using only standard c++ library and i
am opening a file that contains strings.
A text file, you mean, organized into lines of text.
there are numbers , english
and greek letters among the characters.
A text file with a non-standard encoding, thus. There are no
Greek letters in the basic character set; you must use some
extended encoding.
the reading is done fine and
the data are then stored onto a buffer (without undergoing any
processing) as a few int values ( the lengths of the strings), followed
by a long c string - the concatenation of all the strings of the file.
the receive application can then use the lengths of the strings to
separate the strings from the string again.
And now things become sticky. What do you mean by "the lengths
of the strings"? The number of bytes, or the number of
characters? And how do you determine it?
so examining the buffer contents in the shared memory i found that the
numbers have their correct corresponding values (0x30 for char '0'
etcetc.) and the lengths of the strings in the c string are correct
too. but the greek characters and system chars found among them (like
spacebar) where wrong
By spacebar, I presume you mean the ASCII character space, 0x20
(probably---I'll suppose you're not dealing with EBCDIC).
- did not have their expected (extended) ascii
values.
Space is a character in the basic execution set, and in basic
ASCII (where it has code 0x20). I'm not sure which encoding you
mean by extended ASCII; there are no extensions to ASCII, but
there are a large number of different 8 bit encodings which
ensure that the first 128 characters are the same as in ASCII.
(For Greek, the two I'm familiar with are UTF-8 and ISO 8859-7.)
the funny thing is if i printf the strings of the file, they
appear correctly on console!!
What's so strange about that?
i have set LC_ALL environment variable in my linux machine to
en_US.UTF-8, just in case this is an important detail.
It isn't unless you imbue the stream with locale "" (which tells
the library to use the default locale for the environment). By
default, the stream uses locale "C", which guarantees full
binary transparency; the bytes you read are the bytes in the
file.
and the file was
written from vim (not internally from the program).
You'll have to check with vim to see what it writes. (My
versions are configured to write ISO 8859-1, but I think it can
be configured to use any of the ISO 8859 codes, and maybe even
UTF-8.)
is it possible that
the file is written in utf-16 format? utf-8? could it be something
wrong in the code?
Without seeing the contents of the file, it's fairly difficult
to say how it is encoded. Even seeing them, it's not certain
that one could tell.
Nothing to do with C++, but if you are under Linux, you can
write a file with a single character, say Greek small letter pi,
then look at it using "od -t x1". You will then see the hex
codes which vim generates for this letter (followed by a 0x0A,
since vim never generates a text file without a final newline).
If the file contains 0xf0 0x0a, it is encoded using ISO 8859-7;
if it contains 0xcf 0x80 0x0a, it's UTF-8; UTF-16 would surprise
me on a Linux system, but would be 0xc0 0x03 0x0a 0x00 or 0x03
0xc0 0x00 0x0a, possibly preceded by a 0xff 0xfe or a 0xfe 0xff;
UTF-32 would contain something like 0xc0 0x03 0x00 0x00 (or
possibly in the reverse order).
(see below)
CODE:
once i have the independent strings with 'loadInfoConf()' i serialise
them and send them to the shMem using:
string
detailsStr;
details.writeStructToStream(GUILastMessage.oNL,GUILastMessage.oTL,GUILastMessage.sNL,GUILastMessage.sLL,GUILastMessage.iDL,GUILastMessage.mDL,detailsStr);
strcpy(&GUILastMessage.oAndS_str,detailsStr.c_str());
---------------------------------------------------------------
void InfoClass::loadInfoConf()
{
string curLine="",sumLine="", lines[6];
int i=0;
ifstream infoConfFile(INFOCONF_FILENAME);
if (infoConfFile.is_open())
{
while (!infoConfFile.eof())
Just a nit, but this will terminate too soon if the last line
isn't correctly terminated. You almost never use eof() on a
stream, and never before the stream has failed.
The "standard" idiom for reading lines is:
while ( std::getline( infoConfFile, curLine ) ) {
// process line...
}
For the rest, there's not much to say. Character encoding is a
thorny issue, and requires everyone who processes the characters
to be in sych: the C++ program may think it is dealing with
UTF-8, but if the fonts active in display says ISO 8859-7, it's
going to appear as ISO 8859-7. You don't say what you had, and
what you expected, so it is difficult to say more, but as soon
as you leave the simple world of US ASCII, things become
complicated. (Under X, it's quite possible to set up different
console windows to use different fonts, so cat of your file will
appear different in different windows.)
--
James Kanze (GABI Software) email:james.kanze@gmail.com
Conseils en informatique orient?e objet/
Beratung in objektorientierter Datenverarbeitung
9 place S?mard, 78210 St.-Cyr-l'?cole, France, +33 (0)1 30 23 00 34
--
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]