Re: Converting EBCDIC to Unicode
On Sep 30, 7:46 am, Saeed Amrollahi <amrollahi.sa...@gmail.com> wrote:
On Sep 28, 7:33 pm, James Kanze <james.ka...@gmail.com> wrote:
On Sep 28, 8:27 am, Saeed Amrollahi <amrollahi.sa...@gmail.com> wrote:
I wrote a program to convert a EBCDIC text file in OS/400
environment to Unicode (UTF-16) in Windows XP. Because, the
text file contains information of Shareholders in Persian
(Farsi), I had to find the mapping table of Persian
characters. You may be know, Unlike English, in Persian some
characters has one form, some of them two forms and for some
characters, there are more than two forms. I mean there are
Initial, Medial and Final forms.
And isolated, no?
Yes. That's right. You are clever.
But that's usually a problem for the rendering machine, not for
your program.
I can't understand. By rendering machine, what do you mean? You mean
my local computer?
Rendering machine or rendering engine. The mechanism which
converts the internal code to human readable format. In
other words, the encoding should just store the letters,
without regards to the form. The engine which actually
generates the display or the graphic format should choose
the appropriate form depending on context.
I found them using Character Map (One of System Programs in
Windows XP). I really like to know your general and special
opinion. If someone already worked on the subject even in
other languages (like Arabic) h(is/er) advice may be help so
much.
1. Because the EBCDIC is 8-bits encoding and Unicode (UTF-16)
is 16 bits (or more precisely 21 bits) encoding, I use for
input file an ifstream object (character files) and for output
file wofstream object (Wide character file)
That's the way it was designed to work. (Actually, it was
designed so that you imbue a Persian EBCDIC local in a wifstream
when reading. If you can find such.)
2. I use the int() function to know the ordinal number behind the
characters.
In C++, all you have *is* the ordinal number. What you probably
do have to do is convert the input char to unsigned char.
OK. I try it.
I use the convention:
If the returned number is positive, it should be English
letter or numeric, in other words it isn't Persian and If it
is negative, it is Persian
You can't count on that. The type char may be signed or
unsigned. Convert to unsigned char, then compare to 128.
OK.
Except that that doesn't work at all for EBCDIC, where 'a' is
0x81, and the Persian characters are probably scattered about in
the unused spaces. Or it uses some sort of shift-in/shift-out
scheme with two different encodings. Or IBM has given up on
EBCDIC for non Latin scripts, and is using ISO 8859-6 or MS
Windows CP-1256 (although I'm not sure that either of these has
the extra characters needed for Persian).
<Nod> You are right. The Persian characters are scattered
in unordered way in unused space. An analogy: 'b' is not after 'a'
necessarily. Would you mind explain the Shift-in/Shift-out scheme?
A shift-in/shift-out scheme is a solution which basically
uses two different encodings, with special characters to
shift from one to the other. With 7 bit characters, for
example, one might have one encoding for Persian characters,
another for Latin (with some common characters like space in
both), and two reserved codes, one which says that what
follows is Latin, the other that what follows is Persian.
Such schemes were common many years ago. They have serious
disadvantages (like, loose one of the shift characters in
translation, and everything is off, or that you can't just
skip ahead n characters without looking at every character).
From what you say above, I don't think this is your case.
About Windows Code Page 1256, there is a problem in my
current project. As you know, there is just one form of
each Persian character (the Initial/Medial one), for the
isolated/Final, a space should be added to the word. It is
the problem.
I'm not at all familiar with the Windows code pages. I do
know that in general, Arabic (and certainly also Persian)
normally only encode the character, not its form. It's only
when rendering that the correct form is chosen, according to
context.
In current application, there is another problem with
CP-1256. We have a field with 3 Persian characters (The
first 3 characters of shareholder family name) and
5 digits and They are concatenated. take an Analogy:
'Amr00023' Unfortunately, in CP-1256, after meet the first
digit, the last character will change from medial form to
final form and it is wrong.
That sounds like a bug in the rendering engine. Or maybe in
your expectations: I would expect a final form before
a sequence of digits, see section 3.5 of
http://www.unicode.org/reports/tr9/#Shaping. (If I'm not
mistaken, digits are right to left in Persian, which means
that there is a change in the direction when you switch from
letters to digits.)
[...]
A better solution would be to create a codecvt facet, and use it
directly in the istream.
OK. I try it.
Just be warned that it is more work. The codecvt has
a somewhat perverted interface (probably because it was
designed before there was an std::string).
[...]
I want to extend my program to convert Unicode to EBCDIC,
EBCDIC to XML, ... I mean Generic converter.
You mean iconv. It already exists.
I don't know iconv. Is it the product by Dinkumware company?
No. It's GPL (I think---a free to use license, anyway).
It's both a library, for use within your code, and
a stand-alone command line program. It's generally part of
Unix distributions, but you can get it for Windows as well.
How to apply Policy class design?
Generally, I've hear policy used to refer to some sort of
template metaprogramming technique. Perhaps you mean the
strategy pattern.
By policy class I mean something like this (Pseudo-code):
template<class ConversionPolicy>
class Convertor {
// ...
public:
convert();
};
Convertor<EBCDIC2Unicode> c;
c.convert();
OK. In my experience, using the strategy pattern is
preferrable. Sooner or later, you'll end up wanting the
decision to be made at run-time.
--
James Kanze