Re: Binary file IO: Converting imported sequences of chars to desired
type
On Oct 25, 3:13 pm, Rune Allnor <all...@tele.ntnu.no> wrote:
On 23 Okt, 10:27, James Kanze <james.ka...@gmail.com> wrote:
On Oct 23, 9:07 am, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
[...]
But if you have a choice, it's IMO almost always better to
write the data as text, compressing it first using something
like gzip if I/O or disk space is an issue.
Totally agreed. Especially for the maintenance programmer,
who can see at a glance what is being written.
The user might have opinions, though.
File I/O operations with text-formatted floating-point data
take time. A *lot* of time.
A lot of time compared to what? My experience has always been
that the disk IO is the limiting factor (but my data sets have
generally been very mixed, with a lot of non floating point data
as well). And binary formatting can be more or less expensive
as well---I'd rather deal with text than a BER encoded double.
And Jorgen said very explicitly "if you have a choice".
Sometimes you don't have the choice: you have to conform to an
already defined external format, or the profiler says you don't
have the choice.
The rule-of-thumb is 30-60 seconds per 100 MBytes of
text-formatted FP numeric data, compared to fractions of a
second for the same data (natively) binary encoded (just try
it).
Try it on what machine:-). Obviously, the formatting/parsing
speed will depend on the CPU speed, which varies enormously. By
a factor of much more than 2 (which is what you've mentionned).
Again, I've no recent measurements, so I can't be sure, but I
suspect that the real difference in speed will come from the
fact that you're writing more bytes with a text format, and on a
slow medium, that can make a real difference. (In one
application, where we had to transmit tens of kilobytes over a
50 Baud link---and there's no typo there, it was 50 bits, or
about 6 bytes, per second---we didn't even consider using text.
Even though there wasn't any floating point involved.)
In heavy-duty data processing applications one just can not
afford to spend more time than absolutely necessary.
Text-formatted data is not an option.
I'm working in such an application at the moment, and our
external format(s) are all text. And the conversions of the
individual values has never been a problem. (One of the formats
is XML. And our disks and network are fast enough that even
that hasn't been a problem.)
If there are problems with binary floating point I/O formats,
then that's a question for the C++ standards committee. It
ought to be a simple technical (as opposed to political)
matter to specify that binary FP I/O could be set to comply to
some already defined standard, like e.g. IEEE 754.
So that the language couldn't be used on some important
platforms? (Most mainframes still do not use IEEE. Most don't
even use binary: IBM's are base 16, and Unisys's base 8.) And
of course, not all IEEE is "binary compatible" either: a file
dumped from the Sparcs I've done most of my work on won't be
readable on the PC's I currently work on.
--
James Kanze