Re: What influences C++ I/O performance?
On Feb 17, 2:28 pm, SzH <szhor...@gmail.com> wrote:
On Feb 17, 12:22 pm, James Kanze <james.ka...@gmail.com> wrote:
On Feb 16, 10:19 pm, rpbg...@yahoo.com (Roland Pibinger) wrote:
On Sun, 3 Feb 2008 06:04:39 -0800 (PST), SzH wrote:
I would like to read in large integer matrices from text files, as
quickly as possible.
...
It is more likely
that there are some relevant settings for iostreams (with a
significant impact on performance) that I am not aware of.
Any suggestions for making this program perform well portably would m=
e
most welcome!
Avoid iostreams. They are slow 'by design'.
Compared to what? The best implementations generally beat
stdio. (Most wide spread implementations are designed to be
simple, rather than fast, since experience has shown that
they're fast enough anyway.)
You know, it's just terribly disappointing and frustrating when one
finds out that even a stupid scripting language outperforms one's C++
implementation. For as long as I'm doing numerical stuff, I don't
even touch the standard library (std::vector and the likes).
Interesting. In the implementations I use, there is no
difference in performance between std::vector and a C style
array, at least when it comes to access times. About the only
time I use C style arrays today is when I need static
initialization (although I can also see their use for small,
fixed length arrays---constructing an std::vector of a fixed
size will definitely be more expensive than just defining a C
style array).
If I do, and by accident it isn't slow, there's a good chance
that trying a different compiler will make it dog slow.
I have a good feeling for what is slow and what is fast when
doing numerical calculations, so I implement my own classes.
But for I/O I *have* to rely on the standard library and I
have no idea about what are the little things that I have to
pay attention to avoid bad performance. I'm very
disappointed.
And, if possible, I don't want to overcomplicate things ...
C++ is supposed to make things easy and convenient over C, so
I don't want to use <stdio.h> when the C++ version is so much
cleaner and easier to write.
Yet, in the following example, even Python outperforms C++. I
just wanted to calculate how many distinct integers are there
in each row in the same matrix (same dataset).
The timings are:
gcc: 1:12.406
vs: 54.421
python: 17.312
gcc's fast iostreams are of no use here ... the slow std::set
makes it even slower than VS. Or maybe there is some little
trick that one should know about std::set to make it fast.
Is std::set doing too much. You don't need the order, and most
scripting languages use a hash table here, which will be faster
if there are a large number of disinct integers. Also, I don't
know Python, but some scripting languages will use the text
representation, directly read from the file, as an index, rather
than converting it to int (and thus, counting 01 an 1 as two
distinct integers).
But even if there is, it is not documented (for my compiler),
so an outsider like me cannot use it!
And before anyone accuses me for reading from a gzipped file:
just the decompression of the file takes only 1.750 seconds.
Gzipping the datafile actually increases performance for large
files, because it makes the data processing CPU-bound instead
of disk-bound.
----- count.cpp ---------
#include <iostream>
#include <sstream>
#include <string>
#include <set>
using namespace std;
int main() {
string line;
while (getline(cin, line)) {
istringstream ln(line);
set<int> specount;
int species;
while (ln >> species)
specount.insert(species);
cout << specount.size() << '\n';
}
return 0;
}
Of course, you've chosen an example where garbage collection
(and thus garbage collected languages) is a big win:-). It
should be easy to improve the performance here considerably by
using a custom allocator for the set. But you're right, you
shouldn't have to; the language should have garbage collection,
and handle this optimization on its own.
--
James Kanze (GABI Software) email:james.kanze@gmail.com
Conseils en informatique orient=E9e objet/
Beratung in objektorientierter Datenverarbeitung
9 place S=E9mard, 78210 St.-Cyr-l'=C9cole, France, +33 (0)1 30 23 00 34