Re: Fast way to read a text file line by line

On May 10, 1:49 pm, Thomas Kowalski <t...@gmx.de> wrote:


The fastest solution is probably mmap, or it's equivalent under
Windows, but that's very system dependent. Other than that, you
can read the file in one go using something like:
    std::istringstream tmp ;
    tmp << file.rdbuf() ;
    std::string s = tmp.str() ;
or:
    std::string s( (std::istreambuf_iterator< char >( file )),
                   (std::istreambuf_iterator< char >()) ) ;

If you can get a good estimation of the size of the file before
hand (which again requires sytem dependent code), then using
reserve on the string in the second example above could
significantly improve performance; as might something like:

    std::string s ;
    s.resize( knownFileSize ) ;
    file.read( &s[ 0 ], s.size() ) ;

The only system I know where it is even possible to get the
exact size of a text file is Unix, however; under Windows, all
of the techniques overstate the size somewhat (and under other
systems, it might not even be possible to get a reasonable
estimate). So you might want to do something like:
    s.resize( file.gcount() ) ;
after the above. (It's actually a little bit more complicated.
If there are not at least s.size() bytes in the file---and under
Windows, this will usually be the case if the file is opened as
text, and GetFileSizeEx was used to obtain the size---then
file.read, above, will appear to fail. In fact, if gcount() is
greater than 0, it will have successfully read gcount() bytes,
and if eof() has been set, the read can be considered to have
successfully read all of the bytes in the file.)

(Note that this is not guaranteed under the current C++
standard. It works in practice, however, on all existing
implementations of the library, and will be guaranteed in the
next version of the standard.)

Two other things that you might try:

 -- using std::vector< char > instead of std::string---with some
    implementations, it can be faster (especially if you
    construct the string using istreambuf_iterator), and

 -- reading the file as binary, rather than text, and handling
    the different end of line representations manually in your
    own code.

Concerning the latter, be aware that on some systems, you cannot
open a file as binary if it was created as text, and vice versa.
Just ignoring extra '\r' in the text is often sufficient,
however, for this solution to work adequately under both Unix
and Windows; ignoring only the '\r' which immediately precede a
'\n' is even more correct, but often not worth the extra bother.
And if the file is read as binary, both stat (under Unix) and
GetFileSizeEx (under Windows) will return the exact number of
bytes you can read from it.

--
James Kanze (GABI Software) email:james.kanze@gmail.com
Conseils en informatique orient=E9e objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place S=E9mard, 78210 St.-Cyr-l'=C9cole, France, +33 (0)1 30 23 00 34