Re: File Processing

From:

Victor Bazarov <v.Abazarov@comAcast.net>

Newsgroups:

comp.lang.c++

Date:

Tue, 30 Sep 2008 15:35:24 -0400

Message-ID:

<gbtv1t$kn6$1@news.datemas.de>

Jeff wrote:

I want to read and process and rewrite a very large disk based file
(>3Gbytes) as quickly as possible.
The processing effectively involves finding certain strings and replacing
them with other strings of equal length such that the file size is unaltered
(the file is uncompressed btw). I wondered if anyone could advise me of the
best way to do this and also of things to avoid. More specifically I was
wondering :-

-Is it best to open a single file for read-write access and overwrite the
changed bytes or would it be better to create a new file?

It is always a good idea to leave the old file intact, unless you
somehow can ensure that a single write operation will never fail and
that an incomplete set of find/replace operations is still OK. Ask in
any database development newsgroup.

-Is there any point in buffering bytes in rather than reading one byte at a
time or does this just defeat the buffering that's done by the OS anyway?

You'd have to experiment. C++ language does not define any buffering
AFA OS is concerned.

-Would this benefit from multi-threading - read, process, write?

Unlikely. Processing will take so little time compared to the I/O, and
I/O is going to be the bottleneck anyway, so...

[..]

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask