Re: Best Way to Process Large Text Files
On Wed, 10 Feb 2010, Michael Powe wrote:
My thought was to have a read thread and a write thread and create a
buffer into which some amount of input would be written; and then, when
a threshold was reached, the data would be written out.
Is this a good idea?
I'm slightly skeptical. If the processing is simple, then most of the time
will be spend doing IO even with a simple implementation. Adding threads
to overlap IO and processing might not be a big win. You could try writing
a sequential version of the program (with sufficiently large buffers - a
few megabytes, maybe?), then measuring how fast it runs - if the total
input and output data rate is close to your storage subsystem's capacity,
then no amount of programming cleverness will make it much faster.
If, OTOH, there's significant headroom above the rate you reach, then
using threads as you describe would be a good thing to try. Either that or
non-blocking IO via the NIO package, but i think you'd get decent results
from threads.
And finally, I need pointers as to how I would create such a buffer. The
threaded read/write part I can do.
You could try java.io.PipedInputStream and PipedOutputStream. If you want
a bigger buffer, you could grab the code for these from OpenJDK and modify
it. Mind you, circular buffers are a pretty standard bit of programming,
so there will be dozens of other implementations and descriptions out
there on the web.
tom
--
It's rare that you're simply presented with a knob whose only two
positions are "Make History" and "Flee Your Glorious Destiny." --
Tycho Brahae