Re: Best Way to Process Large Text Files
On 2/10/2010 3:28 AM, Michael Powe wrote:
Hello,
I am tasked with writing an application to process some large text
files, i.e.> 1 GB. The input will be csv and the output will be in the
format of an IIS web server log.
I've done this sort of thing before. In the past, I've just
brute-forced it, with a BufferedReader and BufferedWriter handling the
input/output line by line.
I have a little time to complete this project and I'd like to build
something more efficient, that won't peg the CPU for an hour.
My thought was to have a read thread and a write thread and create a
buffer into which some amount of input would be written; and then, when
a threshold was reached, the data would be written out.
Is this a good idea? Are there better ways to manage this?
And finally, I need pointers as to how I would create such a buffer.
The threaded read/write part I can do.
Thanks for any help.
mp
Depending on how processor intensive the transformation is, you might
not gain anything from threading.
If you are using regex to parse, you may be better off optimizing your
regexs, or using hand-coded parsing instead. A naive regex which "works"
may have some performance problems. Use greedy matching where
appropriate is one way to improve performance.
--
Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>