Re: How can you make idle processors pick up java work?
Patricia Shanahan wrote:
qwertmonkey wrote:
...
Please, let me know if you have a better idea of how
to read very large text
files in a faster way (while using a regular/sub-optimal box) ~
...
Another approach is to farm out one file per thread, rather
than multiple threads per file.
Part of the game of concurrency is to figure out the right
size for a chunk.
Your problem also seems amenable to a map-reduce approach.
It sounds as though your throughput is limited by disk read head time -
splitting across physical disks increases that resource.
You *may* be able to improve the efficiency of one drive by using NIO
non-blocking reads to keep more disk prefetch reads going at once. Given
a pool of reads, the drive can optimize their order to reduce total head
movement, and therefore time the read head wastes not actually doing a
transfer.
You can also perhaps improve efficiency with the one thread-per-file
approach.
I/O in each thread should be bursty, especially if you use
'BufferedReader' with large buffers. Each thread's I/O time
is potentially CPU time for other threads. Also, multiple
threads' I/O might bunch up for elevator seeking as Patricia
describes.
However, threads hit a limit of usefulness if you get too many.
Fortunately that limit should be higher than that imposed by
your I/O subsystem. You control this by setting your thread
pool's maximum count and other control factors appropriately.
For the kind of situation you describe I'd predict the
optimum maximum to be somewhere around two to four times your
core count.
I don't know whether NIO or thread-based expansion will help
you more. Even the predictions I am making are implicity
heavily burdened with caveats to measure and take nothing for
granted.
I'd likely use a one thread-per-file approach for structural
reasons, without much initial concern for whether it's the
best performing. It's a clean architecture that allows good
locality of working variables and a natural path to a service
approach. I wouldn't expect any optimization beyond large
buffers to help individual file performance, so I'd plan for an
architecture that scales well with increased hardware. One file
thread per file scales well to one service call per file,
readily distributed across a cluster. It also fits well with
expansions to an I/O subsystem.
Amdahl's Law is your friend if you use it to plan units of
work that have little to no common paths.
--
Lew