Re: I-O Performance
Eric Sosman wrote:
Lew wrote:
The output files will be a limiting factor. The requirement that
"[l]ines will be appended unchanged to a small number of output files"
implies that consumers of the Readers will have to wait their turn at
the output side to prevent interleaving.
This is a classic producer-consumer scenario with a few standard
solution architectures.
You seem to have extracted a lot more detail from the
problem statement than I was able to find. The only things
I know (think I know) about the problem are:
- There is a "large number" of input files,
- They contain "lines" (not, e.g., serialized JButtons),
- They are "relatively small," containing "a few hundred
lines each,"
- The lines are "appended" to output files (whether those
pre-exist or not isn't clear),
- The number of output files is "small," and
- An existing solution uses Buffered{Reader,Writer}.
I don't even know for sure that the performance of the
existing solution is unsatisfactory -- That would be one reason
for asking the question in the first place, but it might also
have been asked purely out of curiosity.
You are correct.
OTOH, this is a discussion group, not a help desk; there is a good chance that
my interpretation of the OP's question has at least a smidgen of relevance,
and the algorithms and structures I discussed are valid. Many readers of the
newsgroup might find the concepts interesting and applicable. If the OP
doesn't get full benefit and someone else does, it's still a win.
My discussion was high level. I addressed matters common to many scenarios in
terms that left the details dependent on the OP's details. I'll make a
gentleman's bet that the producer-consumer architecture or one of the cognates
will apply to the OP's situation, but I won't be heartbroken if they don't.
I have found the resource-manager perspective invaluable over the years for a
panoply of situations.
Backing up one meta-level, thinking about a problem domain broadly - pondering
what patterns may apply, drawing analogies like the miner's cart, grokking the
semantics of the target enterprise - is a necessary and productive initial
stage of software development. Often one finds that one has misapprehended
the requirements, just as I may have done in my response to the OP. Broad
thinking catches such misapprehensions early, and is nimble to adapt to new
understanding.
All else is mere hypothecation, until and unless Loial
chooses to amplify his description.
--
Lew