Re: Searching a disk-backed Map
On Tue, 18 Aug 2009, Patricia Shanahan wrote:
Stefan Ram wrote:
This should be a common need. Yet I am not aware of anything
like it in Java SE. What is the most common (pure Java)
solution to it?
I would like to have an implementation of java.util.Map,
which is constructed with an int ?m? and a java.io.File ?f?.
It will use no more than ?m? bytes of memory, but ?swap? out
(the least often used) entries to the file ?f?, when they do
not fit into the given memory size anymore.
Have you considered putting the data in a database instead, and using
java.sql to access it? The data structures and algorithms that Java uses
for in-memory maps are not very suitable for disk-based maps. Database
managers use structures and algorithms designed for the job.
'The job' in question being relational data access. Stefan doesn't want
that, he wants to do stores and lookups by key, and nothing else (well,
that and removals, and iteration - but i would imagine the priority is
fast storage and lookup). Yes, this is a subset of what you can do with a
relational data store, but it's quite possible that an implementation
which does keyed storage and nothing else will do it faster and more
efficiently.
The prototype of this is the ancient unix DBM:
http://en.wikipedia.org/wiki/Dbm
And its zillions of successors (which include java wrappers and pure java
implementations - although i don't know if any have a Map interface).
These are (generally held to be) quite a bit faster than an RDBMS.
It's worth stressing, for anyone who hasn't fully grokked this, that
RDBMSs are *not* fast. Not in the slightest. This is why none of the huge
web players - Google, Yahoo, Facebook, etc - use them in their
high-throughput apps. The reason RDBMSs are popular nonetheless is because
they're fast enough for most purposes, they're reliable and trustworthy,
and they're highly, exceptionally, *insanely* flexible.
tom
--
The square-jawed homunculi of Tommy Hilfiger ads make every day an
existential holocaust. -- Scary Go Round