Re: advice on loading and searching large map in memory
On 02/20/2011 08:00 AM, Martin Gregorie wrote:
On Sat, 19 Feb 2011 17:43:36 -0800, eunever32@yahoo.co.uk wrote:
Hi
We have a requirement to query across two disparate systems. Both
systems are read-only so no need for updates and once loaded and no need
to check for updates. I would plan to reload the data afresh each day.
Records on both systems map one-one and each has 7million records.
The first system is legacy and I am reluctant to redevelop (C code). The
second is standard Java/tomcat/SQL
The non-relational query can return up to 1000 records.
This could therefore result in 1000 queries to the relational system
(just one table) before returning to the user.
To avoid 1000 relational queries I was planning to "cache" the entire
relational table in memory. I was planning to have a web service which
would load the entire relational table into memory. The web service,
running in a separate tomcat could then be queried 1000 times or maybe
get a single request with 1000 values and return all results in one go.
Having a separate tomcat process would help to isolate any memory issues
eg JVM heap size.
Can people recommend an approach?
How big are the items in each collection?
How does the search process recognise an item?
If there are specific key terms, how big are they and how many terms are
there per item?
The answers to these questions can have a large effect on selecting the
optimum approach.
No one is asking the all-important questions!
What is the current performance? Is the set of queries a bottleneck? How do
they know? Under what load conditions? Couldn't they just submit a single
query with the 1000 values instead of 1000 individual queries? (Yes, they
could.)
Asking to optimize something with a technique that likely will have severe
performance problems itself, as the OP did, is a likely sign of sketchy
analysis, at best. Let's at least get the evidence that there's a problem to
solve here that so far has been glaringly absent.
So, OP, tell us - how did you determine that there's a bottleneck in the area
you suspect, and how did you reject all the other approaches to its resolution
but the one you ask about?
--
Lew
Honi soit qui mal y pense.