Re: Any suggestions for handling data of huge dimension in Java?
On 24/03/11 10:09, Simon wrote:
Dear All,
Good day. Regarding the subject, I am doing a research simulation by
using java in eclipse galileo. My laptop is dell studio xps 1645 with
i-7 processor and 4gb ram. When running the java source codes, I have
set the Run Configurations > Arguments > VM Arguments > -Xmx1024M -
XX:MaxPermSize=128M and also assign the object to null when it is no
longer needed. However, i keep facing java heap problem. Most of the
time, i am using HashMap<String,Double> and StringBuilder to hold the
data. Dimension of my data is around 5000 columns (or features) x 50
classes x 1000 files, and I need to extract that data into one file
for classification purpose. Therefore, is there any suggestions or
articles for me to cope with such problem?
What is the problem? You haven't actually stated you have a problem,
only a task to perform. What problem are you actually seeing?
If you are running out of memory (OOM errors) then determine whether
it's the heap or PermGen (do you have any evidence that PermGen needs
increasing?). to determine whether you are keeping objects beyond their
sell-by date use a profiler. There's one in NetBeans which will do the
basics, but I don't know Eclipse that well. It's all too easy to not
release every reference to an object, thus not allowing it to be GC'd,
even when you think you have.
If you are not hanging on to objects unnecessarily you can try to
increase the heap size to the max. allowed by your system memory limits.
If you still cannot fit your data into the system then you need better
hardware/OS (a 64bit OS would allow you to use more per-process VM), or
a different algorithm (one which doesn't try to hold all your data in
memory at the same time).
--
Nigel Wade