Re: Reading from very large file
On 05/09/2010 02:57 AM,
Lew wrote:
<http://sscce.org/>
<http://sscce.org/>
<http://sscce.org/>
Hakan wrote:
Sorry about the mistake, but the file is actually 13 GB. I can read to a
Well, now, that's a horse of a different color.
character array buffering about 30 million characters before the heap
Allocate a larger -Xmx.
space is overflowed. This is still only a part of the file.
What about buffering?
The sscce site is down and not accessible when I tried. What I have been
It's up now as I check it. Look again.
doing so far is something like this in rough code:
static int nchars=27000000;
int startpos=0;
File readfile="../x.txt";
Relative paths from inside Java code can be tricky.
Class.getResourceAsStream() (around which you'd wrap a Reader, of course) can
help with that.
FileReader frd=new File;
This line will not compile.
Show ACTUAL code. How many people have told you this in this thread?
You keep showing disrespect for the people trying to help you, and that could
hurt your chances of getting good help.
String searchs="20020701";
char[] arr=new char[nchars];
while (more dates to search for)
{
frd=new FileReader(readfile); /*reopen file
frd.skip(startpos); /*move to file pointer where final place of last
date was found
frd.read(arr,0,nchars); /*10
find number of date occurrences in arr with pattern matching
update searchs (first time to "20020702" and so on
startpos=startpos+(last place of pattern match)
output result for this date
}
None of this will compile. What is this garbage?
Have you ever heard of indentation?
Use up to four spaces indent per logical level for Usenet posts.
This in all tends to use one to two minutes per run of the loop. What I
This all tends not to run at all, since it won't compile in the first place.
would like to do is to a) either preprocess the file such that I get an
input file where only numbers are present or b) change the read call at
label 10 so that it only reads numbers instead of all next characters.
Thank you so much for your help!!
Once again, since you seem to have ignored or missed it the first couple of
times people have told you, read in the file sequentially with a relatively
large buffer (say, 8 MiB? 16?)
Why do you disregard substantial portions of the advice several people have
given you, then come back with the exact same question again and again?
--
Lew