Re: open a large file in win64
Mycroft Holmes wrote:
[computing digest from large file]
My experiments seem to leadto the opposite conclusion:
I tried fread'ing chunks of 64K and chunks of 16MB, it takes more than 400
seconds anyway... so I'm still confused.
Mapviewoffile seems the fastest way to read, but the utility I downloaded
is much faster...
Hehe. ;)
I guess your design simply doesn't scale. Try a multithreaded design, one
thread calling fread on a chunk of memory and then passing it to the other
thread. While the other thread is creating the digest, the first thread
already fills a second buffer.
The advantage is that you ideally always have one thread maxing out the CPU
with the digest (assuming source data is available) and a second thread
maxing out the IO subsystem (i.e. the harddisk). If you sequentially read
and digest, you are alternating between waiting for IO and waiting for CPU.
Make sure you are not unnecessarily copying the data around (reading in
above context could be simply touching the memory-mapped range) and try to
keep the working set of data small so you can exploit caching.
Uli