Re: open a large file in win64
Mycroft Holmes wrote:
Try a multithreaded design, one
thread calling fread on a chunk of memory and then passing it to the
other thread. While the other thread is creating the digest, the first
thread already fills a second buffer.
The advantage is that you ideally always have one thread maxing out the
CPU with the digest (assuming source data is available) and a second
thread maxing out the IO subsystem (i.e. the harddisk). If you
sequentially read and digest, you are alternating between waiting for
IO and waiting for CPU.
Hmm... the crc algorithm is irrelevant with respect to i/o activity: I can
even comment it out and the performance does not change.
Okay, that means that my assumption was false. I was simply assuming that
you had the two states CPU busy/IO idle and CPU idle/IO busy alternatingly.
Using two threads you could have maintained CPU busy/IO busy up to the
level where one of them is saturated.
assuming I have two equal buffers, the digest-computing thread would take
about 0 and the buffer-filler thread would take LARGE_T...
I'm pretty sure multithreading in this case would not improve a bit.
My implementation "waits" for i/o only once per gigabyte.
That is not the point, the point is how long it waits and what it could do
while it is waiting!
(reading in
above context could be simply touching the memory-mapped range)
I'm not sure I understand. could you explain your statement?
Memory-mapping does not per-se trigger any IO. Only when you touch the
mapped page a page fault is triggered and an exception handler in the OS is
triggered. The OS then looks up the page mapping, sees that it is mapped to
a file and reads the file, i.e. the actual IO is only done on demand. In
order to have one thread to keep the IO system busy, it is not enough if
that thread only maps the file but it must actively touch the pages
(trigger the page fault that causes the IO) for the precaching to succeed.
Uli