Re: writing multiple files vs single file using fwrite/fopen
Hi Rahul,
It's tough to say for sure why there would be a lot of difference, but I
know from experience that opening and closing files is a time consuming
operation. In my application I have to open files to do things like set the
time/date on the file (SetFileTime() needs a handle) and it takes quite a
bit of time. Have you considered trying CreateFile instead. It has lots of
different options for opening that you might be able to play around with to
improve your results.
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx
Tom
"Rahul" <rahulsharma@lucent.com> wrote in message
news:1323c118-2f6f-4051-8bb9-6835fdf612b6@a9g2000prl.googlegroups.com...
On May 14, 5:30 pm, Joseph M. Newcomer <newco...@flounder.com> wrote:
So your two scenarios could be expressed as
for(int i = 0; i < 400; i++)
{
CString s;
s.Format(_T("File%d.txt"), i);
FILE * f = _ftopen(s, _T("w"));
...write data
fclose(f);
}
and
for(int i = 0; i < 400; i++)
{
FILE * f = _ftopen(_T("File.txt"));
... write data
fclose(f);
}
It looks like 80 seconds is going into the fopen, that computes to
200ms/file, seems
within reason. It probably has to do with the fact that the directory
block has been
cached and it can update it in the second case directly in memory, without
having to go
out to the disk to create a new block.or deal with committing the
directory block back to
the disk.
Sounds reasonable. What's the problem in understanding why it takes
longer? Accessing
the directory for 400 unique files means that you are consuming a lot of
directory blocks
to handle this, and I/O to a disk takes time. 200ms seems a bit high, but
not
unreasonable, unless your files are out on a server, in which case 200ms
seems quite
reasonable.
joe
On Tue, 13 May 2008 23:12:40 -0700 (PDT), Rahul <rahulsha...@lucent.com>
wrote:
Hi,
I have a VC++(2005) application (2 "Generator" threads creating a data
buffer, and a "Writer" thread to write it).
The application creates and writes about 400 files (containt ~20MB
data each)
The Generator threads creates next data file while the Writer is
writing the previous (Generator waits until writer completes and
starts next file only when the current is handed over to Writer)
The writer thread does
My Problem is as follows
When the Writer thread creates and writer date into 400 different
files then the end to end jobtime is ~ 280 seconds (cpu usage time
remains same with teh below case)
Instead of creating 400 different files if I overwrite in a single
file always then the end to end jobtime is ~ 200 seconds.
So the slowdows is happening because of creating and writing into 400
different files.
I am using fopen/fwrite.
I want to know why is writing into multiple files slow.
There were no other apps running at the time of test (I killed all
network drivers also)
Joseph M. Newcomer [MVP]
email: newco...@flounder.com
Web:http://www.flounder.com
MVP Tips:http://www.flounder.com/mvp_tips.htm- Hide quoted text -
- Show quoted text -
Hi,
My application is creating the fils on a local drive (there is only
one partition on the disk C:) So 200ms/file seems unrealistic to me.
Moreover the application behaves differently on different systems. on
one computer I saw the fopen time (total for all 400 files) varying
from 20 seconds to 100 seconds, while on other the fopen time remained
same but the fwrite time (total for all files) changed from 80 to 140
seconds.
I know there can be variations, but 60-80 seconds variation seems very
un realistic to me when the system virtaully free (only few process
other then my app are doing I/O at the time of testing and their
amount is quite less, ~ kb per minute)