Re: High throughput disk write: CreateFile/WriteFile?

From:

"Tom Serface" <tom.nospam@camaswood.com>

Newsgroups:

microsoft.public.vc.language

Date:

Thu, 28 Jun 2007 15:25:13 -0700

Message-ID:

<uz4$8MduHHA.1204@TK2MSFTNGP03.phx.gbl>

I did a project recently where I had to read and write files pretty quickly.
I used the following code which seemed to work the fastest for me.
Hopefully this makes sense just from the code... I'm just copying bytes so
I used the char on purpose. The program itself is compiled for Unicode, but
I specifically did not want a TCHAR here.

Tom

// Copy a file to the destination either creating a new one or adding
// to the one that is already there
#define BUF_SIZE (8192*2)
char buf[BUF_SIZE];
CRITICAL_ERROR CMyAppDlg::CopyFile(CString &csDestination, CString
&csSource, bool bTruncate)
{
// DWORD err;
HANDLE hFileInput;
HANDLE hFileOutput;

DWORD64 nBytesCompleted = 0;

if(m_bCancelCopy)
  return CRITICAL_ABORT;

hFileOutput = CreateFile(csDestination, GENERIC_WRITE | GENERIC_READ, 0,
NULL,
     bTruncate?CREATE_ALWAYS:OPEN_EXISTING,
     FILE_FLAG_RANDOM_ACCESS, NULL);

if(hFileOutput == INVALID_HANDLE_VALUE)
  return CRITICAL_ABORT;

hFileInput = CreateFile(csSource, GENERIC_READ, 0, NULL,
                   OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);

if(hFileInput == INVALID_HANDLE_VALUE) {
  CloseHandle(hFileOutput);
  return CRITICAL_ABORT;
}

if(!bTruncate) {
  LARGE_INTEGER li;
  li.QuadPart = 0;
  SetFilePointerEx(hFileOutput, li, &li, FILE_END);
// err = GetLastError();
  nBytesCompleted = li.QuadPart;
}

DWORD nBytesRead, nBytesWritten;
int nTimes = 20;
SetLastError(0);
while(ReadFile(hFileInput,buf,BUF_SIZE,&nBytesRead,NULL) && nBytesRead > 0)
{
  if(WriteFile(hFileOutput,buf,nBytesRead,&nBytesWritten,NULL)) {
   nBytesCompleted += nBytesRead;
   UpdateTotalProgress(m_nTotalBytesCompleted + nBytesCompleted,
m_nTotalBytesToCopy);
   if(m_bCancelCopy) {
    CloseHandle(hFileOutput);
    CloseHandle(hFileInput);
    return CRITICAL_ABORT;
   }
   if(--nTimes <= 0) {
    GiveTime(); // Allow UI to update every so often
    nTimes = 20;
   }
  }
  else { // Write failed
   DWORD nError = GetLastError();
   if(nError != ERROR_SUCCESS) {
    CloseHandle(hFileOutput);
    CloseHandle(hFileInput);
    return DisplayCriticalError(nError);
   }
  }
  SetLastError(0);
}

DWORD nError = GetLastError();
if(nError != ERROR_SUCCESS) {
  CloseHandle(hFileOutput);
  CloseHandle(hFileInput);
  return DisplayCriticalError(nError);
}
CloseHandle(hFileOutput);
CloseHandle(hFileInput);
return CRITICAL_NONE;
}

"Brandon" <killerhertz@gmail.com> wrote in message
news:1183068885.527618.274000@q75g2000hsh.googlegroups.com...

Hello all,

I have an application where I need to write text and binary (numeric)
data to a file at ~200 MB/s (8MB/40ms). I am currently using fprintf
and fwrite, respectively, but I'm not achieving the desired
throughput. I've been looking into the Windows CreateFile based
methods instead.

I am using a pcix hardware raid adapter with RAID0 across 4 drives. I
have benchmarked my system using h2benchw v3.6 with good results. For
my desired file sizes, anywhere from 8MB to 64MB (I'm flexible on how
many 8MB data sets are stored in a single file), I am able to achieve
up to 275MB/s.

I'm hoping it's possible to get close to that using alternatives from
the standard C options. Any suggestions?

For now I'm trying CreateFile/WriteFile, without success. My code
looks like this:

<SNIP>
   HANDLE hWriteFile = NULL; // File handle
   LPDWORD lpNumBytesWritten = NULL; // Number of bytes written
(WriteFile)
   char szTextToAppendToLog[1024]; // Temp char buffer
   char szFilePath[1024]; // Output file name
   char szTimeStamp[32]; // Time character string
   char szTemp[128]; // Temp character string

               // Open the output file for write.
               hWriteFile = CreateFile(
                   szFilePath, // File path
                   GENERIC_WRITE, // Open for write
                   NULL, // Do not share
                   NULL, // Default security
                   CREATE_ALWAYS, // Overwrite existing files
                   FILE_FLAG_WRITE_THROUGH,//FILE_FLAG_OVERLAPPED, //
Normal file
                   NULL); // No template
               if (hWriteFile == INVALID_HANDLE_VALUE)
               {

sprintf_s(szTextToAppendToLog,sizeof(szTextToAppendToLog),
                       "ERROR: Output file %s failed to open.",
                       szFilePath);
                   pThis->UI->AppendToStatLog(szTextToAppendToLog);
               }

...

               // Write ASCII header to file.
               sprintf_s(szTemp, sizeof(szTemp), "MyData: Date:%s\r
\n",szTimeStamp);
               WriteFile(
                   hWriteFile,
                   szTemp,
                   (DWORD) sizeof(szTemp),
                   lpNumBytesWritten,
                   NULL);
</SNIP>

As soon as I hit WriteFile(...) my application hits an unhandled
exception:
"Unhandled exception at 0x7c810e0c in DataCapture.exe: 0xC0000005:
Access violation writing location 0x00000000."

I'm guessing I'm not using CreateFile correctly?

Thanks,
-Brandon