Re: CStdioFile... with a twist (please)

From:

Alexander <the44secs@yahoo.com>

Newsgroups:

microsoft.public.vc.mfc

Date:

Mon, 4 Aug 2008 19:01:04 -0700 (PDT)

Message-ID:

<fa6f1915-7009-4008-a5b0-8d0dbe1ab614@i24g2000prf.googlegroups.com>

The benchmark results are positive. Your approach matches the speed of
CStdioFile and since I have to read the file several times, loading
the contents only once actually makes this approach faster.

Memory mapped files look interesting. I'll take a look around and see
what I can... understand.

Speaking of (lack of) understanding, I can't get CStringA to
compile... gosh darnit, I'm dumb! What's the include? I'm using VC6...

On Aug 5, 5:16 am, Joseph M. Newcomer <newco...@flounder.com> wrote:

Another advantage of the file mapping is that it keeps your entire workin=

g set size down.

joe

On Mon, 4 Aug 2008 10:40:22 -0700 (PDT), Alexander <the44s...@yahoo.com> =

wrote:

Thank you very very much, Joe. I now have all I need and then some.

On Aug 5, 12:37 am, Joseph M. Newcomer <newco...@flounder.com> wrote:

See below...

On Mon, 4 Aug 2008 07:12:40 -0700 (PDT), Alexander <the44s...@yahoo.co=

m> wrote:

Thank you for taking the time to post an algorithm, Joe.

Your observations have, of course, hit on the issues that I've come
across. CString operations are the biggest time suckers for sure (the
file is ASCII but the code must be UNICODE).

****
Note I used a CStringA. If, at some point, you need to convert, you=

can do

CStringA t;
CString s(t);
and s will be the Unicode version of the ANSI characters in t. I use t=

his trick fairly

often when I have to deal with 8-bit data streams in Unicode apps. =

But if you can

postpone the conversion so the conversion is folded in with the creati=

on, it will work

better; for example
CString s(p, n);
where p is an LPSTR/LPCSTR (not LPTSTR/LPCTSTR) and n is the number of=

characters to

convert will give you a Unicode string with just one copy operation (a=

nd a

MultiByteToWideChar conversion)
****>I wonder, though, about

loading the entire file at once... somehow I had convinced myself tha=

it would slow things down.

****
Why? You have to read all the file eventually...
****

The test file is small, approx. 4 MB (the "real" one 60~90 MB). I'll
add this code to the benchmark and see how it fares.

****
Memory-mapped files may also give better performance than a large Read=

File; they're harder

to use but you don't actually bring the data in until it is touched. =

If you window data,

it is a bit tricky because the windows have to fall on 64K boundaries,=

so you have to

"back up" to the previous 64K boundary if you hit an edge, e.g., if ea=

ch of the letters

below is a 64K block
ABCDEFG
and you map AB into memory, when you get to the end of B you have only=

a partial line, so

you would then have to map
BC
and then
CD
and so on so you could see the entire string (assuming it is <64K; oth=

erwise you would map

more pages to cover the maximum string length). Note that you don't=

have to map the

minimum number of pages; you could map 100 pages or 300 pages or whate=

ver you want, but

when you hit a boundary, you have to include the 64K block in which th=

e record started in

the next mapping.
joe
****

On Aug 4, 10:15 pm, Joseph M. Newcomer <newco...@flounder.com> wrot=

How large are your files?

CFile f;
if(!f.Open(....))
    deal with error

ULONGLONG size = f.GetLength();
ASSERT(size <= 0x0FFFFFFFull);
// Arbitrary choice of maximum length; you are probably in trouble
// if it is > 100MB or so, and the code below won't work, so you
// might choose a smaller length, e.g.
// ASSERT(size < 100000000ull);

CStringA b;
LPSTR p = b.GetBuffer(size + 1);

f,Read(p, size);
p[size] = '\0';

b,ReleaseBufferSetLength((int)size);

// Using ReleaseBufferSetLength means it doesn't have to search
// for the terminating NUL character to determine the length

int start = 0;
while(true)
   {
    int n = b.Find('X', start);
    if(n < 0)
        { /* not found */
         everything from start to end of string is the re=

cord of interest

         break;
        } /* not found */
    everything from start to n-1 is the record of interest
    start = n + 1;
   }

This is rather simplistic, but if you don't try to create intermedi=

ate CStrings it can be

very fast. It works well only for small files (say, < 100MB). =

For larger files, you

would apply the technique above to a memory-mapped file (there is s=

ome trickiness and you

can't use CStringA in this case, you have to go a bit lower-level b=

ecause the strings will

not necessarily be NULL-terminated at the endpoint, and you have to=

deal with windowing

the mapping view into the larger file, but I'll assume your files a=

re of moderate size and

therefore this more complex solution is not needed)
=

joe

On Mon, 4 Aug 2008 00:00:30 -0700 (PDT), Alexander <the44s...@yahoo=

..com> wrote:

Ok. I've written a couple of implementations (derived from CStdioF=

ile

and streams). They work fine but are much slower than CStdioFile w=

hich

is slow to being with. I need something fast for this.

Any ideas?

On Aug 4, 2:05 pm, "Check Abdoul" <check abdoul at mvps dot org>
wrote:

Derive a subclass from CStdioFile and overwrite ReadStri=

ng() function

and change its implementation[ ReadString() is virtual ]

Cheers
Check Abdoul
---------------------

"Alexander" <the44s...@yahoo.com> wrote in message

news:94225650-b700-490b-a1c8-c62a71f52700@a6g2000prm.googlegroups=

..com...

I need a class exactly like CStdioFile but that on ReadString f=

etches

up to a character other than EOL.

Does such a thing exists? Thank you all.

Joseph M. Newcomer [MVP]
email: newco...@flounder.com
Web:http://www.flounder.com
MVP Tips:http://www.flounder.com/mvp_tips.htm