Re: read huge text file from end

From:

Eric Sosman <esosman@acm-dot-org.invalid>

Newsgroups:

comp.lang.java.programmer

Date:

Wed, 01 Nov 2006 07:46:04 -0500

Message-ID:

<E6SdnVBCdreSCtXYnZ2dnUVZ_tydnZ2d@comcast.com>

Mike Schilling wrote:

"Eric Sosman" <Eric.Sosman@sun.com> wrote in message
news:1162335115.956942@news1nwk...

Hopefully, the compression would be handled by the underlying OS, and
it
would all work "transparently" to your application.

It might "work" in the sense of "get to the data as
desired," but only by reading and decompressing everything
before that point -- which sort of vitiates the performance
advantage of the seek, don't you think?

But that's not how OS file compression works. Generally, there's a page
size (8K or thereabouts), and each page is compressed seperately, with the
OS keeping track of where each compressed page actually starts. A
random-access read requires figuring out where the pages containing the byte
range live and decompressing only those pages.

Look among the bits and pieces of snippage lying about on the
cutting-room floor, and you'll notice I wrote about files that
were "progressively compressed" or "progressively encrypted."
My terminology is probably inexact, but I meant "progressivly"
to describe the sort of compressor/encryptor whose state at a
given point in the data stream is a function of the entire history
of the stream up to that point. gzip, for example.

--
Eric Sosman
esosman@acm-dot-org.invalid