Re: Smartest way of compressing numbers over a webservice?

Casper wrote:


     The loose description amounted to encoding as a twelve-bit
quantity (not sure why; ten bits would suffice), packed into a
byte array in some unspecified way (two bytes per value? Less?)
and then compressed with GZIP (compression ratio unknown). I
don't know how you get from that loose description to the fairly
specific value of 90 kilobits ~= 11.3 kilobytes, and I don't see
how you can characterize the reduction from 500KB to 11.3KB as
"70-80 times" better: it seems like maybe one forty-fourth. (Or
about two elevenths, if "90Kb" actually means "90KB.")


     GZIP will almost certainly do better. Huffman encoding is
only an encoding, and does not in itself embody a probability
model, a.k.a. a "predictor" of the future stream. If you couple
it with a static model of symbol probabilities it will usually
"predict away" less of the entropy than will an adaptive method.
Simple thought experiment: Consider a data stream of one million
zero bits followed by one million ones. Each value occurs with
probability 50% so static-probability Huffman encoding will assign
one encoded bit per original bit. An adaptive method like GZIP
will "learn" the pattern of the one million zeroes and encode them
very compactly, then will be "surprised" at the transition and
lose efficiency for a while before it syncs up to the new reality.
Once it does, you'll find it once again encoding hundreds or even
thousands of original bits per compressed bit.

     There are newsgroups devoted to compression topics where you
might find more information. It's been years since I followed
them, but I recall their FAQs as being highly informative.

--
Eric Sosman
esosman@acm-dot-org.invalid