Re: Smartest way of compressing numbers over a webservice?

From:

"Oliver Wong" <owong@castortech.com>

Newsgroups:

comp.lang.java.help

Date:

Thu, 12 Apr 2007 13:51:56 -0400

Message-ID:

<1luTh.12515$ef6.26267@wagner.videotron.net>

"Casper" <casper@jbr.dk> wrote in message
news:5EtTh.7574$O_6.29560@weber.videotron.net...

I have to transfer a large amount of numbers over a webservice
(document-literal). Using standard JAX-RPC1.6 double XML serialization
causes major overhead that is just too slow for the purpose. So I was
wondering how I could optimize this. Here's what I know:

- Potentially >100.000 numbers
- Each number has a fixed span (I.e. can only be between -50.0 and 50.0)

My initial though is to take advantage of the fact that each number has
a span of 1001 values. This fits nicely within 12bit. The list of
numbers (each 12 bit) could then be in a byte array which is thrown
though a GZIPOutputStream only to be attached binary (or Base64 encoded
and included as an XML string element for simplicity).

Does this sound like a good idea or could I somehow improve the
throughput? Particulary I am wondering about compression schemes that
would work well on numbers rather than the very generic LZ* algorithm of
GZIP.

Ask in comp.compression for more details, but almost all compression
schemes are based around the idea of taking advantages of expected
patterns in the output. And what patterns you'll expect is entirely domain
specific. For example, in compressing English text, you can almost always
assume that if you see a character 'q', the next character will be 'u',
but you may not be able to make such an assumption for other languages, or
compressing things other than natural-language texts.

- Oliver