[fitsbits] Rice compression from the command line

[Mark Calabretta]

On Wed 2006/07/19 10:58:29 +1000, Mark Calabretta wrote
in a message to: Rob Seaman <seaman at noao.edu>
and copied to: fitsbits at nrao.edu

>should not be compressible.  I measured the 10% figure for the FLC
>animations distributed from my web site.  If the files are intended
>for download, especially very large files, the extra time taken to
>compress/uncompress them is more than compensated by the faster
>download time.

Some measurements for a few different types of files I had lying around
are appended.

For the FITS binary table, 7zip is costly in CPU time for compression
but beats gzip and bzip2 handsomely in compression ratio.  However, 7zip
is not nearly so costly in elapsed time for decompression.  If these
results are typical then 7zip would have to be the compressor of choice
for FITS data distributed on the web.

For the FLC animation, 7zip again wins on compression ratio but the
compression time is now very large.  However, decompression time is
comparable to gzip so it still might be preferable for distributing a
static archive.  Possibly the compression time could be reduced by
twiddling 7zip's compression options.

The ASCII test used a portion of the freedb.org CD database which has a
high degree of repetition and is therefore very compressible.  7zip again
goes off the rails in compression time, but wins on decompression.
However, bzip2 would probably be the choice for distributing this file.

Mark Calabretta
ATNF

>>>

--------------------------------------------
FITS binary table
Uncompressed size: 563 MiB

                compress          uncompress
           U+S/E(s) MiB  (%)       U+S/E(s)
          -------------------     ----------
gzip        85/ 107  376 (67%)      15/ 52
bzip2      268/ 300  366 (65%)     131/152
7zip       623/ 691  277 (49%)      95/114

--------------------------------------------
FLC animation
Uncompressed size: 690 MiB

                compress          uncompress
           U+S/E(s) MiB  (%)       U+S/E(s)
          -------------------     ----------
gzip       263/ 323  246 (36%)      18/ 71
bzip2      289/ 373  217 (31%)     149/159
7zip      2206/2377  186 (27%)      56/ 77

--------------------------------------------
ASCII text (freedb.org "misc" CD database)
Uncompressed size: 1086 MiB

                compress          uncompress
           U+S/E(s)  MiB  (%)       U+S/E(s)
          -------------------     ----------
gzip       103/ 125  231 (21%)      22/ 99
bzip2      391/ 434  155 (14%)     140/152
7zip      1303/1389  162 (15%)      62/ 83

--------------------------------------------
Notes:
* Tests done on a quiet Dell Latitude D600.
* Default compression options used.
* 7zip used with LZMA algorithm.
* U+S is the user + system CPU time (s).
* Typically S/(U+S) < 3%.
* E is the elapsed time (s).
* 1 MiB = 1024^2 bytes.