[fitsbits] Rice compression from the command line

[Rob Seaman]

On Jul 12, 2006, at 9:32 PM, Mark Calabretta wrote:

> I now mostly use bzip2 (GPL, bzip.org), it usually outperforms gzip by
> about 10% for binary data.

It's quite a bit slower, however.

> I have also recently started to use 7zip, an archiving compressor that
> mostly seems to outperform bzip2.  The unix GPL distro (p7zip.org)  
> also
> supports gzip and bzip2 and various archive formats.  [...]  wikipedia
> has excellent articles on all of these.  It also mentions PPM which
> is supposed to be the best, at least for text.

One needs to distinguish between the compression algorithm and the  
FITS tile-compression convention.  The latter should allow output  
from most any algorithm to be embedded in a FITS file complete with  
readable headers and transparent access to the pixels.  A Rice  
compressed image expressed as a bintable looks like the original  
image array to cfitsio, for instance.

In the absence of further suggestions for tile-compression options  
beyond the cfitsio example imcopy program, I've decided to proceed to  
roll my own.  Bill Pence is making corresponding improvements to  
cfitsio to address some of the issues mentioned in my prior message.

I've appended the envisioned command line help for a new cfitsio  
layered program called "fpack".  There will also be "funpack" which  
will share the capital letter arguments below.  Fpack will be able to  
compress a list of FITS files in place, or to create a single large  
output compressed MEF.  Funpack will be able to uncompress in place,  
or to restore compressed HDUs to the original list of individual files.

I see these commands as combining features from the FITS tile- 
compression, checksum and foreign encapsulation conventions, and  
benefiting from various cfitsio functionality such as HDU  
validation.  The final product will likely differ in (hopefully)  
small ways from the concept below, but the idea should remain to  
provide rich functionality in a simple package (e.g., this command  
won't descend directory trees - if you need that, use FITS  
encapsulation first).

It has also come to the notice of its authors that the foreign file  
encapsulation proposal is not widely known.  Somehow it was never  
published after it appeared as an ADASS IX poster (same year as tile- 
compression).  We intend to remedy this via the new registry of FITS  
conventions after having a chance to review our latest (very mature)  
draft.  Conforming writer and reader have been available since 1999  
as host callable tasks in the IRAF fitsutil external package.

Comments about the fpack concept, or FITS compression in general,  
would be quite welcome.  Folks intending to add tile-compression  
support to their own FITS libraries, image processing environments,  
or web services would be deliriously so.

Rob Seaman
NOAO
------

fpack, a FITS tile-compression engine.  Version 0.0.2 (18 July 2006)

usage: fpack [-r|-g|-p] [-w|-t <axes>] [-n <bits>] [-FLAGS] <FITS>

   -r         Rice compression [default], or
   -p         PLIO compression, or
   -g         GZIP (per-tile) compression

   -n <bits>  noise bits to preserve for real pixels [default=4]
   -t <axes>  CSV list of tile sizes [default=one tile/image row], or
   -w         override tile size to be whole image

   -F         clobber output file(s) (overwrites input in-place by  
default)
   -K         keep (don't delete or overwrite) input files
   -A <file>  append all output to single <file>, create if needed, or
   -P <pre>   prepend <pre> to create separate output filenames

   -U|-u      unpack files, instead of pack (same as funpack)
   -Z         when unpacking, do not restore original names/times

   -L         list-only file contents
   -Q         silent operation
   -H         print this message
   -V         print version number

   <FITS>     FITS files to (un)pack

All flags must appear (separately) before all filenames.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20060718/2b1792c1/attachment.html>