[fitsbits] Potential new compression method for FITS tables

[William Thompson]

Preben Grosbol wrote:

> As a side comments, I participated in a meeting for general archives a few
> years ago.  There they did not recommend to save compress data since
> the effect of single bit errors is more serious than for raw data, not
> justifying the gain of disk space.

Data compression addresses three issues:

1.  The cost of archiving the data.
2.  The rate at which data can be delivered to users
3.  The ability of users to locally store data for scientific research

I mentioned before that the Solar Dynamics Observatory (SDO) mission is using 
the Tiled Image Compression convention to help them manage their ~2 TB/day of 
data volume.  This is actually to address points 2 and 3 above than the issue of 
archiving the data.  The data are not archived in FITS format.  Instead, the 
data are stored in a database management system, with the images and associated 
metadata stored separately.  The FITS files are generated on-the-fly as 
requested.  This is how the same data can be delivered in both uncompressed and 
Rice compressed formats.

I'm not sure how the images are stored within the database management system, 
whether they're compressed or uncompressed, but I suspect that they're 
compressed.  I do know that the raw data from the telescopes undergo compression 
on board the spacecraft before being telemetered down.  At least for the AIA 
telescope, this on-board compression is lossy--I'm not sure about HMI.

Bill Thompson

-- 
William Thompson
NASA Goddard Space Flight Center
Code 671
Greenbelt, MD  20771
USA

301-286-2040
William.T.Thompson at nasa.gov