[fitsbits] Potential new compression method for FITS tables

[Rob Seaman]

One additional point is that many vendor mass storage technologies can be configured with general purpose compression "built-in".  Such a system can't take advantage of algorithms tailored for the particular data in question.

Rob
--

On Dec 17, 2010, at 9:27 AM, William Thompson wrote:

> Preben Grosbol wrote:
> 
>> As a side comments, I participated in a meeting for general archives a few
>> years ago.  There they did not recommend to save compress data since
>> the effect of single bit errors is more serious than for raw data, not
>> justifying the gain of disk space.
> 
> Data compression addresses three issues:
> 
> 1.  The cost of archiving the data.
> 2.  The rate at which data can be delivered to users
> 3.  The ability of users to locally store data for scientific research
> 
> I mentioned before that the Solar Dynamics Observatory (SDO) mission is using 
> the Tiled Image Compression convention to help them manage their ~2 TB/day of 
> data volume.  This is actually to address points 2 and 3 above than the issue of 
> archiving the data.  The data are not archived in FITS format.  Instead, the 
> data are stored in a database management system, with the images and associated 
> metadata stored separately.  The FITS files are generated on-the-fly as 
> requested.  This is how the same data can be delivered in both uncompressed and 
> Rice compressed formats.
> 
> I'm not sure how the images are stored within the database management system, 
> whether they're compressed or uncompressed, but I suspect that they're 
> compressed.  I do know that the raw data from the telescopes undergo compression 
> on board the spacecraft before being telemetered down.  At least for the AIA 
> telescope, this on-board compression is lossy--I'm not sure about HMI.
> 
> Bill Thompson