[fitsbits] Potential new compression method for FITS tables
seaman at noao.edu
Fri Dec 17 12:11:58 EST 2010
One additional point is that many vendor mass storage technologies can be configured with general purpose compression "built-in". Such a system can't take advantage of algorithms tailored for the particular data in question.
On Dec 17, 2010, at 9:27 AM, William Thompson wrote:
> Preben Grosbol wrote:
>> As a side comments, I participated in a meeting for general archives a few
>> years ago. There they did not recommend to save compress data since
>> the effect of single bit errors is more serious than for raw data, not
>> justifying the gain of disk space.
> Data compression addresses three issues:
> 1. The cost of archiving the data.
> 2. The rate at which data can be delivered to users
> 3. The ability of users to locally store data for scientific research
> I mentioned before that the Solar Dynamics Observatory (SDO) mission is using
> the Tiled Image Compression convention to help them manage their ~2 TB/day of
> data volume. This is actually to address points 2 and 3 above than the issue of
> archiving the data. The data are not archived in FITS format. Instead, the
> data are stored in a database management system, with the images and associated
> metadata stored separately. The FITS files are generated on-the-fly as
> requested. This is how the same data can be delivered in both uncompressed and
> Rice compressed formats.
> I'm not sure how the images are stored within the database management system,
> whether they're compressed or uncompressed, but I suspect that they're
> compressed. I do know that the raw data from the telescopes undergo compression
> on board the spacecraft before being telemetered down. At least for the AIA
> telescope, this on-board compression is lossy--I'm not sure about HMI.
> Bill Thompson
More information about the fitsbits