[fitsbits] start of Public Comment Period on compressed FITS image and tables

[William Pence]

I only know of a couple projects that are considering using the 
compressed table convention when they eventually start producing massive 
amounts of binary table data.  Otherwise I don't know of any projects 
that are currently using the compressed table convention.

Many people are probably unaware of this convention (documenting it in 
the FITS standard would certainly raise its visibility).  There was a 
brief email on FITSBITS in November 2013 when it was added to the 
registry for public review, but no one commented on it.

It should also be noted that, unlike tile compressed images, there is no 
existing software (including CFITSIO) that can read the contents of a 
tile compressed table on the fly. The compressed table is a perfectly 
valid FITS file, but the contents of the table are pretty opaque until 
it is uncompressed.  One can compress any binary table using fpack, but 
then one must uncompress it with funpack before any existing software 
can make any sense of the data that is stored in the table. So the tile 
compressed table convention is mainly useful for archiving large set of 
tables.  It typically produces much more compression than one gets by 
simply gzipping the binary table.

-Bill

On 6/25/2015 9:50 AM, Erik Bray wrote:
> On 06/25/2015 04:37 AM, Mark Taylor wrote:
>> On Wed, 24 Jun 2015, Mandel, Eric wrote:
>>
>>>>
>>>> How interesting.  I'm not sure if your experience argues for or
>>>> against the incorporation of tile compression in the standard.
>>>
>>>
>>> For me as a developer, tiled compression already is part of the
>>> standard:
>>> the very first email we got after announcing release 1 of JS9 (October
>>> 2014) was a complaint that it did not  handle rice-compressed tiled
>>> images.  I eventually came to the conclusion that cfitsio *was* the
>>> practical standard, at least for anyone trying to write software for the
>>> community at large. I know Bill Joye has spent a lot of time
>>> implementing
>>> tiled compression in DS9 (which does not use cfitsio), just because of
>>> widespread demand.
>>
>> I don't deal much with images, but interestingly, I have never
>> had complaints or requests concerning use of tile-compressed FITS
>> tables in STIL/STILTS/TOPCAT.  That suggests that tile compression
>> for table data is much less used than for image data.  Can anybody
>> comment on whether that's the case?
>
> I have never encountered compressed tables in the wild.  If I had to
> guess, this is in part because the documentation for the convention was
> not easy to come by until recently.  It was not available on the
> registered FITS conventions website, or at least, I never noticed it.
> Nobody has every asked for support in PyFITS or Astropy.
>
> Tile-compressed images, however, I would consider an essential part of
> the standard regardless whether it's classified as a "mere convention".
> Not supporting it renders a significant number of FITS files effectively
> unreadable.  (I'd have no problem with supporting compressed tables
> either if there was significant demand for it, but I haven't seen that...)
>
> Erik
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits


-- 
____________________________________________________________________
Dr. William Pence    Astrophysicist     William.Pence at nasa.gov
NASA/GSFC Code 662     [Emeritus]       +1-301-286-4599 (voice)
Greenbelt MD 20771                      +1-301-286-1684 (fax)