[fitsbits] Potential new compression method for FITS tables

[William Pence]

The idea of creating a new FITS extension type based on the contents of 
a FITS table has been suggested in the past.  For instance, when the 
hierarchical grouping convention was developed (in the 1990s?) there was 
a suggestion to create a new 'GROUPING' extension that would in fact be 
identical to the structure of a BINTABLE extension, but in the end it 
was agreed to use the BINTABLE extension for this purpose.

The FITS standard currently defines 3 standard extension types - IMAGE, 
ASCII table, and BINTABLE - which represent 3 distinctly different types 
of containers for storing and organizing data.  The intention in FITS 
(as stated in section 3.4.2 of the standard) is to only define one type 
of container for a given type of data organization.  In the current case 
of this prototype table compression method, the compressed table has the 
same overall organization as in the original uncompressed  binary table 
(i.e., the compressed table contains rows and columns of data, and also 
makes use of the variable length array convention within binary tables). 
  It would violate the spirit of FITS, in my opinion, to create a new 
type of container that duplicates the structure of an existing container 
type, simply because the contents of the container is different.

I think there are much simpler and less invasive ways to mitigate any 
confusion by end users of compressed tables (if in fact they are ever 
widely used, which I think is still an open question).  As mentioned 
before, COMMENT keywords can be added to the header to explain that that 
the table is in compressed format and provide instructions on how to 
uncompress it.  There are other possibilities as well.  Since most 
software applications identify a particular table extension by the 
EXTNAME keyword, we could perhaps modify the EXTNAME value so that 
unsuspecting software would not recognize it.  For example, if the 
uncompressed table has EXTNAME = 'EVENTS', then this could be modified 
in the compressed table to be EXTNAME = 'COMPRESSED EVENTS' (or 
something similar).  The original EXTNAME value would be restored when 
the table is uncompressed.

In any case, I don't think this is a major problem in the long run. 
Either this compressed table idea will never be widely used, of if it 
is, then the major FITS table readers (such as CFITSIO and fv) will be 
adapted to transparently support this compressed format.

regards,
Bill

On 12/22/2010 11:45 AM, Forveille thierry wrote:
>    On 12/22/2010 05:32 PM, Tim Pearson wrote:
>> As a FITS user, i.e., an astronomer, I am very uncomfortable with the idea that I may be sent a valid FITS bintable that I can't make sense of. I regularly use tools like TOPCAT and fv (and my own homegrown tools) to inspect binary tables, and I would very much prefer that the tools would tell me "I do not understand this file" rather than just displaying garbage.  When considering new ways of storing data within binary tables, please keep in mind naive users like me!
>>
>> My concerns could be simply addressed by using a new extension type instead of BINTABLE. Programs that understand the compression convention would accept such extensions transparently; those that do not would tell me something is wrong.
> For what it's worth, I am on the same line. I see little to be gained by
> using the BINTABLE extension
> for compressed tables, and significant potential trouble for end users.
>

-- 
____________________________________________________________________
Dr. William Pence                       William.Pence at nasa.gov
NASA/GSFC Code 662       HEASARC        +1-301-286-4599 (voice)
Greenbelt MD 20771                      +1-301-286-1684 (fax)