[fitsbits] XTENSION = 'FITS' proposal

[William Pence]

Here's my $0.02 worth of comments on Perry's XTENSION = 'FITS' proposal:

This proposal offers a way to group a sequential set of HDUs in a FITS
file.  The HDU with XTENSION = 'FITS' basically serves as a marker to say
that all the HDUs contained in following NAXIS1 bytes should be treated as a
group.  However, having to update the NAXIS1 keyword value (and the proposed
'XINDn' index keywords) whenever any of the HDUs within that group are
modified, or when HDUs are added or deleted in the group, is a big
disadvantage.  An alternate way to sequentially group HDUs would be to
define a 'begin group' and 'end group' HDU that would delimit all the HDUs
in that group.  These group marker HDUs could simply be dataless IMAGE
extensions (with NAXIS = 0) that have EXTNAME = GROUP_BEGIN and EXTNAME =
GROUP_END, respectively.  As with Perry's proposal, one could have groups
imbedded within groups.  Using these group marker HDUs would eliminate the
need to update any pointers that define the size of the group, and make it
easy to insert or delete HDUs within a group.

One big advantage to using these GROUP_BEGIN and GROUP_END marker HDUs
instead of a new XTENSION = 'FITS' HDU is that existing FITS software would
still be able to access any of the HDUs within the file, even if it knows
nothing about the grouping convention. 

I'm not convinced, however, that either of these ways to sequentially group
HDUs provides really useful new functionality.  There are other ways to
group HDUs within a FITS file that would not require the HDUs in the group
all be sequentially ordered, and would require no new types of HDUs.  For
example, one could invent a convention that simply use the EXTNAME keyword
itself to define the hierarchical structure of the HDUs in the file.  For
example, (and this simply illustrates the concept, and is not fully
developed proposal) a  FITS file might contain 6 HDUs which have the
following EXTNAME values:

        EXTNAME = 'IMAGE1'
        EXTNAME = 'IMAGE2'
        EXTNAME = '[IMAGE1]ExpMaP'
        EXTNAME = '[IMAGE2]ExpMap'
        EXTNAME = '[IMAGE1]BadPix'
        EXTNAME = '[IMAGE2]BadPix'

This defines 2 hierarchical groups, each consisting of a base image and
associated exposure maps and bad pixel lists, but in this case there is no
restriction that the HDUs in the group be physically adjacent in the FITS
file, or even co-located within the same file.

Of course there already exists a well developed proposal for hierarchically
grouping of FITS HDUs by Jennings, et al (Full Disclosure Notice: I am one
of the co-authors) at http://adfwww.gsfc.nasa.gov/other/convert/group.html. 
This proposal may appear complex at first reading, but it is very general
and provides a lot of capabilities. There also already exists a set of
high-level support software routines, distributed with the CFITSIO library,
which perform common types of operations such as creating a Grouping table,
adding or deleting HDU members to a group, verifying that all the member
HDUs in a group actually exist, etc. See the 'Hierarchical Grouping' chapter
in the CFITSIO users guide for more details (on line at:
http://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node58.html).

The INTEGRAL Gamma-ray mission, to be launched later this year, is using
this grouping convention in a big way in their data analysis pipeline, so
this is no longer just a theoretical concept.

-Bill
-- 
____________________________________________________________________
Dr. William Pence                          pence at tetra.gsfc.nasa.gov
NASA/GSFC Code 662         HEASARC         +1-301-286-4599 (voice)     
Greenbelt MD 20771                         +1-301-286-1684 (fax)