[fitsbits] XTENSION = 'FITS' proposal

[Perry Greenfield]

"William Pence" <pence at tetra.gsfc.nasa.gov> wrote in message
news:3CBB0666.8FFA71F9 at tetra.gsfc.nasa.gov...


> Stepping back a minute to look at the broader questions raised by this
> proposal, I think the biggest issue is that it would basically transform
the
> FITS file structure from a simple linear sequence of HDUs into a
> hierarchical structure with HDUs within HDUs (within HDUs (within
> HDUs...)).  I worry about the impact that this fundamental change could
have
> on existing software and existing applications libraries (like CFITSIO).
>
> This would not be a problem if only a very few specialized applications
> would ever need to be able to directly interpret the contents of this new
> 'FITS' extension, and if most general FITS software could just skip over
it.
> The implication in Perry's first message, however, was that it would be
> desirable for general FITS software to be able to directly access HDUs
that
> are embedded within this hierarchical structure.  This would probably
> require significant changes to the syntax of the user interface in order
to
> specify the location of the desired HDU within the hierarchical structure.
>
> This could also have a significant impact on software libraries like
> CFITSIO;  the data model supported by CFITSIO assumes that the HDUs in the
> FITS file are linearly arranged, and a subroutine is provided to allow
> applications programs to move forward or backward in the file and open the
> next or previous HDU.  But under this 'FITS' proposal, CFITSIO would need
to
> support a whole new dimension, to be able to move up and down the
hierarchy
> as well as forward and back in the file.  Applications would somehow need
to
> be able to tell CFITSIO that they either want to a) skip over the whole
> 'FITS' HDU to the next HDU at that same level in the file, or b) that they
> want to move down into the next level of the hierarchy and open an HDU
that
> is contained within the 'FITS' HDU.  At the moment, it is not clear to me
> how this change could be implemented in CFITSIO and what impact it might
> have on existing software.
>
Yes, this is certainly an issue. There are various ways one may want to
approach it however:

1) Easiest: The library does little to address it explicitly, but utilitles
are provided
to unpack the FITS file one level at a time. This is of no use to
applications but
does allow the user to get at the components through manual unpacking. I'm
assuming that this would be very easy to do, if not useful for application
writers.

2) The library is modified to itself handle recursive FITS objects. If on
opening a FITS file when accessing a FITS extension, one is returned some
sort
of object or handle that allows the same routines to access the FITS
extension
object as though it were a top level FITS object. (This is the latter of
your
possiblities above, b.) I can't say that this is easy to do in all FITS
packages, but
it could be very easy for some (I don't believe it would be hard in PyFITS,
though
we haven't actually looked at doing so). Depending on the structure of the
library,
it may be very easy to implement. But maybe not. (One reason for raising
this
issue). The hardest issue would be how to handle attributes (keywords) that
apply to
multiple extensions. This is equivalent to the inheritance issue raised
years ago,
and it can be a pain to implement well (at least for updates).

3) Providing a syntax for referencing a FITS extension that is nested
directly (not requiring a user or program to drill down into the heirarchy).
I suppose
that depends a lot on what the existing syntax is for selecting extensions
and whether
there is room in that syntax for hierarchical specifications (by name,
number, or both?).

I'm not arguing that it isn't a problem, it could well be. For what we are
doing, I don't
think it is, but I don't know if it is for others or not.

> The other alternative schemes that have been mentioned for supporting
> 'groups' of HDUs (i.e., the Jennings et al. proposal, or the idea of
having
> 'GROUP_BEGIN' and 'GROUP_END' HDUs to mark the group, or some new EXTNAME
> keyword convention to define groups) do not have this problem because
under
> any of these schemes the FITS files themselves remain a linear sequence of
> HDUs.
>
Yes, but these schemes don't eliminate the problem. If the library doesn't
handle it, then the user is forced to do the bookkeeping so what would be
gained? If the issue is that these proposals allow existing FITS readers
need
no changes at all, that is true, but they aren't very useful in interpreting
the
structure. If they can interpret the structure, then they must solve the
same problems
(i.e., what syntax can be used to access a nested object, and how does the
software
refer to a nested entity).

Viewed with the mindset that the library does not need to interpret the
structure, and
support merely consists of being able to access end-point extensions there
are
two approaches I would offer:

1) Have the user manually unpack the file. The only changes to the library
are to skip
'FITS' extensions and extend the existing utilities to create separate FITS
files for each
component at the top level. This approach is not very attractive since it
requires
iterative steps, and does not let utilities list out the contents very
easily unless they
are so modified.

2) Have the library in effect convert the 'FITS' extension header into a
standalone
null data extension. Then the file can be read sequentially and have
something sensible
to return for each element it finds. Would this really be hard to implement?
To be more explicit, the code that examines the next extension finds
XTENSION='FITS',
Either moves NAXIS value elsewhere (different keyword) or otherwise provides
a means
of indicating no data present. It then moves on to the next extension. All
the nested
extensions are simply interpreted this way in a sequential process. User
software
can reconstruct the grouping by using the sizes of extensions. Unlike the
GROUP_BEGIN
proposal, there are "group" keywords available. To make writing of such
files
possible, the library must allow a nonzero NAXIS1 for the XTENSION='FITS'
case only.
Perhaps I'm wrong, but these seem like minimal changes to support sequential
access.

Perry