[fitsbits] XTENSION = 'FITS' proposal

[Perry Greenfield]

----- Original Message -----
From: "Arnold Rots" <arots at head-cfa.harvard.edu>


> Perry argues that the proposed FITS extension provides for a
> hierarchical indexing system.  In my opinion that functionality is so
> limited that it is not worth the trouble.
>
> For a hierarchical indexing scheme to be useful one needs to be able
> to construct a table with parameters and links (URLs) to individual
> extensions in any file.  Being able to link extensions across files in
> a table that just contains the indexing information is far superior to
> the limited functionality of this proposal that only allows linking
> extensions within the same file and requires the entire file to be
> embedded as well.
>
> Just my 25c...
>
Whether a table that links files is "far superior" depends on the context
and the kind of data being associated. There are trade-offs to bundling
data in the same file and keeping it in different files. Having many large
datasets in one file is admittedly clumsy. Can all associations of data be
handled by placing it in one file? Of course not. On the other hand,
handling
associations with a table and separate files also has its problems. First
of all it is brittle. Files are easily manipulated outside of the
'interface'.
They may be moved, modified, or renamed without the table being
appropriately updated. They move part of the naming scheme into
the realm of the operating system, which unfortunately, do not share
the same naming capabilities or relationships. Despite the commonalities
in OS filesystems these days, there are still annoying issues to deal with.
This is what makes a 'linking' system particularly inappropriate for
archival use. When contained in a file, the structure is manifest and not
brittle.

Furthermore, there are many issues where the data elements to be bundled
are not large and are most appropriately included in the same file. Things
like bad pixel lists, associated coordinate information, engineering data
are
often small or moderate in size and best bundled in the same file.

We have repeatedly found the naming scheme for FITS files to be limiting
and had we been able to nest data within a file, we would have readily
taken advantage of it to avoid the more clumsy solutions that were used.
(For example, HST STIS extracted spectra).

But the biggest issue is that there is no clear convention for hiearchical
FITS data yet (I'm aware of the one proposal). To get one accepted
would take a long time (if FITS WCS is any guide). For FITS within
FITS, what is there to decide? Virtually nothing. It's as straightforward
as could be. Many FITS readers could be adapted to use it compartively
easily. Isn't this the classic 'low hanging fruit'? So what if it doesn't
solve
all problems. It solves a lot of ones we have deal with and that's
good enough for me ;-)

Perry