[fitsbits] EXTEND/NEXTEND reality check

[Doug Tody]

Well, we are talking about FITS here though.  In any case, one can
make a strong case, for data beyond a certain level of complexity,
for composing complex datasets by aggregating primary data objects
(image, table, etc. - standard components) in a general container, and
describing the structure logically via a combination of relationships
and higher level metadata.  MEF works reasonably well for this.
The more "obvious" approach of explicit structure (XML etc.) makes
the structure more directly visible to a human, but is less flexible
and extensible.  Basically this is the old relational vs hierarchical
structure debate.  There is much more that could be said about this;
suffice it to say that it is by no means obvious that the explicitly
structured alternatives are a better choice, although both approaches
have their place.

Getting back to the FITS issues, the main point is that we already
routinely produce complex datasets, usually containing raw or
calibrated instrumental data, composed as a FITS MEF, often with
some higher level metadata in the primary header (or sometimes in a
special extension).  This is quite an important and powerful way to
use FITS and should not be deprecated.  One of the biggest problems
we have currently is that there is no way in standard FITS to warn a
generic client application that it cannot safely modify the file (e.g.,
edit individual extensions) without the possibility of breaking things.

One sees this with something as simple as NEXTEND.  Basically there are
two types of MEF, one where the extensions are completely independent,
and one where they are related in some fashion.  One could easily argue
that, rather than do away with things like NEXTEND, we could use a few
more extension-related keywords in the primary header to say something
about how extensions are used in this particular MEF.

	- Doug


On Sat, 25 Aug 2007, Thierry Forveille wrote:

> Doug Tody a écrit :
> > MEF files are an important part of FITS, but unfortunately this issue
> > of describing the overall MEF as a structured object has never been
> > adequately addressed.  While it may be possible for a MEF to be a random
> > collection of FITS extensions that are concatenated together, this is
> > not the only model nor necessarily even the best one.  In many applications
> > one would like to consider the MEF to be a structured object, and be able
> > to add high level information to describe the object, indendent from
> > what is stored in the individual extensions.  The most logical place to
> > put this information in FITS currently is in the primary header.
> > 
> > One could easily argue that we need *more* keywords in the primary
> > header to properly describe a MEF, not less as seems to be the trend
> > of this discussion.
> > 
> Or one could take the (in my view more logical) view that a MEF file
> is the wrong choice when strong structural information needs to be
> included :-)
>