[fitsbits] updates to the FITS standard document

[Mark Calabretta]

On Tue, 23 Jun 2015 18:53:04 -0600 (MDT)
Douglas Tody <dtody at nrao.edu> wrote:

Hi Doug,

> If the extensions are
> actual binary tables, then there may be one additional level of
> hierarchy and inheritance.

Are you referring to the Green Bank convention?  It would be of interest
to know of another application outside radio astronomy.

For the record, WCSLIB implements yet another kind of inheritance where
PHDU WCS keywords (e.g. CTYPEia) are recognised in a bintable header and
provide default values for any bintable WCS keywords (e.g. iCTYPna) that
are omitted.

As for inheritance at the HDU level, I have to agree with Bill Pence
that nowdays this is best avoided by putting multiple images in a single
bintable.

> The common/inherited information can be considerable and should not be
> discounted.  This is why it remains an issue for flat vs normalized
> tables in production databases, in VOTable (PARAM/FIELD), in VO ObsTAP
> (Observation vs Observation data product), and so forth.  The issue is
> not merely redudant and wasted storage, but an issue of data integrity
> if information is duplicated.
> 
> This complexity plus the ambiguity about what the primary HDU is used
> for, is the main problem with INHERIT.  Ideally it is not used unless
> the data creator is rigorous about providing a global primary HDU and
> extensions that extend this.  There are additional issues with what to
> do if a single extension is extracted from the MEF - the inherited
> fields must be filled in, or information is lost.  Naive software will
> not propagate the inherited fields hence information can be lost.
> 
> Clearly the general issue remains important for complex structured data.
> The problem is that FITS does not deal with the issue in a rigorous
> fashion.  Hence one solution is merely to avoid relations and
> inheritance and merely duplicate the information in every FITS entity.
> INHERIT attempts to deal with the issue, but is not rigorous enough due
> to the only partially-relational (single PHU/extension/record) nature of
> FITS.

The current disagreement between whether ignoring INHERIT is benign or
malign may stem from a disconnect between INHERIT's documentation and
knowledge of how it is used in practice.  In the example header given at

  http://fits.gsfc.nasa.gov/registry/inherit/mef_inh.txt

the inherited keywords are mostly not ones recognised by the standard,
except for a few like OBJECT, TELESCOPE, and OBSERVER.  On the contrary,
as per Mark Taylor's email, the rest of us have visions of WCS keywords
and bintable descriptors being inherited.  Losing them, by ignoring
INHERIT, would surely be malign.

In thinking how INHERIT might be improved, I consider the following
extra rules would be required:

1) No keyword recognised by the FITS standard may be inherited, except:
   DATE, ORIGIN, TELESCOPE, INSTRUME, OBSERVER, OBJECT, AUTHOR, and
   REFERENC.

   This satisfies the example header except for DATE-OBS, MJD-OBS,
   RADECSYS (archaic), and TIMESYS.

2) A block of PHDU keyrecords that are to be inherited must be delimited
   by INHERIT = T immediately preceding them, and INHERIT = F (or END)
   immediately following them.  There may be several such blocks.  They
   signal a FITS reader to cache keyrecords for possible inheritance by
   an extension.

> On this general issue of incorporating FITS conventions into the FITS
> standard, I think the main issue is, are we merely introducing the major
> conventions in the standard document as optional extensions, or are we
> recommending these as integral components of the standard?  These are
> different things.  Probably some belong in one category or the other.  I
> am ok with introducing the major conventions as optional elements within
> the standard, often mainly relevant to some subsector of astronomy, with
> references to external documents for the details, but think it is a
> different matter if an existing convention is to be promoted to a part
> of the standard.  We should distinguish between the two cases, in which
> case it becomes much simpler to decide what to focus the effort on.

The working groups' suggested alterations of the text of the standard
implies that each of these conventions are being proposed as new
standards.  At least, that is my interpretation.

Regards,
Mark Calabretta