[fitsbits] updates to the FITS standard document

[Douglas Tody]

The INHERIT convention originally came from IRAF and it is probably
accurate to say that STScI and STSDAS was shared responsiblity.  The
concept is similar to for example, VOTable or the GreenBank convention,
where elements which do not vary in each "record" (FITS extension in
this case) are allowed to be extracted out to a higher level and
inherited by each record.  In essence they are virtual columns in the
final realized table record.  In a MEF, the primary HDU may (should in
many cases) apply globally to the entire file, and the extension headers
are essentially records in a virtual table.  If the extensions are
actual binary tables, then there may be one additional level of
hierarchy and inheritance.

The common/inherited information can be considerable and should not be
discounted.  This is why it remains an issue for flat vs normalized
tables in production databases, in VOTable (PARAM/FIELD), in VO ObsTAP
(Observation vs Observation data product), and so forth.  The issue is
not merely redudant and wasted storage, but an issue of data integrity
if information is duplicated.

This complexity plus the ambiguity about what the primary HDU is used
for, is the main problem with INHERIT.  Ideally it is not used unless
the data creator is rigorous about providing a global primary HDU and
extensions that extend this.  There are additional issues with what to
do if a single extension is extracted from the MEF - the inherited
fields must be filled in, or information is lost.  Naive software will
not propagate the inherited fields hence information can be lost.

Clearly the general issue remains important for complex structured data.
The problem is that FITS does not deal with the issue in a rigorous
fashion.  Hence one solution is merely to avoid relations and
inheritance and merely duplicate the information in every FITS entity.
INHERIT attempts to deal with the issue, but is not rigorous enough due
to the only partially-relational (single PHU/extension/record) nature of
FITS.

On this general issue of incorporating FITS conventions into the FITS
standard, I think the main issue is, are we merely introducing the major
conventions in the standard document as optional extensions, or are we
recommending these as integral components of the standard?  These are
different things.  Probably some belong in one category or the other.  I
am ok with introducing the major conventions as optional elements within
the standard, often mainly relevant to some subsector of astronomy, with
references to external documents for the details, but think it is a
different matter if an existing convention is to be promoted to a part
of the standard.  We should distinguish between the two cases, in which
case it becomes much simpler to decide what to focus the effort on.

 	- Doug



On Tue, 23 Jun 2015, Erik Bray wrote:

> On 06/22/2015 09:19 PM, Mark Calabretta wrote:
>> On Mon, 22 Jun 2015 10:16:41 -0400
>> Erik Bray <embray at stsci.edu> wrote:
>> 
>> Erik,
>> 
>>> I believe that the INHERIT keyword convention, in particular, is *not* 
>>> familiar
>>> to most users of FITS in large part because many readers ignore it, and it 
>>> is
>>> benign when ignored.
>> 
>> Given the context, I take it that you meant to say "malign" here,
>> rather than "benign".
>
> No, I did mean that it is benign when ignored.  STScI is, I'm told, partially 
> responsible for creating this keyword in the first place.  And yet even 
> STScI's own FITS software has almost always chosen to ignore its presence.
>
> I suppose if you absolutely needed the keyword to be heeded, ignoring it 
> would be a problem.  But the common consensus seems to have been that *not* 
> ignoring it causes even more problems.  So as it is there is some custom 
> software that checks for INHERIT where it does matter for some particular 
> instrument's data model.  But for generic FITS files it doesn't seem to be a 
> big deal.
>
> Erik
>
>>> Suddenly requiring it to be interpreted would be very surprising to many 
>>> users,
>>> and even if support for it were implemented I anticipate "bug" reports 
>>> from
>>> users surprised that keywords that don't physically appear in some header 
>>> are
>>> suddenly showing up (somehow...the other problem with the INHERIT 
>>> convention, as
>>> documented, is that its interpretation by software is unspecified, which I
>>> believe to be dangerous and user-hostile).
>>> 
>>> If I had a vote it would be "NO" on the INHERIT convention--the others I 
>>> think
>>> have better existing support in the community and are benign and/or
>>> well-specified.  But I think this deserves more time for discussion.
>> 
>> Regards,
>> Mark Calabretta
>> 
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>