[fitsbits] start of Public Comment Period on the INHERIT convention

Thu Jun 25 16:52:56 EDT 2015

I think Franks's message below agrees with my own feelings about the INHERIT 
convention.  There are a number of different types of information which can be 
stored in a FITS header.  This can be put in the context of the traditional "six 
Ws": who, what, when, where, why, and how.  Generally, one can think of the 
World Coordinate System as addressing the when and where, while other keywords 
such as TELESCOP, INSTRUME, OBSERVER, OBJECT, etc. address the who, what, why, 
and how.

It seems to me to be quite reasonable to store general metadata about an 
observation in the PHU, while metadata specific to the data arrays would be in 
the extension headers.  I'm aware of projects which do just that.  Thus, the 
primary header could include the keywords mentioned above, along with technical 
information such what filters were used, the location of the observatory, etc. 
I would even include in that *some* keywords related to time, such as DATE-OBS, 
DATE-END, XPOSURE, so long as they apply to the observation as a *whole*.  This 
kind of information, useful for cataloging observations for use by virtual 
observatories and the like, is traditionally found in the primary header, and I 
totally agree with Frank's point about having a single point of maintenance. 
For these reasons, reserving the INHERIT keyword to signal that this is being 
done makes a lot of sense to me.

I'm less comfortable, however, with using the INHERIT mechanism for keywords 
which pertain to specific data arrays, such as CRPIX1, CRVAL2, etc.  There are 
instances where this might be useful.  For example, if one had a number of 
arrays which were exactly matched pixel-by-pixel to each other, it might make 
sense to only specify the physical coordinates once.  For example, one might 
have an image in one HDU, and the exposure times for each pixel in another HDU. 
  Keywords such as CRPIX1, CRVAL1, etc. often depend on calibrations, and are 
subject to change as one's understanding of the instrument matures.  Having a 
single point of maintenance for these calibration-dependent parameters also 
makes a lot of sense.

On the other hand, as has been pointed out, there's certainly a potential for 
confusion if the INHERIT convention is misused, especially once the data starts 
to be modified.  For instance, one can image the image in the above example 
being processed in a manner in which it's no longer in a pixel-by-pixel 
correspondence with the array containing the exposure times.  This could 
potentially cause a mess.

All this being said, I'm not sure where I stand at this point about the INHERIT 
convention being incorporated into the standard.

Bill Thompson

P.S.  There's also been some discussion about using binary tables versus 
multiple HDUs.  Personally, I'm a big fan of binary tables.  However, I'm also 
aware that there's major resistance to binary tables within some sections of the 
community.

On 06/23/15 19:42, Frank Valdes wrote:
> My comment is a positive for its usefulness.  While one argument for this
> convention was about minimizing space, particularly for lots of small images
> like a collection of 1D spectra, my appreciation has been greatest for the
> aspect of updating information in MEF files.  By this I mean that there is only
> one place that needs updating.   An example would be a collection of CCD images
> from a mosaic camera, say DECam, where the observer keyword is wrong.  To
> remediate this just requires changing the OBSERVER keyword in the PHU rather
> than repeating this for all 60 extensions.  I think it also encourages data
> format designers to think about good decomposition much along the design of
> databases that are well normalized.
>
> I am not particularly commenting on the legalese description that would be
> needed in a standard or whether it should remain as a convention.  But if it was
> codified I think it would need to be stronger about distinguishing format
> keywords from documentary keywords.  What I mean is that it would not have
> occurred to me to save space with TFORM, NAXIS, CRVAL, etc. type of keywords.
>   If it stays as a convention then I would object if words such as "deprecated"
> or "not recommended" were used.  Clearly there is a lot of data with this
> convention and I would continue to use it in its intended form as a way to
> encapsulate descriptive metadata about collections in one place but allow
> application to access them as if they were part of the extension without having
> to decide that the keyword is missing and then go to the PHU.  In other words
> for this last point, the joining of PHU and EHU belongs in the supporting
> software and not the science application.
>
> Frank Valdes, NOAO
>
> P.S. I really don't like the suggestion that multiple "images", say traditional
> CCD images, which are related should be "packed in a single binary table".
>
>> *From: *William Pence <William.Pence at nasa.gov <mailto:William.Pence at nasa.gov>>
>> *Subject: **Re: [fitsbits] start of Public Comment Period on the INHERIT
>> convention*
>> *Date: *June 23, 2015 at 11:56:57 AM MST
>> *To: *FITSBITS <fitsbits at nrao.edu <mailto:fitsbits at nrao.edu>>
>>
>> I share many of the concerns that others have expressed here about promoting
>> the INHERIT convention into the FITS Standard.
>>
>> It seems to me that the motivation for using the INHERIT convention to save
>> what typically amounts to less than a few 100K bytes of disk space per file is
>> now greatly diminished from when it was first developed back in 1995, since
>> the cost of disk storage has dropped so dramatically. Also, data providers are
>> now much more comfortable with more efficient ways of packing multiple data
>> arrays into a single binary table (which were only officially approved the
>> year before in 1994), instead of writing each array as a separate image
>> extension. Thus, I think it is questionable whether this convention continues
>> to serve a useful purpose.
>>
>> Implementing this convention in FITS reading and writing software also seems
>> to me to be a rather complex and somewhat ambiguous task.  It is not exactly
>> clear what action should be taken with regard to preserving the inherited
>> keywords during complex data processing operations where the values of some of
>> the inherited keywords may be modified, or when copying an HDU which uses the
>> INHERIT convention into another FITS file.  Because of these complexities, I
>> have never attempted to support the INHERIT convention in the CFITSIO library.
>>
>> So far, all the comments about the INHERIT convention have been quite
>> negative.  Does anyone want to offer any positive comments in support of
>> incorporating this convention into the FITS standard?
>>
>> -Bill
>
>
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>

-- 
William Thompson
NASA Goddard Space Flight Center
Code 671
Greenbelt, MD  20771
USA

301-286-2040
William.T.Thompson at nasa.gov