[fitsbits] once FITS always FITS

[William Pence]

Here are a few comments on several related recent FITS issues:

1.  The EXTEND keyword:  Since the STEREO mission has started producing 
FITS files with EXTEND = F, it is too late to prohibit this usage in the 
FITS Standard (because of the "once FITS, always FITS" rule).  The 
technical panel was not aware of this usage when it drafted the change 
to the EXTEND keyword definition (which was modeled after the definition 
of the BLOCKED keyword which can only have a T value, never F).  The 
definition of the EXTEND keyword should probably be revised to explain 
what is meant by EXTEND = F, e.g., something perhaps like this:

   "If the EXTEND keyword value is the logical constant F, then this
   suggests that there are no extensions following the primary array.
   This keyword is only advisory, however, so extensions may
   follow the primary array even if EXTEND = F, but this is not
   recommended."

2. Repeated keywords:  Since the recent discussions have made it clear 
that it is not a rare occurrence for existing FITS files to contain the 
same keyword multiple times (the technical panel had hoped this was not 
the case), it appears not feasible to simply declare that this practice 
is prohibited in a conforming FITS file (again because of the "Once FITS 
always FITS" rule).   It seems that the most we can do at this point is 
to strongly discourage this practice, because it is indeterminate which 
of the keywords will read by software systems when retrieving the value.

3.  On the "Once FITS, always FITS" rule:  I'm not a true FITS 'old 
timer' since I only seriously got into the FITS business in 1990, but I 
believe the origin of this basic FITS rule can be traced back to the 
following statement in the 1988 "Generalized Extensions" paper by 
Grosbol, Harten, Greisen, and Wells:

"The most important rule for designing new extensions to FITS is that 
existing FITS tapes must remain valid.  We are not permitted to alter 
the basic format in such a way as to make existing FITS tapes invalid or 
unreadable by standard FITS tape reading programs".

When the FITS Standard was drafted in circa 1990, this rule was then 
restated as "Any structure that is a valid FITS structure shall remain a 
valid FITS structure at all future times."  (see section 3.7 of the 
proposed new Standard).

In practice, I believe this rule has generally been interpreted to mean 
that if there are existing FITS files that implement a certain FITS 
feature, then that feature must remain legal for all time in the FITS 
Standard (although features can be deprecated, or strongly discouraged 
from use).  If, however, a *hypothetically* valid FITS feature has never 
been used, then it is possible to revise the FITS rules to declare that 
feature is not considered valid in a FITS file. (I listed a number of 
these types of changes that have been made in a previous email).

Note also that this "Once FITS, Always FITS" rule does not provide 
blanket protection to existing software packages from changes to FITS 
that might break the software.  I think we are in fact sensitive to any 
changes that might disrupt existing software, but there is no absolute 
ban on making such changes to FITS.

Bill Pence
-- 
____________________________________________________________________
Dr. William Pence                       pence at milkyway.gsfc.nasa.gov
NASA/GSFC Code 662       HEASARC        +1-301-286-4599 (voice)
Greenbelt MD 20771                      +1-301-286-1684 (fax)