[fitsbits] Abuse of EXTEND keyword

[Tim Pearson]

> 1.  The EXTEND keyword:  Since the STEREO mission has started  
> producing
> FITS files with EXTEND = F, it is too late to prohibit this usage  
> in the
> FITS Standard (because of the "once FITS, always FITS" rule).  The
> technical panel was not aware of this usage when it drafted the change
> to the EXTEND keyword definition (which was modeled after the  
> definition
> of the BLOCKED keyword which can only have a T value, never F).  The
> definition of the EXTEND keyword should probably be revised to explain
> what is meant by EXTEND = F, e.g., something perhaps like this:
>
>    "If the EXTEND keyword value is the logical constant F, then this
>    suggests that there are no extensions following the primary array.
>    This keyword is only advisory, however, so extensions may
>    follow the primary array even if EXTEND = F, but this is not
>    recommended."

Perhaps I am misunderstanding or misremembering something here, but I  
think EXTEND=T does actually have a purpose, and to suggest that both  
EXTEND=T and EXTEND=F are merely advisory is crazy. We should not add  
the words "nor does the absence of this keyword necessarily imply  
that the file does not contain extensions" to the FITS standard.

The Grosbøl et al. paper (1988 A&A) said:

"Note that the presence of EXTEND=T in a primary FITS header ...  
indicates that the file _may_ have extensions records and that any  
special records will conform to the rules below." The important part  
of this statement is the last bit, "any special records will conform  
to the rules below" - meaning that the special records follow the  
rules laid down for standard extensions, so that in particular the  
size of each extension and hence the starting point of the next  
extension can be determined.

Note that in this paper "special records" are any 2880-byte records  
that follow the last record of the primary HDU. It is (or was) legal  
FITS to put anything you like in these records, so long as EXTEND=T  
is not present in the primary header.

If EXTEND is not present in the primary header, or if EXTEND is  
present with any value other than T, then the reader should not  
attempt to interpret any special records as standard extensions.

My guess (it's a long time ago) is that there were FITS files out  
there that had special records that did not conform to the rules for  
standard extensions, so the EXTEND keyword was needed. Or perhaps  
Grosbøl et al wanted to leave open the possibility of other uses for  
the special records, which would be indicated by different header  
keywords.

So if we change the standard to make EXTEND optional in all cases, we  
are effectively prohibiting use of the "special records" for anything  
except standard extensions. This sounds like an incompatible change  
to me.

The proposed FITS 3.0 standard talks about "special records"  
following the last extension, although it "deprecates" them. I don't  
think this is or ever has been legal FITS usage: if you have one  
standard extension (and EXTEND=T), all the special records must be  
standard extensions. If EXTEND is not T, then you can have special  
records that are not standard extensions. The only legal options are:

Primary HDU with EXTEND=T, followed by zero or more standard  
extensions (HDUs)
Primary HDU without EXTEND=T, followed by zero or more "special records"

If standard extensions are followed by additional special records,  
there is no way to indicate where the last extension ends and any  
"special records" begin, except to say that any records which don't  
conform to the rules for standard extensions must be additional  
special records. I don't think Grosbøl et al. intended that, and I  
think their paper prohibits such additional special records.

- Tim Pearson