[fitsbits] Abuse of EXTEND keyword

[William Pence]

Tim Pearson wrote:
> Perhaps I am misunderstanding or misremembering something here, but I  
> think EXTEND=T does actually have a purpose, and to suggest that both  
> EXTEND=T and EXTEND=F are merely advisory is crazy. We should not add  
> the words "nor does the absence of this keyword necessarily imply  
> that the file does not contain extensions" to the FITS standard.

Can you explain why this should not be done?  Also, the EXTEND = T 
keyword has always been advisory, so it seems consistent to declare that 
EXTEND = F should also be advisory.

Whether or not a FITS file has conforming extensions following the 
primary array is a simple observable fact; the presence or value of the 
EXTEND keyword will not change that fact, so the EXTEND keyword seems to 
serve little useful purpose, and even worse, it can create confusion if 
EXTEND is not present, or has the value = F in a file that actually has 
extensions.  So the purpose of the proposed change to the EXTEND keyword 
is to make it clear to software developers that they should not depend 
on the existence, or value, of the EXTEND keyword when deciding whether 
or not to read or write an extension in a FITS file.

> The proposed FITS 3.0 standard talks about "special records"  
> following the last extension, although it "deprecates" them. I don't  
> think this is or ever has been legal FITS usage: if you have one  
> standard extension (and EXTEND=T), all the special records must be  
> standard extensions. If EXTEND is not T, then you can have special  
> records that are not standard extensions. The only legal options are:
> 
> Primary HDU with EXTEND=T, followed by zero or more standard  
> extensions (HDUs)
> Primary HDU without EXTEND=T, followed by zero or more "special records"
> 
> If standard extensions are followed by additional special records,  
> there is no way to indicate where the last extension ends and any  
> "special records" begin, except to say that any records which don't  
> conform to the rules for standard extensions must be additional  
> special records. I don't think Grosbøl et al. intended that, and I  
> think their paper prohibits such additional special records.

The FITS Standard clarified the meaning of "special records", such that 
a FITS file may contain the following structures (ignoring random 
groups, which are a special case):  The primary array, followed by 
optional conforming extensions, followed by optional special records. 
Special records are defined as a sequence of 1 or more 2880 byte records 
that do not obey the rules for the primary array or a conforming 
extension (in particular, special records cannot begin with "XTENSION" 
or "SIMPLE".  It is now proposed to deprecate any further use of special 
records, without the explicit approval of the IAU FITS Working Group, on 
the grounds that a) conforming extensions should be sufficient for most 
future needs, and b) allowing anyone to tack on arbitrary data records 
to the end of their FITS files is not conducive to the interoperability 
of data.

Bill Pence
____________________________________________________________________
Dr. William Pence                       pence at milkyway.gsfc.nasa.gov
NASA/GSFC Code 662       HEASARC        +1-301-286-4599 (voice)
Greenbelt MD 20771                      +1-301-286-1684 (fax)