[fitsbits] Proposed Changes to the FITS Standard

[Don Wells]

Dear Friends-of-FITS,

William Pence wrote:
> ..1.  A short (9 page) summary of the main recommendations..

I have examined the 'Recommended Changes' document dated 07-18, and find 
that the recommendations are acceptable.  My compliments to the chefs!

-=-=- On escape hatches -=-=-

As I read the document, one big issue caused me to pause and think.   
That is the concept of 'escape hatches' in FITS.  This feature has been 
essential for the success of FITS over the past 28 years.   We have a 
long history of leaving certain 'holes' in the specifications, precisely 
to permit clever people to create new designs.  This recommendation 
document appears to be closing some of those escape hatches, and that 
makes me nervous.

Item 4 of section 1 recommends that we deprecate the special records 
convention.   The justification given is appropriate.  The special 
records rule was the biggest single escape hatch in the original FITS 
Agreement of 1979; it was the basis for the Generalized Extensions 
Agreement of 1984, which led to TABLE, BINTABLE and IMAGE extensions.  I 
agree that the Generalized Extensions Agreement "appears to be 
sufficient to meet most future needs".   Indeed, I would go further and 
say that the burden of proof is on anybody who claims that the 1984 
Agreement is not able to satisfy some astronomical data structure 
requirement.  The variety of conventions that have been built on top of 
BINTABLE show that it alone has enough versatility and functionality to 
solve an enormous range of problems.  Also, if we encounter something 
that BINTABLE cannot do, it is always possible for us to encapsulate 
*any* other data format within FITS, if the other format solves a 
problem that FITS can't solve, by creating extensions with 
XTENSION='<other_format>' and the other file in an NAXIS=1 byte array.   
Therefore, although this item makes me nervous, I have decided to accept 
deprecation of special records.

Item 10 says XTENSION type names *must* be registered.   I would prefer 
a strong urging accompanied by the usual justification rather than a 
'must', precisely because XTENSION is our most important escape hatch, 
and I don't want to inhibit experimentation unnecessarily.   However, I 
won't insist on this matter.

Item 21 recommends changes to section 7.3.3.2 that would "remove the 
option to use the heap and the PCOUNT keyword in ways other than 
described in the variable length array section 7.3.5".   This wording 
makes me nervous because tricks with the heap and the gap area are an 
obvious potential escape hatch that we left in the Binary Tables 
Agreement in 1991.  I decided to examine the color-coded differences 
document (very nice!).   I see changes marked that seem to be related to 
this item, but I am unsure which changes accomplish the goal of the 
item.   Perhaps this item in the Recommended Changes document should 
have some text added to clarify this point, or maybe the differences 
document is missing some color-coding.

While examining the differences document, I encountered two sentences in 
7.3.5 that seem to me to be inconsistent.  The first is "The meaning of 
a negative value for either of these integers [in the array descriptor] 
is not defned by this standard."  The second is "The storage referenced 
by an array descriptor must lie entirely within the heap area; negative 
offsets are not permitted." The first sentence leaves open the option of 
a negative offset (and that is an obvious escape hatch, because the 
integers are definitely *signed*), but the second sentence closes off 
the option.  (The text of the second sentence is present in v2.1, so 
this isn't the change made for item 21; I don't remember when this 
prohibition was introduced; certainly the IAU-FWG could remove it if 
they wish.)  Apparently the second sentence overules the first sentence, 
but the situation makes me nervous.   Negative offsets would address the 
gap area, and this might be the basis for some clever design that I 
cannot imagine at this moment.   However, I see nothing to stop a 
designer from using simple integer fields in the binary table to contain 
pointers that her software would construe as data descriptors for the 
gap area.  I have decided not to request removal of the negative offset 
prohibition, but I do suggest that the apparent conflict of the two 
sentences be reviewed.

-Don Wells
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dwellscho.vcf
Type: text/x-vcard
Size: 197 bytes
Desc: not available
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20071017/2a75197c/attachment.vcf>