[fitsbits] Proposed Changes to the FITS Standard (2)

[William Pence]

Don Wells wrote:
> While examining the differences document,
[at http://fits.gsfc.nasa.gov/fits_draft.html]
> I encountered two sentences in 
> 7.3.5 that seem to me to be inconsistent.  The first is "The meaning of 
> a negative value for either of these integers [in the array descriptor] 
> is not defned by this standard."  The second is "The storage referenced 
> by an array descriptor must lie entirely within the heap area; negative 
> offsets are not permitted." The first sentence leaves open the option of 
> a negative offset (and that is an obvious escape hatch, because the 
> integers are definitely *signed*), but the second sentence closes off 
> the option.  

Back in 2005 when the IAU FITS Working Group approved the variable 
length array convention as part of the FITS Standard, it was decided to 
specifically define that the 2 integers in the "P" array descriptor are 
"signed" 32-bit integers (and not "unsigned" integers) and to add the 
statement that "The meaning of a negative value for either of these 
integers is not defned by this standard."

Allowing negative array descriptor values is not necessarily a 
contradiction to the later statement that "negative offsets [in the 
heap] are not permitted", because it is possible to define a convention 
that maps (via some mathematical function) a negative descriptor value 
into a positive heap offset value.  Note that the same thing was done 
long ago with the BITPIX keyword; originally the BITPIX value had to be 
positive, but later the definition was extended so that if the value is 
negative then the actual number of bits per pixel is derived by 
multiplying the BITPIX value by -1.

Arguably, the most useful function for mapping a negative array 
descriptor value into a positive heap offset is this one:

      heap_offset = 2**32 + descriptor_value
                  = 4294967296 + descriptor_value

(where the descriptor_value is a negative signed 32 bit integer in the 
range -1 to -2147483648).  This function effectively doubles the useful 
size of the heap:  positive descriptor values can refer to heap offsets 
in the range 0 to 2147483647, and the negative descriptor values map 
into heap offsets in the range 2147483648 to 4294967295.

Incidently, it is no coincidence that this mapping function produces 
exactly the same positive heap offset value as would be given if one 
were to interpret the 32-bit descriptor value as an unsigned 32-bit 
integer.

This begs the question of what, if anything, should be done to clarify 
the use or legality of negative array descriptor values in variable 
length array columns in binary tables.  Should the mapping function 
described above be viewed as a blatant attempt to circumvent the 
restriction on interpreting the array descriptors as "signed" integers 
and thus should be expressly forbidden in the Standard, or, on the other 
hand, is this a clever adaptation to work within the rules of FITS to 
allow new functionality (by doubling the useful size of the heap area)? 
    My on feeling on this is that we should either leave the wording of 
the Standard as it is (with the intentional ambiguity about the meaning 
of negative array descriptor values), or, go back to the fundamental 
issue and redefine the array descriptors to be unsigned integers, which 
by definition would eliminate the need to explain the interpretation of 
negative descriptor values.

Bill Pence
--
____________________________________________________________________
Dr. William Pence                       pence at milkyway.gsfc.nasa.gov
NASA/GSFC Code 662       HEASARC        +1-301-286-4599 (voice)
Greenbelt MD 20771                      +1-301-286-1684 (fax)