[fitsbits] BLANK keyword misinterpretation

[Steve Allen]

We are currently testing of the new CCD mosaic which is being readied
for shipment to Keck to replace the old detector in the HIRES
spectrograph.  The laboratory setup permits illumination patterns
which are hard to duplicate in the instrument after deployment.
In the course of examining the images we discovered that I had made a
misinterpretation of the FITS standard.

I believe that the FITS standard is unambiguous.  Nevertheless, slight
changes to the wording of the standard could be very useful in
preventing other such misinterpretations.  Until it becomes possible
to make such changes other implementors should be cautious when using
the BLANK keyword.  It is also possible that changes to the APIs that
are used for writing FITS images would help.

Our data originate via a 16-bit AtoD conversion which produces bits
corresponding to C "unsigned short" on modern big-endian processors.
The pixel values are always offset such that all values are positive.
We store the bits directly in the image arrays as unsigned short
integers using the traditional trick of offsetting the physical data
values using the keywords

BITPIX   = 16
BZERO    = 32768
BSCALE   = 1

The complete HDU structure of the mosaic FITS files is created at the
beginning of the CCD readout, and at this time the values in the data
arrays of all IMAGE extensions are initialized to be all zero bits.
During the readout the FITS file is stored in shared memory where it
can be accessed by FITS viewer to provide real-time display of the
incoming image.  In our systems the pixels may not arrive in sequence,
so the images being constructed may have "holes" during the readout.

The "holes" in the image during the readout confound the automatic
contrast setting algorithm in the FITS viewer.  However, the FITS
viewer that we use understands the usage of the BLANK keyword.  In
order to instruct it ignore pixels that do not yet have data I
modified the FITS headers to include the keyword

BLANK    = 0

Whereas this had the intended effect during real-time readout, this
was a mistake for the archival data.

FITS is defined by NOST 100-2.0 as printed in
http://adsabs.harvard.edu/cgi-bin/nph-bib_query?bibcode=2001A%26A...376..359H

The interpretation of BITPIX is shown by Table 5.2 in section 5.4.1.1.
I believe it would be helpful to many implementors if the text where
slightly enhanced such that the entries
		16-bit twos-complement binary integer
		32-bit twos-complement binary integer
were changed to read
		16-bit twos-complement (signed) binary integer
		32-bit twos-complement (signed) binary integer

The definition of the data representation for BITPIX = 16 is given by
section 6.2.2.  It is completely unambiguous.

The definition of the BLANK keyword is given in section 5.4.2.5.
Its relevant sentence reads:
    The value field shall contain an integer that specifies the
    representation of array values whose physical values are
    undefined.
This is unambiguous, but I misinterpreted it.  I would have got
the interpretation right if there had been one extra sentence
akin to this:
    This number corresponds to the data array values before applying
    any transformation indicated by BSCALE and BZERO.

The problem is that with these values of BZERO and BSCALE the signed
FITS array value of 0 corresponds to the unsigned CCD data value of
32768.  The result of my misinterpretation is that images with a
gentle ramp of CCD data values in the vicinity of 32768 have sprinkles
of single pixels where a conforming FITS viewer flags the pixels as
NULL or undefined.

In our software the fix for this is relatively straightforward.  The
image in shared memory during readout is currently used only for
real-time display.  There is a final pass over each HDU where keyword
are modified before the image is written to disk for archiving.  In
that pass the software will reset the keywords such that

BLANK    = -32768

This change will have no significant impact on real-time performance.

We are using CFITSIO as the API for creating these FITS images.
This problem could also be alleviated if the API provided an
explicitly defined means of indicating BLANK along with other
parameters of the image.

--
Steve Allen          UCO/Lick Observatory       Santa Cruz, CA 95064
sla at ucolick.org      Voice: +1 831 459 3046     http://www.ucolick.org/~sla
PGP: 1024/E46978C5   F6 78 D1 10 62 94 8F 2E    49 89 0E FE 26 B4 14 93