[fitsbits] Bintable proposals

[Mark Calabretta]

On Mon 2001/11/19 16:01:07 CDT, William Pence wrote
in a message to: Mark Calabretta <Mark.Calabretta at atnf.CSIRO.AU>
and copied to: fitsbits <fitsbits at NRAO.EDU>

Dear Bill,

>is defined by the TNULLn keyword for integer columns or is a 'Not-a-Number'
>value for floating point columns).   The advantage of this solution is that
>the FITS header is perfectly legal;  the only non-legal aspect of this file
>is that it is shorter than is implied by the NAXIS2 keyword value, but as

Legal, perhaps, but doesn't it amount to exploiting a loophole in the
standard?  NOST describes the representation of null values but not the
semantics of a null-value filled table row.  (For example, what if all but
one of the values were null?)

With the null-value filled table row, each data type in the table has to
be correctly nulled.  Then presumably the reader has to check the whole row
rather than just the first column.  It also precludes additional extensions
and variable length arrays.  The reader also somehow has to intuit that
this is a sequential file on which random access must not be attempted.

In fact, this was my original desperation solution but I have shown that
there is a much better one which allows additional extensions and, if anyone
cares to promote it, variable length arrays.  I think this application is
important enough to introduce new semantics rather than try to coax the
existing semantics to perform tricks they weren't designed for.

Note that it is a feature of the design that NAXIS2 = -1 will cause unaware
readers to fail!  NAXIS2 = -1 means "there are new semantics here, if you
don't understand them then give up now".  It's effectively SIMPLE = F, but
with the reason given.

Anyway, NAXIS2 = -1 is essentially for our own internal use.  However, if
anyone else wants to adopt the ideas then I would advocate that they used
the same syntax and semantics.  In saying this, I note and agree with the
subtext of this discussion that it is extraordinarily difficult to
introduce new standards into FITS.  However, I also note that some
advocate the fait accompi of de facto standards as a legitimate route to
formal standardisation and, speaking from experience, this seems to apply
with unusual vigour to FITS.

>long as the application program just reads the rows of data sequentially,
>and does not attempt to read beyond the row containing the null values, then
>most existing FITS interface software (such as CFITSIO) should have no
>problem reading the file.  It would be a simple matter to update the NAXIS2
>keyword with the correct value after the file is copied off of tape to
>magnetic disk. 

>3.  I think it is time to finally discard the claims by some that FITS is
>not suitable as a data acquisition or interactive data analysis format. 

Well, at least not with the addition of this very simple construct which
is easy to understand, easy to code, and as efficient as allowed by the
constraints of sequential media.

As a data analysis format, FITS requires that the header be parsed every
time a data file is to be accessed.  Use of fixed-structure data files
allows analysis packages to fetch a header value from a specific location
without having to search for it.  However, I wouldn't say that this
overhead is not negligible these days.

>There are plenty of examples of applications that successfully use FITS
>formats in very demanding computational environments.  There may have been
>some legitimate concerns about the overhead of swapping bytes or having to
>insert new header keywords in large files 20 years ago when we were all
running on VAX 780's, but today these are really non-issues.

Yes, the 2560 byte block size and use of VAX floating point format in
RPFITS arose precisely for this reason.  That, among RPFITS' other now
unnecessary oddities, is why I would like to abandon it!

Cheers, Mark