[fitsbits] Bintable proposals

[William Pence]

Just a few brief comments on some of the previous posts on this subject:

1.  Mark Calabretta advocated using NAXIS2 = -1 in binary tables written to
tape in which the number of rows in the table is not known before hand.  He
also suggested adding an integer column to the table which would be set = 0
to mark the end of the table.   In this special circumstance (in which it is
also known that there will be no HDUs following the table, and there is no
heap data) I would suggest instead to use a very large, but still legal
value for NAXIS2 (e.g., NAXIS2 = 10**9), and mark the actual end of the
table by writing a row containing null (not zero) values.  (The null value
is defined by the TNULLn keyword for integer columns or is a 'Not-a-Number'
value for floating point columns).   The advantage of this solution is that
the FITS header is perfectly legal;  the only non-legal aspect of this file
is that it is shorter than is implied by the NAXIS2 keyword value, but as
long as the application program just reads the rows of data sequentially,
and does not attempt to read beyond the row containing the null values, then
most existing FITS interface software (such as CFITSIO) should have no
problem reading the file.  It would be a simple matter to update the NAXIS2
keyword with the correct value after the file is copied off of tape to
magnetic disk. 

2.  I agree that setting SIMPLE = F in this case is of no practical value. 
All it would do is tell a human reader that there is something funny about
the file.  FITS reading software, if it is to read the file at all, must
ignore SIMPLE = F.

3.  I think it is time to finally discard the claims by some that FITS is
not suitable as a data acquisition or interactive data analysis format. 
There are plenty of examples of applications that successfully use FITS
formats in very demanding computational environments.  There may have been
some legitimate concerns about the overhead of swapping bytes or having to
insert new header keywords in large files 20 years ago when we were all
running on VAX 780's, but today these are really non-issues.  Take the
example of having to shift all the data blocks in order to insert a new
header keyword.  On my Pentium III machine it takes less than 2 seconds for
CFITSIO to insert a new block (able to hold 36 new keywords) at the
beginning of a 100 MB FITS file.  This operation only needs to be performed
every 36 new keywords, so on average it only takes 2/36 = 0.05 seconds to
add a keyword to a 100 MB file, or 0.5 seconds to a 1 GB file.  This is
certainly not an excessive amount of time to expect users to endure in an
interactive data processing environment.  

-Bill Pence
-- 
____________________________________________________________________
Dr. William Pence                          pence at tetra.gsfc.nasa.gov
NASA/GSFC Code 662         HEASARC         +1-301-286-4599 (voice)     
Greenbelt MD 20771                         +1-301-286-1684 (fax)