[fitsbits] Bintable proposals

[Bill Cotton]

Mark,

   What you address is a fundamental FITS limitation rather than just
a binary tables one.  Each header is required to say what the length
of its data unit is.  This does present problems for data acquisition.
A slightly messier, but legal, solution is to break up the data into
smaller table units of, say, 1000 or 10000 rows and start a new table
each time one fills.  The last table could be null filled.  Buffering
entire tables in memory could get around the variable length array
problem.  
   This solution also helps with another tape problem.  If there are
media errors, the reading program can skip to the next table header
as it has predictable strings.  This will limit the data lost to
aging tapes.

-Bill

Mark Calabretta writes:
 > 
 > On Fri 2001/11/02 14:57:07 CDT, William Pence wrote
 > in a message to: FITSBITS <fitsbits at cv.nrao.edu>
 > 
 > >Does anyone have any comments on these 2 proposals before they are submitted
 > >to the WGAS FITS committee?
 > 
 > Greetings,
 > 
 > I have used binary tables extensively for single dish data, including use
 > of the variable length array convention.  In fact I instigated the use of
 > variable length arrays in SDFITS.  I found their implementation within
 > cfitsio to be virtually transparent.
 > 
 > However, I have a comment on their use which relates to the requirement
 > that NAXIS2 store the number of rows in the table.
 > 
 > The fact that NAXIS2 appears in the header and has no default value means
 > that when writing to a streaming device such as a tape drive the size of
 > the output binary table must be known before writing can commence.  In
 > practice, this precludes the use of binary tables for acquisition of data
 > without using a disk as intermediary since the table size is not generally
 > known in advance.
 > 
 > The application in question (at Parkes) involves dumping prodigious amounts
 > of pulsar pulse profile data directly to DLT at close to its maximum I/O
 > rate.  It was not considered feasible to stage this data to disk.
 > 
 > However, since each table row is a fixed length and the binary table is the
 > only extension, one solution is simply to write NAXIS2 = -1 and let the FITS
 > reader find the eof.  Later on, as part of the processing, this invalid
 > binary table, or the products thereof, could be made legal by rewriting the
 > true value of NAXIS2.  So given the choice either of setting NAXIS2 = -1 or
 > avoiding the use of binary tables altogether we opted for the former.
 > 
 > In general, the only requirement is to be able to determine the end of the
 > table and I can only think of the following impediments:
 > 
 >    1) Extensions following the BINARY table,
 > 
 >    2) Table row length less than 2880 bytes (i.e. whole table rows may fit
 >       in the block padding),
 > 
 >    3) Use of variable length arrays.
 > 
 > (1) and (2) do not affect us but anyway they do not appear to be basic
 > problems.  Only (3) is fundamentally limiting.
 > 
 > Since we're using a non-standard construct and don't intend to write
 > variable length arrays their standardization is not an issue for us and
 > this note should not be considered an objection to that.  However, if
 > NAXIS2 = -1 ever did become standard usage it would have to preclude the
 > use of variable length arrays.
 > 
 > Mark Calabretta
 > ATNF
 > 
 > 
 > _______________________________________________
 > fitsbits mailing list
 > fitsbits at nrao.edu
 > http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits