[fitsbits] variable length arrays

[William Pence]

Jennifer Hatchell wrote:
> 
> I'm working on FITS-based storage of raw data for APEX (a single-dish
> submm telescope) and considering use of variable length arrays.  These
> have obvious advantages in terms of storage efficiency, but I have
> concerns about access speed and because they are not universally
> supported.  Should I be avoiding these?

If it would significantly reduce the size of the files, then I can see
little reason to avoid using 
variable length arrays in FITS binary tables.  Several major projects have
been using variable length arrays for many years without problems.  The
CFITSIO library, as well as several other FITS I/O libraries listed on the
FITS Support Office Web site, support reading and writing variable length
arrays.

Access speed should not be a concern except in extreme cases where the
vectors are all very short and the overhead of reading or writing the 8-byte
descriptor to each variable length vector becomes significant.  Typically it
is faster to read or write a table containing variable length arrays simply
because the data file is smaller than if the same information is written to
fixed length arrays.  

The main disadvantage to using variable length arrays, compared to using the
more conventional fixed length arrays, is that it is more cumbersome to make
updates to the vectors, especially if the length of the vector is modified. 
For this reason variable length arrays are better suited for use in fairly
static data files that will not need to be modified in place very
frequently.

The variable length array feature is currently described in an unofficial
appendix to the FITS Standard document, but in November 2002  the WGAS FITS
Committee (the North American
regional FITS standards committee) voted 17-0 to officially recognize this
convention in the formal FITS Standard document. Hopefully, the other 2
regional FITS committees and then the IAU FITS committee will also vote on
this soon.

Bill Pence
-- 
____________________________________________________________________
Dr. William Pence                       William.D.Pence @ nasa. gov
NASA/GSFC Code 662         HEASARC      +1-301-286-4599 (voice)     
Greenbelt MD 20771                      +1-301-286-1684 (fax)