[fitsbits] Bintable proposals

[tam]

Bill,

The only concern I would have with adopting these proposals is the
restriction to 4-byte integer offsets within the variable length array
convention.  Effectively this limits the total size of variable length records
to about 2 GB (4 GB absolute maximum) in an extension.
This is already marginally inadequate for the sizes of datasets
that we have today and may be a serious bottleneck by the time the
change gets approved.  Should we consider adding a 'Q' data type for
binary tables which uses a pair of 8 byte integers in the same way as
the P type uses 4 bytes now, and work to adopt this expanded recommendation?
I think the need for 8 byte integers for file offsets is much greater than
in data types -- and there your poll showed strong support for an eventual extension.

Gigapixel arrays are not too far in the future...

I'd hope this would be a relatively non-controversial change -- compared to adding
8 byte integers to images and binary table colums, but if it will delay things
by years (decades!) I'd go ahead with the current proposal.

		Tom

> Proposal 1:
> -------------------------------------------------------------------------
>        Recommendation to Incorporate the "Variable Length Array"
>           Facility in Binary Tables into the FITS Standard
> 
>      W.D. Pence (NASA/GSFC), D. Tody (NOAO), and W.D. Cotton (NRAO)
>                            2 November 2001
> 
> The "Variable Length Array" FITS data format that is described in Appendix
> B.1 of the FITS Standard is now used by a number of major data analysis
> packages.  Examples of current usage include the "Redistribution Matrix
> Files" (RMF), widely used in X-ray astronomy to store calibration
> information about the instrumental detector gain and detector energy
> resolution, and the pixel mask files used in the IRAF system.   Several
> software systems provide general capabilities for reading and writing data
> in this variable length array format, including the CFITSIO library
> (maintained by the HEASARC at NASA Goddard Space Flight Center) and IRAF.
> 
> Appendix B.1 is not an official part of the FITS Standard and is included in
> that document for informational purposes only.  We believe that the
> implementations described above have now fully demonstrated the usefulness
> and importance of the Variable Length Array Facility, and thus we hereby
> propose that Appendix B.1  be officially approved and incorporated onto the
> FITS Standard in its entirety.