[fitsbits] EXTENDing FITS

[Rob Seaman]

(I see Doug has also replied, arguing a middle path.)

Thierry Forveille wrote:

> Or one could take the (in my view more logical) view that a MEF  
> file is the wrong choice when strong structural information needs  
> to be included :-)

Presumably we would then not want to absolutely deprecate special  
records :-)

Astronomical data is both large and often composed of large chunks.   
A modest sized single data object of almost any nature can be  
squeezed into a bintable with a purpose tailored schema.  One such  
example would be a tile-compressed image.  A large but fine-grained  
data object, e.g., a catalog, also wants something equivalent to a  
bintable.

It is the chunkiness (of FITS and of the underlying data) that  
demands an MEF-style mapping.  Whatever the nature of each chunk -  
array or table or something else - there are usage scenarios  
requiring simple access to at both the level of the individual chunks  
as well as the entire file.  MEF is one solution that provides this.   
A tightly coupled global tabular structure would be significantly  
more unwieldy, e.g., Unix-style software tools would no longer work,  
etc.

If we don't think MEFs are sufficient to the task - or that they  
could be modified to serve the purpose - but we do want to retain  
FITS as our foundation - then something other than stuffing  
everything into a single monolithic bintable will be needed.  Special  
records would be the only place to go.

Rather, it seems to me that both FITS and MEF are worthy - if  
idiosyncratic - platforms for the next round of evolution of  
astronomical data.  Our formats need to scale to data past the 2 GB  
divide, e.g., LSST's 3.2 gigapixel camera - yet astronomers will want  
to use familiar image processing tools (IRAF, IDL, etc.) on portions  
and the whole of the focal plane.  It is immaterial if these looming  
data cows choose some alternate internal format, since unless FITS is  
going to be retired community-wide, it will have to retain  
significant utility with new data products anyway.  (Or perhaps IVOA  
has a master plan of sinking FITS beneath the waves?)

Rather, I think something along the lines of "Encapsulated FITS"  
would combine both a semantically mature skeleton (fine grained per- 
HDU data model with versioning/name-spacing) with coarse grained  
structuring - perhaps a new INDEX extension type to be appended as  
needed to complex MEF files.  Per-HDU semantics plus per-file  
indexing.  There are options such as embedding XML structures to make  
everybody happy (or maybe just as well - equally unhappy :-)

Meta-discussions as with EXTEND or duplicate keywords or keyword  
continuation reflect a need for coherently extending our vision of  
FITS.  The status quo likely won't scale to projects currently under  
development.  The alternatives therefore are to use something other  
than FITS, or to extend FITS.  Whichever it is, we should get started  
soonish.

Rob