[fitsbits] EXTEND/NEXTEND reality check

[Rob Seaman]

Steve Allen inquired:

> If there is anyone on whose equipment the process described by Bill  
> Pence is burdensome, speak now, otherwise don't reply.

I'm not sure we should be discussing burdensome demands placed on  
anybody's equipment, and you guys can do whatever you want with [N] 
EXTEND, but it behooves us to consider that even in a world of  
limitless resources some requirements pertain to serialization, not  
random access.

Not all FITS files are realized on hard media, random or otherwise -  
in particular, any number of applications must parse FITS received  
via the standard input or similar byte stream transport.  Certainly  
NOAO's Save-the-bits does.

The question isn't only whether you need to seek willy-nilly through  
a file, the question is whether it is efficient to seek in whatever  
manner for whatever application.  STB doesn't care since my mandate  
is to - well - save all the intervening bits.  And supporting some  
modest complexity MEF file on disk likely won't be a challenge since  
you only have to parse and skip a few dozen times.

On the other hand, perhaps there are applications relying on data  
files with hundreds or thousands of extensions that require  
idiosyncratic routing?  In that case, even on random access media [N] 
EXTEND likely doesn't do *enough* and one might be tempted to  
maintain a table of XTENSION keyword byte offsets in the primary (or  
some other) HDU.

The tape-based STB relied on constructing a bi-directional linked  
list of FITS header objects since a single large(ish) tape file might  
contain hundreds of small IR images packed as IMAGE extensions.  Even  
so, the problem was made much simpler by requiring an intermediate  
random access copy of the new MEF before writing it to tape.  This  
was deemed much more robust in an uncertain mountain environment than  
copying the serial input direct through to the serial output.  A  
similar logic applied to the CD-based STB which (of course) used the  
normal two step process of 1) creating ISO image on scratch disk, and  
2) mastering CD.

Rob