[fitsbits] TDIMn

[Tom McGlynn]

Preben Grosbol wrote:
>
...
> 
> With respect to B.1 (variable length arrays), I have some reservations.
> This proposal requires a full implementation of the HEAP and the P column
> format.  Technically, the B.1 proposal is good and I would endorse it as such.
> The issue is that (as least to my taste) FITS files which conform to the
> standard should be readable by most readers.  The success of FITS is
> to some extend based on this, namely that one having a standard FITS
> file does not need to worry - it is, by default, readable by almost all
> systems.
> 
> If B.1 is included into the standard I am concerned that a significant number
> of systems will be unable to read and access such data.   It is not just a
> matter of if HEAPs can be accessed by software libraries but more if full
> data processing systems can access such data as if they had been in a
> 'normal' table column.  Beside leaving B.1 as it is, there may the following
> options to include it into the standard:
>   1) Adding an clear, explicit preample which states that users cannot assume
>       the HEAP concept to be implemented by all readers and therefore should
>       be cautious in using it.
>  2) Verify that most readers actually already implements the HEAP which
>       would satisfy my concern.
> 

The table at http://fits.gsfc.nasa.gov/fits_libraries.html suggests that
most FITS libraries that support binary tables do support variable length
arrays.  Preben's comments could have been made about binary tables
(or even more strongly ASCII tables) which are not supported by substantial
numbers of FITS readers.  Indeed, if one were to look at the distinct software
implementations of FITS readers I suspect that the vast majority are simple
readers that handle only primary FITS images.  (I've written many such readers
over the past couple of decades for a variety of special projects).
However when we look at the libraries that are being used in the development
of new major software, then I think that it is clear that both binary
tables and variable length arrays are widely supported and used.

That said I'm not sure I agree with Preben's premise that putting something
in the standard implies a mandate that FITS readers should be able to
handle it.  Certainly the many FITS readers that only handle images will continue
to ignore binary tables regardless of whether they have variable length records,
and FITS readers that wish to ignore heap data in binary tables
should feel free to do so.

In a similar vein, the new WCS papers suggest some quite sophisticated and complex
ways that coordinate information can be encoded.  I expect that the vast
majority of readers will support only elements of the WCS that are necessary
to the kinds of science they are intended to support rather than providing
explicit support for all of the WCS features.

	Regards,
	Tom McGlynn



> On B.3, it would be good to know who is actually using it?  One should
> only include, into the standard, features which are used by a broad community.
> 
> Preben Grosbol
>