[fitsbits] XTENSION = 'FITS' proposal

[Perry Greenfield]

Perhaps this idea has been floated before but since Don Wells did

not specifically recall any such idea being discussed I thought

I would float it myself.

Perry Greenfield

***********************************

Proposal for a "FITS" Extension (XTENSION = 'FITS')

It would appear that a natural FITS extension type is an entire

FITS entity itself. By FITS 'entity' I mean what has traditionally

been regarded as a FITS file. As an extension, a FITS 'file' would

be placed in the data component of a FITS Header/Data Unit of 

an extension HDU. FITS files have blocking requirements

consistent with that of the data components (multiples

of 2880 bytes) so there is no problem embedding a file within the

primary or extension HDU. This extension mechanism is intrinsically

recursive. The embedded FITS entity itself could have extensions

containing other FITS entities and so on. 



XTENSION = 'FITS '

BITPIX = 8

NAXIS = 1

NAXIS1 = <size of FITS entity in bytes>

This proposal allows for a easy and straightforward way of

hierarchically grouping data sets and their attributes and

requires no new keywords (with the possible exception of

those mentioned below).

(The following suggestion is due to Rick White)

Since these embedded FITS entities must be by their nature of

known size (unlike the top level FITS file, particularly if

it is being written to sequentially as would happen with a tape

file), we can consider adding indexing information to the

header associated with the FITS entity. A simple approach

would be to use a series of index keywords to indicate the

locations of each of the FITS extensions, if any. Implicit

is that the primary has an index of 0. As an example, an

embedded FITS entity with 3 extensions could be indexed as

follows:

XIND1 = 2880 / byte offset of 1st extension

XIND2 = 1065600 / byte offset of 2nd extension

XIND3 = 2171520 / ...

...

This will allow for efficient access of FITS components

elminate the need to read all the headers to find the

offset of the N'th extension. Note that the XINDnnnn

keywords are optional, but if present, must be present

and correct for all extensions that exist (and must not

be present for extensions that do not exist). 

Other keyword names are possible, I just used this as an 

example. Preferably it should be short to allow for many

extensions.

ADVANTAGES:

- Hierarchical structure is manifest.

- Relatively easy to modify most FITS libraries to support it.

- Does not require a separate entity to describe hierarchical

structure, minimizing the possibility for inconsistent

information.

- Allows a simple mechanism for indexing extensions for

rapid random access.

- Allows a more complete mapping of XML to FITS (as well for

other hierarchical data formats).

DISADVANTAGES:

- Doesn't explicitly handle shared data entities that don't

fit cleanly into the hierarchical structure. As an example,

one may want to associate data (e.g., reference files or

calibration data) with data elements, but the data referred

to are not naturally part of the hierarchy. But this proposal

doesn't prohibit extension types that act as pointers to

other extensions or data entities outside of the FITS file.

In other words, although it doesn't provide this feature, it

in no way interferes with adding such a feature.

- Modification of structure generally requires rewriting complete

file. (But common to most other approaches also).