FITS/HDF (Was: FITS WCS with Terrestrial Satellite Data)

[Louis Giglio]

In article <77fbdu$483 at owl.le.ac.uk>, Clive Page <cgp at nospam.le.ac.uk>
writes:

>In article <77e9ip$7lh at post.gsfc.nasa.gov>,
>Louis Giglio <giglio at hades.gsfc.nasa.gov> wrote:
>>  (We've used HDF for both of these purposes in
>>the past, but have largely dropped it for a host of reasons.  But I
>>digress...)
>
>I know you were digressing (and I'm not expert enough to help by answering
>your main questions) but I would be interested in knowing some of these
>reasons - from time to time the limitations of FITS seem almost
>overwhelming (after all it was designed  many computer generations ago)
>and in disucssions  someone is sure to say "why don't we switch to
>something more modern and flexible like CDF/netCDF/HDF/HDS...  ".  I can
>think of some of the reasons, but would be interested all the same.


My general reasons for using FITS are:

     * FITS is defined in terms of physical file layout rather than
       a software interface.

     * The simpler structure of FITS files makes them easier to
       understand, read, and write.

     * It's very easy to ignore FITS headers and read data directly if
       necessary.

     * New FITS features/conventions are defined by community consensus
       after a great deal of haggling.  (I suspect this is a
       disadvantage sometimes.)

     * FITS has been around for a long time and is well documented.


Now for the general reasons we've steered away from HDF:

     * HDF is defined in terms of a software interface rather than
       a physical file layout.

     * The HDF format is in fact a collection of several different
       file formats.  (Different versions have different formats.)

     * Poor I/O performance when updating existing HDF files.

     * Extremely difficult to ignore internal HDF information and read
       data directly if necessary.

     * Any HDF operations require the use of the NCSA HDF library.
       If this library cannot be built on a particular platform,
       there's no way to access HDF files.

     * New HDF features/conventions defined by relatively small group
       of people.  (This I'm sure is an advantage sometimes.)

     * Portions of old HDF files are sometimes not readable by newer
       releases of the HDF library.  It is quite possible to have
       information stored in an HDF file that is unreadable by the
       current HDF library.

These very general reasons weren't sufficient to sway me completely.
What really did it was the more mundane, practical issues we run into
on a day to day basis.  These include:

1) We sometimes share data with investigators in relatively poor
countries.  It does no good to give them HDF files (which absolutely
require the use of the memory hungry HDF library) when all they have
is an old IBM AT with 320K of RAM.  Many FITS utilities, however,
will work with such small amounts of memory, and with FITS it's very
easy to read the files as binary when necessary.

2) I've created FITS files from C shell scripts using only standard
UNIX commands such as "cat" and "dd".  I also rolled my own REXX
procedures to write FITS files.  I don't believe it is practical to
write HDF files this way.

3) A huge amount of freely available software has been written to
manipulate FITS files, much of which is quite general and useful
regardless of the type of data contained within the file.  We're able
to use much of this software, saving us from having to write a lot of
code from scratch.

(Contrast this to a few of the folks I work with, who have contracted
a programmer for several _years_ to write an HDF SDS display program
with some image processing bells and whistles [histograms, etc.]).

4) Many of the FITS-related issues ironed out by astronomers are
applicable to other disciplines.  Examples of this include date
and time formats, checksum conventions, and the WCS proposal.

5) A surprising number of my coworkers handle HDF files by converting
them into native binary before working with the data.  Frequently
confusion arises later because all of the metadata was stripped in
the process.  I've never seen them do this with a FITS file.

Don't get me wrong, FITS definitely drags along its own set of ugly
features and peculiarities.  And on paper HDF looks great: groups,
support for arbitrary stuctures, etc.  But when it comes down to
routine, everyday sharing and archiving of data in a simple, efficient
manner, FITS provides considerable advantages in my opinion.

Cheers,
Louis