[fitsbits] Question(s) regarding development of proprietary FITS manipulation software. . .

[LC's NoSpam Newsreading account]

On Fri, 24 Aug 2007, Michael Williams wrote:

> valid FITS files all formatted completely differently.  [...] the most 
> common "formats".  Can someone send me (offline or otherwise) some 
> sample data with accompanying comments/markup regarding what each 
> piece of data means?

You can do this better by yourself. On one hand grab a copy of the FITS 
standard from http://fits.gsfc.nasa.gov/fits_home.html (there is even a 
draft of the most recent revision, which does not change really the 
format, but has maybe clearer explanation), on the other way consult the 
archives of the major ground observatories (ESO, NOAO, NRAO) and of the 
major space missions (NASA and ESA).

> FITS compliant header followed by. . .

All headers I know are "compliant" ... what you should get from the FITS 
standard is the list of : mandatory keywords which appear in ANY header 
of a given sub-type (*) ; reserved keywords which, when they appear, 
must be used as prescribed in the standard ; other keywords which 
essentially can be considered of commentary or specific nature (cfr. 
e.g. the convention registry 
http://fits.gsfc.nasa.gov/fits_conventions.html)

I am not sure what you intend to do. I suppose some sort of new general 
purpose tool. Probably you should concentrate on interpreting the 
mandatory keywords, and some of the reserved keywords, and ignore all 
the rest (even if not of commentary nature, might be useful only for 
some project specific piece of software).

(*) by sub-type I mean here the various kinds of extensions :

 a) primary HDU with image data and IMAGE extensions
 b) binary table extensions
 c) ascii table extensions
 d) primary HDU with random groups
 e) anything else

Let me first say that (e) is extremely unfrequent (that would mean 
conforming extensions not part of the standard, which at present is only 
the FOREIGN extension, which is a way of wrapping non-FITS stuff in a 
FITS file used at some places), so you'd just ignore it.

I'll then add that (d) is an old format, now deprecated, and used 
essentially only by radio astronomers. I guess it is unlikely you'd 
encounter it unless you look in the archives of radio observatories.

Now for your classification

> 1) binary data
> 3) array's of numbers

I'm not really sure where you draw the difference here. I suppose one 
case may be my "a" and another my "b", or perhaps "c". See further 
below.

> 2) non-compliant header-like text

I am not sure what you mean by this too. Whether you mean "c" (ASCII 
tables ... look printable text but not in header format), or whether you 
mean some kind of headers (ESO HIERARCH convention perhaps ? looks odd 
but is compliant !).

Essentially what I say is that it is my a/b/c classification which makes 
sense (and unfortunately is not enough).

(a) the case of images is possibly the simplest. It may occur in the
    primary HDU, or in XTENSION='IMAGE' extensions. The data which
    follows is a binary n-dimensional array  A(i,j,...). In most cases
    a 2-d image A(i,j), in some cases a 1-d array A(i), in some other a
    datacube etc.

    In most cases of the 2-d images they are sky images (and data cubes
    are stacks of sky images), but can be anything (also in some cases
    they are actual detector images, they may be something different,
    consider e.g. a multi-object spectrometer).

    The default action here is to display a 2-d image with some false
    colour mapping. On the X and Y axes just put "pixels".

    If the header has WCS keywords, these would allow to map pixel
    coordinates to sky coordinates OR OTHER "world" (physical) 
    coordinates. Rather complex to handle.

    Example handling program is ds9.

(b) the case of binary tables is nowadays probably the next most diffuse
    but comes in a large number of varieties. First of all it always
    comes as an extension, so you have a dataless PHDU and an extension
    header. 

    The data are a binary array with a number of rows, and a number of
    columns of DIFFERENT data types. To further complicate things,
    a column may have a depth (see the TFORMn keywords). To further
    complicate things, this depth way be constant, or variable (in
    which case the cell contains pointers to data in the HEAP, see
    the P and Q TFORMn)

    Here I'd say that the typical "generic" action (example handling 
    program is fv) is to produce a tabular printout of the content
    (optionally a plot of one column vs another ... but this can be
    chosen only interactively). With some special action for large
    or variable depth cells.

    Some reserved optional kwds can be used to annotate the column
    with a title or plot label, with physical units, with a recommended
    display format ...

    This format is widely used for lots of different things : a
    catalogue of sources, or any database ; a spectrum (but sometimes
    spectra are formatted as 1-d images) ; a time series ; a photon
    list or even more complex things like the RMF (Response Matrix 
    Function) of an high energy detector.

    ... and of course a file may contain many different extensions.
    You should consider each one as independent.

(c) the case of ascii tables is nowadays encountered mainly in older
    (or old-style) archives, typically of catalogues. The data is
    written in fixed-length ASCII record (so it appears printable).

    It is simpler than binary tables (columns have always depth one)
    but for the rest it shall be displayed similarly.

-- 
----------------------------------------------------------------------
nospam at mi.iasf.cnr.it is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.