[fitsbits] New DUMP FITS extension

[LC's NoSpam Newsreading account]

On Thu, 16 Aug 2007, William Pence wrote:

> I recently learned that the FITS data from Solar Optical Telescope on 
> the Hinode satellite [...] contains a DUMP extension with
> 
> BITPIX  =                8
> NAXIS   =                1
> NAXIS1  =              300
> PCOUNT  =                0
> GCOUNT  =                1

... which actually resembles a 1-d image ...

> My understanding is that this extension contains the image header 
> packet from the satellite telemetry.

... if one wishes to encapsulate raw telemetry in a FITS file, the idea 
of using an ad-hoc extension is sound ...


> 1. Is this how the DUMP extension was originally envisioned to be 
>    implemented?


WHO KNOWS ? Who did suggest the original name ?

I think it was a mistake on our part to accept the registration of a 
name without an implementation proposal, without keeping track of who 
might have drafted such proposal, and without setting an expiry date.

We should define a more formal procedure for extensions which either (or 
both) :

 - allows pre-registration of a name for a fixed time (6 months ?) after
   which the name is retired if no implementation proposal has come

 - allows direct registration of an existing implementation (probably
   contextually with a registration of an associated "convention") like
   for FOREIGN


> An alternate scheme, as used in the FOREIGN extension type, would be:
> 
> BITPIX  =                8
> NAXIS   =                0
> PCOUNT  =              300
> GCOUNT  =                1

> Is one of these 2 forms preferable over the other?

The second one WOULD BE preferable for me for the packing of an 
arbitrary blob of data into a FITS file.

However if the arbitrary data is organized as a fixed record length file 
(anybody remembering IBM JCL ?) it might make sense using NAXIS1 and 
NAXIS2

But it is now too late to ask this question (see further below) !


> 2.  How does the DUMP extension as used in the Hinode data differ from an
> IMAGE or FOREIGN extension?  

For me the difference is that an IMAGE extension is an image (a 2-d or 
n-d array, not necessarily a spatial image), a FOREIGN extension is the 
encapsulation of some specific FILE structure (with some relation to 
actual "live" operating systems e.g. for things like file naming, or 
even directory trees, soft links etc (which may become obsolescent in 
the future).  A DUMP will be just a blob of data with no implications as 
files at OS level (users will have freedom to use it in their OS).

A further possibility I thought of in the recent past would be a MIME 
extension, similar to FOREIGN but with no "live" OS dependency, but just 
encapsulating a mime file with its content type and other header (which 
presumably is more general, less OS-related and more long lived).


In a later mail you wrote :

> The Hinode satellite project is now publicly distributing FITS data 
> files with this DUMP extension.  They apparently saw the DUMP 
> extension name in the list of proposed extensions in the appendix to 
> the FITS standard and decided to use that extension name for their 
> telemetry data.  Even though there is no specification of what the 
> DUMP extension keywords should look like

Again, it was our fault that no such specification did exist, and still 
DUMP was in the namespace !

The only thing we can do now is asking the Hinode people to provide a 
documentation of their *convention*, enter it in the convention 
registry, and ideally "assign" to them the registered DUMP extension 
(like FOREIGN is "assigned" to NOAO).

This assumes nobody else is using another DUMP variant. If so one could 
have different (hopefully not inconsistent) conventions : a Hinode DUMP 
convention, an Erewhon DUMP convention ...


On Thu, 16 Aug 2007, Doug Tody wrote in two other messages:

> Since the convention is in use for actual data, we should probably
> follow our existing practice of merely documenting existing
> conventions, and describe the convention/extension as used by Hinode.

I agree with Doug in the terms specified above (the Hinode people should 
provide the documentation).

> My personal view on the more general question is that Image should
> only be used for image data, and it is a trick to stuff arbitrary
> binary blobs in something called an Image.

I also agree with this. It is not "clean" to store a blob as an 1xn 
image nor as an 1xn binary table.

But what is then an image ?

> One answer would be that if it is array data of any dimension it is 
> reasonable to store it as an "image"

I also agree here. I see absolutely no difficulty in storing e.g. a 
chi-square grid vs spectral index and NH as an image. 

I have also seen people storing 1-d spectra or histograms as images, or 
even collections of 1-d arrays (typically spectra) packed in an image 
(e.g. gross, bkg, net and quality flag vs wavelength stored as 4 rows in 
an image). Although in mostt cases my preference would be to use a 
binary table, both usages are legal and probably nearly equally 
efficient.


Lucio Chiappetti

-- 
----------------------------------------------------------------------
nospam at mi.iasf.cnr.it is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.