[fitsbits] New DUMP FITS extension

[Doug Tody]

Hi Bill -

Since the convention is in use for actual data, we should probably
follow our existing practice of merely documenting existing
conventions, and describe the convention/extension as used by Hinode.
This description could be very brief.

My personal view on the more general question is that Image should
only be used for image data, and it is a trick to stuff arbitrary
binary blobs in something called an Image.  If the content of the
extension is a defined data structure with some type which is foreign
to FITS, well that is what the FOREIGN convention was created for.
This tells you that the object is not an image but a more generic
container which can hold anything, and provides some standard keywords
for identifying the contents.

If someone really wants to stuff a binary blob in an image, my
preference would be to set NAXIS=0 and use PCOUNT to express the
data size, as this makes it clear that the contents are not array
data, e.g., not something one might consider plotting or otherwise
displaying.  In other words, there is no image array, and the null
image is followed by a block of binary data.  We can do the same
thing if there is no image data but only a header.  This works to
some extent, but it is still bothersome to call it an "image".

	- Doug


On Thu, 16 Aug 2007, William Pence wrote:

> Doug,
> 
> The Hinode satellite project is now publicly distributing FITS data 
> files with this DUMP extension.  They apparently saw the DUMP extension 
> name in the list of proposed extensions in the appendix to the FITS 
> standard and decided to use that extension name for their telemetry 
> data.  Even though there is no specification of what the DUMP extension 
> keywords should look like, they assumed the keywords would essentially 
> be the same as in an image extension with BITPIX = 8 and NAXIS = 1.
> 
> The question is what do we (the FITS community in general, and the 
> IAUFWG in particular) do now?  Should we draft a definition document 
> describing the DUMP extension (as is currently being done for the 
> FOREIGN extension type)?  Do we want to go further and endorse the DUMP 
> extension for use in other data sets distributed by other projects?  Or, 
> would we rather discourage the use of the DUMP (or any other new 
> extension type) if the data can easily be contained in an existing 
> extension type? Personally, I would rather not see a proliferation of 
> new extension types that are structurally identical to the existing 
> IMAGE extension.
> 
> Bill
> 
> Doug Tody wrote:
> > Hi Bill -
> > 
> > I'm not sure I understand: this DUMP extension was proposed, but has never
> > been specified or implemented?  If this is the case, it doesn't actually
> > exist, and we should be able to ignore it.  (Also it is functionally
> > redundant with the FOREIGN extension).
> > 
> > 	- Doug
> > 
> > 
> > On Thu, 16 Aug 2007, William Pence wrote:
> > 
> >> One of the proposed changes to the FITS Standard is a complete rewrite 
> >> of the Appendix F (previously Appendix I) which lists the reserved FITS 
> >> extension type names (i.e., the value of the XTENSION keyword). One of 
> >> the registered extension types listed in that appendix is as follows:
> >>
> >>   'DUMP    ' - Suggested extension name for storing a stream of
> >>    binary data (such as a telemetry stream) in a FITS file.  This
> >>    extension type was never implemented, but the FOREIGN extension
> >>    type serves a similar purpose.
> >>
> >> This definition may now need to be updated because I recently learned 
> >> that the FITS data from Solar Optical Telescope on the Hinode satellite 
> >> (http://solar-b.nao.ac.jp/index_e.shtml), launched in late 2006, 
> >> contains a DUMP extension with the following minimal set of header keywords:
> >>
> >> XTENSION= 'DUMP    '
> >> BITPIX  =                8
> >> NAXIS   =                1
> >> NAXIS1  =              300
> >> PCOUNT  =                0
> >> GCOUNT  =                1
> >> END
> >>
> >> My understanding is that this extension contains the image header packet 
> >> from the satellite telemetry.  The NAXIS1 value may vary, but otherwise 
> >> all the DUMP extensions look the same.
> >>
> >> The IAU FITS Working Group is charged with maintaining the list of all 
> >> registered FITS extension types, and at least at some level, is 
> >> responsible for defining how each type is supposed to be used.  A number 
> >> of questions regarding the DUMP extension would need to be resolved 
> >> before writing such a definition document:
> >>
> >> 1. Is this how the DUMP extension was originally envisioned to be 
> >> implemented?  An alternate scheme, as used in the FOREIGN extension 
> >> type, would be:
> >>
> >> XTENSION= 'DUMP    '
> >> BITPIX  =                8
> >> NAXIS   =                0
> >> PCOUNT  =              300
> >> GCOUNT  =                1
> >> END
> >>
> >> Is one of these 2 forms preferable over the other?
> >>
> >> 2.  How does the DUMP extension as used in the Hinode data differ from 
> >> an IMAGE or FOREIGN extension?  What are the fundamental differences, if 
> >> any, between the DUMP and IMAGE (or FOREIGN) extensions?  What criteria 
> >> should data providers use in deciding which extension type to use?
> >>
> >> These questions are particularly relevant now because the new draft FITS 
> >> Standard in section 3.4.1 states:  "New extension types should be 
> >> created only when the organization of the information is such that it 
> >> cannot be handled by one of the existing extension types."    (This 
> >> requirement comes from the original "Generalized Extensions" FITS paper 
> >> published in 1987).  Given that an IMAGE extension presumably could be 
> >> used to store the same data, does the DUMP extension satisfy this 
> >> requirement?
> >>
> >> 3.  Can the DUMP extension have a BITPIX value other than 8, and a NAXIS 
> >> value other than 1?  Must PCOUNT always = 0 and GCOUNT always = 1?   If 
> >> any of these keywords can have different values, how does this affect 
> >> the interpretation of the data?
> >>
> >> Presumably these questions, and more, should be answered in a short 
> >> definition document that specifies exactly how the use DUMP extension 
> >> should be used.   At the moment however, it is not clear to me that 
> >> there is any consensus on the answer to these questions.
> 
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits
>