From dwells at nrao.edu  Fri Mar 12 17:50:23 2004
From: dwells at nrao.edu (Don Wells)
Date: Fri, 12 Mar 2004 17:50:23 -0500
Subject: [fitsmime] New MIME-types-for-FITS RFC draft
Message-ID: <16466.16047.710416.603679@polaris.cv.nrao.edu>

Dear Friends of FITS,

A new version of the RFC draft, dated 2004-03-12 (today), is at

    http://www.ucolick.org/~sla/fits/mime/

This version incorporates comments from the December 2003 review of
the previous draft.  Several new sections have been added to the
document, and a FITS header example has been added to the FITS
overview section.  Many minor wording changes have been made.

Comments, suggestions, corrections, proposed additions, etc, should be
posted to the <fitsbits> and/or <fitsmime> mailing lists, or can be
sent to the authors (sla at ucolick.org and dwells at nrao.edu).

Regards,
Don Wells
-- 
  Donald C. Wells             Scientist               dwells at nrao.edu
                    http://www.cv.nrao.edu/~dwells
  National Radio Astronomy Observatory                +1-434-296-0277
  520 Edgemont Road,   Charlottesville, Virginia       22903-2475 USA


From arots at head.cfa.harvard.edu  Wed Mar 17 17:49:58 2004
From: arots at head.cfa.harvard.edu (Arnold Rots)
Date: Wed, 17 Mar 2004 17:49:58 -0500 (EST)
Subject: [fitsmime] New MIME-types-for-FITS RFC draft
In-Reply-To: <16466.16047.710416.603679@polaris.cv.nrao.edu>
Message-ID: <200403172249.i2HMnwoa003270@xebec.cfa.harvard.edu>

Just a few fairly minor questions which probably reveal my great ignorance.

Concerning the issue of restricting image/fits to two non-degenerate
coordinate axes, are there any mimetypes in existence that deal with
multi-dimensional images?  We're not the only ones who work with three
and more dimensional images.  I would have imagined that there would
at least be one for 3-D.

There has been some back and forth on gzipping and identification of
tables.  I just wondered whether that would be an area where the
optional parameters might come in handy.
I must admit that I am not so sure about gzipping - that should be
indicated by its own mimetype, really.
But it might be useful if application/fits could provide some clue as
to the way the file is to be used/interpreted.
In particular, we mainly produce FITS files with an empty primary
where the first extension is the "principal" extension.  It might be
helpful to say: this FITS file consists of binary tables and you
really should first look in extension X - that's the principal, the
others are auxiliary.
HEASARC/OGIP adopted the concept of HDUNAMEi, a number of years ago,
in an attempt to categorize the tables that are in use.  I don't know
whether one would go in that direction - to give the client some idea
of what's coming and to decide whether it can handle it.

  - Arnold

--------------------------------------------------------------------------
Arnold H. Rots                                Chandra X-ray Science Center
Smithsonian Astrophysical Observatory                tel:  +1 617 496 7701
60 Garden Street, MS 67                              fax:  +1 617 495 7356
Cambridge, MA 02138                             arots at head.cfa.harvard.edu
USA                                     http://hea-www.harvard.edu/~arots/
--------------------------------------------------------------------------


From lucio at mi.iasf.cnr.it  Fri Mar 19 08:59:20 2004
From: lucio at mi.iasf.cnr.it (Lucio Chiappetti)
Date: Fri, 19 Mar 2004 14:59:20 +0100 (MET)
Subject: [fitsmime] New MIME-types-for-FITS RFC draft
In-Reply-To: <200403172249.i2HMnwoa003270@xebec.cfa.harvard.edu>
Message-ID: <Pine.OSF.4.30.0403181835430.26424-100000@poseidon.mi.iasf.cnr.it>

I post a few comments (I try not to repeat stuff already posted, but I'm
not sure if all comments apply to CHANGED FEATURES in the draft, or to
preexisting one, since they are not marked), preceded by a quick answer to
Arnold Rots.

On Wed, 17 Mar 2004, Arnold Rots wrote:

> There has been some back and forth on gzipping and identification of
> tables.

gzipping (if you really mean it, i.e. not other forms of possibly more
efficient compression INSIDE the data part of an HDU which have been
discussed in the past) has nothing to do with a new MIME type.

A combination of decent browser and httpd server should be able to handle
a .gz file in a straightforward way using some other HTTP header stuff
(can't remember why). Consider the following *working* example.

In my XMM LSS database, I make available to users some FITS files. In
order to force the browser to download them to disk (the default action
for an "unknown" MIME type), I've taken the liberty to declare them as
"application/fits" in advance of registration (I could have used
"application/x-anything") inserting in the .htaccess file at same position
in my directory tree a line

  AddType application/fits .fits

This applies to all subdirectories below. Now some of those contains .fits
files and some other .fits.gz.

Both are automagically retrieved and expanded on the fly when saving to
disk. I did not do anything in MY configuration. There is a line in the
main configuration file of the httpd server

  AddEncoding x-gzip gz tgz

which should be responsible of this behaviour. I can't remember if it was
part of the default Apache configuration, or of our local setup.


> But it might be useful if application/fits could provide some clue as
> to the way the file is to be used/interpreted.

I believe this goes together with your reference to "identification of
tables". On the latter I'll comment further below.

I have no experience whatsoever with MIME "optional parameters" or the way
they are used. I just note that the FITS file header contains already ALL
CLUES necessary to properly handle any FITS file ("properly handle" may
also mean "sorry this feature is not supported")


Now I supply a few sparse comments to the current text of the draft


(1) page 8, section 4.7

    is there a specific reason to list the observatories in the
    presented order ? Should we use a wavelength order, or some sort
    of chronological order ?

    One could add reference to various ESA missions. I'm sure that
    XMM-Newton uses FITS format throughout. I'm not sure about
    Integral but I think so. While the original IUE (joint NASA-ESA-SERC
    mission) archives somehow pre-date FITS (or at least did not use
    it originally) I'm pretty sure the latest archives do.

    In the context of such missions one could also quote non-ESA sites
    like the XMM SSC in Leicester or the Integral ISDC in Switzerland.

(2) page 10, section 5.1, sub "Applications which use this media type:"

    Again for completeness, I would add the XMM-Newton SAS
    (http://xmm.vilspa.esa.es/external/xmm_sw_cal/sas_frame.shtml and
    refs. therein)

    I'm a bit puzzled by the reference to IDL (which I use) or to
    Mathematica and MatLab (which I do not know). What's the point to
    quote general data handling environment-or-languages (for IDL the
    distinction is not obvious), specially when "support" to FITS is
    provided by add-on libraries.

    If one quotes IDL, why not quote Fortran and C as well ? :-)

(3) page 10-11, section 5.1, sub "Additional information:"

    Why the conformance to published standards is a SHOULD and not a
    MUST ?

    What about the reference to "XHDUs ... of the standard types ...
    or any other conforming type" ? Do we mean only future extensions
    endorsed by the IAU, or also not endorsed extensions ?

    More generally, should we say something about what the recipient
    software may or may not do with such a broad standard ? In particular
    has the recipient software the right to refuse serving some
    particular extensions or XHDUs or combinations thereof ?

(4) page 11, section 5.1, sub "Recommendations for application writers:"

    In the second example bullet, I criticize a bit the wording
    "which encodes an image as a series of photon events". An event list
    may "encode" as well and at the same time an image, a spectrum, a time
    profile !

(5) page 14, section 5.2, sub "Applications which use this media type:"

    Why here (unlike 5.1) astronomical and non-astronomical are listed
    separately ? (may be the same should be done in 5.1 as well)

    I suppose one could add things like the ESO RTD, or Skycat. I leave
    to our ESO colleagues to supply proper references !!

(6) page 15, section 5.2, sub "Additional information:"

    I'm a bit confused here. I can definitely imagine web users outside
    of the astronomical community to be interested in FITS (plain, 2-d)
    images, and therefore accept the existence of "image/fits" alongside
    with "application/fits".

    However I wonder if more unusual forms of "images" should not be
    discouraged (i.e. accepted ONLY as "application/fits".

    For instance I wonder about the statement in parenthesis in the
    first paragraph ("In rare cases it may be appropriate to describe a
    NULL image ..."). Is it misleading ? Should it be more explicitly
    discouraged (with a SHOULD NOT if not even a MUST NOT) ?

    I believe also that (further down pag. 16, "Recommendations for
    application writers:", para 5) the usage of NAXIS=1 images should
    be discouraged (personally I'd prefer a table to store such data,
    although I admit there are cases to use an image)

    On the contrary I do not see why NAXIS=3 images should be privileged
    with respect to a FITS file made from 1 or 0 PHDU and n XHDUs which
    are EXCLUSIVELY of "IMAGE" type.

(7) nowhere special

    There are two things which are not said anywhere, and I wonder
    whether they are worth saying.

    One is a requirement or recommendation or suggestion that, in case
    a viewer is uncapable of dealing with a FITS file "displaying" it
    (whatever display may mean ... a 2-d pixel image, or an HTML view
    of a table, or a plot) e.g. because it has a combination of HDUs
    or types or keywords which are not supported, it shall/should/may
    offer the option to Save to Disk.

    My inclination is that this option SHALL always be used, and also
    that is SHALL NEVER be forbidden.


    The second thing concerns tables. I am not advocating to define a
    further MIME type for "plain" tables (similarly to image/fits for
    would-be "plain" images). As we run in difficulty defining a "plain"
    image, so we would when defining "plain" tables (one with a null PHDU
    and a *single* TABLE or BINTABLE XHDU ?  one where all columns have
    "depth 1" ?)

    Nevertheless we could suggest a recommended way of displaying any
    tabular XHDU. This should be to render it textually, as an HTML table
    with the cells rendered according to the appropriate format
    (e.g. TDISPnn or alike ; n-depth column elements may pose some
    problems). Is this something we should worry about in the RFC, or
    do we leave everything to the developers ?

    I guess many of us have some CGI scripts rendering FITS tables
    into HTML pages (I have one for my own use which I developed
    independently after a conversation with Clive Page which has his own).
    I'm not sure how one does transform a CGI or external program into
    something integrated in the browser, or in a browser plugin (of
    course one can always define an external program as an external
    viewer in .mailcap or equivalent, but I guess that's too naive)

----------------------------------------------------------------------------
Lucio Chiappetti - IASF/CNR - via Bassini 15 - I-20133 Milano (Italy)
For more info : http://www.mi.iasf.cnr.it/~lucio/personal.html
----------------------------------------------------------------------------



From William.D.Pence at nasa.gov  Mon Mar 22 16:31:31 2004
From: William.D.Pence at nasa.gov (William Pence)
Date: Mon, 22 Mar 2004 16:31:31 -0500
Subject: [fitsmime] Re: New MIME-types-for-FITS RFC draft
Message-ID: <405F5B33.69A07597@nasa.gov>

In the interest of promoting more discussion, here are some highlights of
the latest FITS MIME draft proposal available at
        http://www.ucolick.org/~sla/fits/mime/
Note that since this is a formal Internet draft document, words like
"MUST", "REQUIRED", "SHALL", "SHOULD", and "MAY" have a precise
interpretation.  They  are defined in RFC-2119, but for convenience I've
attached the relevant text to the end of this message. 

For those who have not yet read it, this FITS proposal requests that 2 new
Internet MIME types be defined: "application/fits" and "image/fits".  A MIME
type is a sort of descriptive label for the data file that gets passed along
with it when it is downloaded;  Web browsers and other applications may use
the MIME type to decide how to display or process the file.

Section 5.1 describes the "application/fits" MIME type, and says that files
described with this media type "SHOULD conform" to the FITS standards.  In
an ideal world one might prefer to say "MUST conform", but the current
wording is perhaps more realistic, since it allows FITS files that may have
only minor transgressions from the Standard.

Section 5.2 describes the "image/fits" MIME type and says:

 - files SHOULD have a Primary HDU with positive integer values for
   the NAXIS and NAXISn keywords, and hence SHOULD contain at 
   least 1 pixel.  

 - In rare cases it may be appropriate to describe a 
   NULL image -- a dataless container for FITS keywords,
   with NAXIS=0 or NAXISn=0 -- as 'image/fits' but this usage is 
   discouraged because such files may confuse simple image viewer
   applications.

 - files MAY have one or more conforming extension HDUs following
   the primary HDU.

 - SHOULD be principally intended to communicate the single data
   array in the primary HDU.

 - One-dimensional arrays (NAXIS = 1) may be described as image/fits.

 - Three-dimensional data arrays (NAXIS = 3, and NAXIS1, NAXIS2,
   and NAXIS3 all greater than 1) may be described as image/fits.

 - Files with degenerate axes (i.e., with one or more NAXISn = 1)
   MAY be described as image/fits, but the first axes SHOULD be 
   non-degenerate, i.e., the degenerate axes should be the highest
   dimensions.   Writers of new applications that produce this
   type of file SHOULD consider using the WCSAXES keyword to
   express the dimensionality, instead of using degenerate axes.

 - Files with 4 or more non-degenerate axes SHOULD be described
   as application/fits, not image/fits, except in rare cases it
   MAY be appropriate to do so.

 - Mosaic images SHOULD NOT be described as image/fits.

 - Multi-exposure-frame (MEF) mosaic images MUST be declared
   as application/fits and not as image/fits.

 - Random groups FITS files MUST be described as application/fits
   and not as image/fits.

 - Any FITS file that can be described as image/fits could instead
   be described as application/fits, depending on context and
   intended usage.

---------------------------------------------------------------

The following terms are defined in RFC 2119, at:
  http://www.faqs.org/rfcs/rfc2119.html
  http://www.ietf.org/rfc/rfc2119.txt


1. MUST   This word, or the terms "REQUIRED" or "SHALL", mean that the
   definition is an absolute requirement of the specification.

2. MUST NOT   This phrase, or the phrase "SHALL NOT", mean that the
   definition is an absolute prohibition of the specification.

3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.

4. SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
   there may exist valid reasons in particular circumstances when the
   particular behavior is acceptable or even useful, but the full
   implications should be understood and the case carefully weighed
   before implementing any behavior described with this label.

5. MAY   This word, or the adjective "OPTIONAL", mean that an item is
   truly optional.  One vendor may choose to include the item because a
   particular marketplace requires it or because the vendor feels that
   it enhances the product while another vendor may omit the same item.
   An implementation which does not include a particular option MUST be
   prepared to interoperate with another implementation which does
   include the option, though perhaps with reduced functionality. In the
   same vein an implementation which does include a particular option
   MUST be prepared to interoperate with another implementation which
   does not include the option (except, of course, for the feature the
   option provides.)

6. Guidance in the use of these Imperatives

   Imperatives of the type defined in this memo must be used with care
   and sparingly.  In particular, they MUST only be used where it is
   actually required for interoperation or to limit behavior which has
   potential for causing harm (e.g., limiting retransmisssions)  For
   example, they must not be used to try to impose a particular method
   on implementors where the method is not required for
   interoperability.

-- 
____________________________________________________________________
Dr. William Pence                   William.D.Pence(nospam)@nasa.gov
NASA/GSFC Code 662         HEASARC         +1-301-286-4599 (voice)     
Greenbelt MD 20771                         +1-301-286-1684 (fax)


From Mark.Calabretta at atnf.CSIRO.AU  Sun Mar 21 20:14:43 2004
From: Mark.Calabretta at atnf.CSIRO.AU (Mark Calabretta)
Date: Mon, 22 Mar 2004 12:14:43 +1100
Subject: [fitsmime] New MIME-types-for-FITS RFC draft 
Message-ID: <200403220114.MAA19894@grus.atnf.CSIRO.AU>

Comments on the rfcFITS-20040312.txt:

1) As previously, my main comment/question relates to "Recommendations
   for application writers" for application/fits.

   What does it mean in practical terms to say that "An application
   intended to handle "application/fits" SHOULD be prepared to
   encounter..."?  Specifically, does any utility currently exist that
   can do something sensible with ANY general FITS file?  E.g. I know
   that 'fv' does a good job with BINTABLES, etc., but can it even list
   pixel values for a 999 dimensional image?  What about a random groups
   file?  Are there any others that even come close?  The statement

     "Complete interpretation of the meaning and intended use of the
      data in each of the HDUs typically requires the use of heuristics
      that attempt to ascertain which local conventions were used by the
      author of the FITS file."

   implies that a certain amount of black magic is required.

2) Do the lists of software packages on page 11 (and 15) really count as
   *applications* that can handle "application/fits" (or "image/fits")?
   When setting up a mailcap entry you need to specify the name of an
   executable, e.g. 'xv' or 'fv' into which you can feed the file.
   IRAF, AIPS, miriad, etc. do not fit the bill here, though certainly
   they do contain some particular applications that can process some
   specific types of FITS file.

3) The last paragraph on p17 should be merged into the first sentence of
   "Additional information" on p16 to make it clear from the outset that
   NAXIS is limited to 3 non-degenerate axes.

4) Is it reasonable for image/fits to preclude IMAGE extensions?  Perhaps
   a comment is in order.

Additional comments:

p4, Sect. 4.1: The term "lines" in the phrase "2880-byte blocks which
    hold 36 80-character lines" is undefined.  You could say "2880-byte
    blocks that are divided into 36 records of 80 bytes".

    ASCII blank (0x20) is not a "non-printing" character - printing
    characters range from 0x20 to 0x7E inclusive.  CR, LF, FF and TAB
    are better described as print control characters.  You could also
    mention NUL which causes a lot of confusion for some software.

p9, Sect. 4.7: ATCA archive data is in the form of RPFITS which
    decidedly is not FITS.  (HIPASS data is in FITS.)

Grammar: "which" is routinely used incorrectly as a relative pronoun in
    place of "that" (e.g. the first eleven occurences of "which" in the
    document should be "that", the twelfth is the first correct usage).

Mark Calabretta
ATNF




From hanisch at stsci.edu  Wed Mar 24 08:56:54 2004
From: hanisch at stsci.edu (Robert Hanisch)
Date: Wed, 24 Mar 2004 08:56:54 -0500
Subject: [fitsmime] Re: New MIME-types-for-FITS RFC draft
References: <405F5B33.69A07597@nasa.gov>
Message-ID: <008b01c411a7$dd75ec00$7deca782@stsci.edu>

Bill, Steve, Don,

  I finally had a chance to read the latest RFC draft.  Overall I think it
is an excellent job.  Mark and Lucio have made some good points, and I also
have some specific comments and suggestions for the document, more along the
lines of copyediting rather than major revisions.  How would you like me to
handle these?

  One thing I find a little strange is the way references are done.  Perhaps
this is standard RFC style, but it is jarring to come along to a reference
like [NOST] or [Remark] (the first because the acronym will be meaningless
to most readers of the document, the second because it is so generic).

Cheers,
Bob



From dwells at nrao.edu  Tue Mar 30 14:47:19 2004
From: dwells at nrao.edu (Don Wells)
Date: Tue, 30 Mar 2004 14:47:19 -0500
Subject: [fitsmime] 03-29 revision of MIME-types-for-FITS
Message-ID: <16489.52935.584791.820925@polaris.cv.nrao.edu>

Dear Friends-of-FITS,

Steve Allen writes:
 > On Mon 2004-03-29T17:53:39 -0500, Don Wells hath writ:
 > > I have attached new text, PS, PDF files.
 > > I have also attached PS & PDF of the diff wrt the 03-12 version.
 > 
 > I have published the elements sent in this message on the web pages.

See

    http://www.ucolick.org/~sla/fits/mime/

for my second revision of the MIME RFC draft.  I have made almost all
of the changes that were recommended by the reviewers of the 03-12
version.  I have noted some outstanding issues on the cover page.  I
expect that the diff listing will prove useful for this revision.

I thank the reviewers for their efforts!

Regards,
Don
-- 
  Donald C. Wells             Scientist               dwells at nrao.edu
                    http://www.cv.nrao.edu/~dwells
  National Radio Astronomy Observatory                +1-434-296-0277
  520 Edgemont Road,   Charlottesville, Virginia       22903-2475 USA