[fitsbits] ADASS FITS BoF Minutes

[William Pence]

Minutes of the ADASS FITS Birds-of-a-Feather Session
----------------------------------------------------

The FITS Birds-of-a-Feather (BoF) session, held each year in conjunction
with the ADASS conference, took place in Pasadena, California on 24 October
2004.  The purpose of the FITS BoF sessions is to encourage members of the
FITS user community to discuss current issues related to the FITS data
format.  About 70 people attended this 90 minute session which was led by
the chairman of the IAU FITS Working Group, William Pence.  Here are some
brief minutes of the main items that were discussed:


1.  The meeting opened with a note of congratulations to Don Wells (in
absentia) on his recent retirement from NRAO and appreciation for all his
contributions to the FITS community over the past 3 decades.  Even though
Don is officially retired, it is good to know that he will be continuing his
involvement in FITS-related activities.

2.  Recent changes to the FITS Committees

The IAU FITS Working Group (serving under IAU Commission 5) was reorganized
at the end of last year and now has 22 members:  11 from North America, 9
from Europe, and 1 each from Japan and Australia. See
http://fits.gsfc.nasa.gov/iaufwg/iaufwg.html for the complete list.

A new Regional FITS committee was also formed to represent FITS users in
Australia and New Zealand.  This new committee will be chaired by Mark
Calabretta, and it joins the 3 previously existing regional committees for
North America, Europe, and Japan which all provide advice and guidance to
the the IAU FITS Working Group.

3.  New FITS Approval Procedures

The rules by which the FITS committees approve new FITS proposals have been
clarified and are documented in detail at
http://fits.gsfc.nasa.gov/iaufwg/iaufwg_rules.html.  The main steps in the
review process can be summarized as:

    1.  A formal Public Comment Period is held on the FITSBITS mail list
        (and the related sci.astro.fits newsgroup).

    2.  The regional FITS committees vote on the proposal
        with a 2/3 majority in each committee required for approval.

    3.  The IAU FITS Working Group votes with a 3/4 majority required
        for approval.

One of the comments made during the discussion of this topic was that these
formal procedures should also include the traditional requirement of
'interoperability testing' of the proposal, at least in cases where it is
relevant.

4.  Status of the FITS MIME type proposal

Steve Allen reported that the proposal for 2 new Internet MIME types
("image/fits" and "application/fits"), which had previously been approved by
the FITS committees, is now in the final stages of approval by the relevant
Internet standards committee.  [Final approval was subsequently granted on
1-Nov-2004].  Congratulations and thanks are due to Steve Allen and Don
Wells for drafting and shepherding this proposal through the long approval
process!

5.  Status of the WCS Paper III on Spectral Coordinate Systems

Several of the authors of this paper were in attendance (Eric Greisen, Frank
Valdes, and Steve Allen) and led a brief discussion. This paper is now in
the Public Comment Period phase of the formal approval process, so everyone
was encouraged to carefully review the paper and submit comments on FITSBITS
by 12 November.  Much of the discussion concerned what, if anything, should
be required in the way of interoperability testing before the FITS
committees vote on the proposal. It was noted that Mark Calabretta's wcslib
software library supports most of the WCS conventions defined in the paper
but there is little in the way of other software to enable independent
testing. There was no clear consensus on what interoperability testing
should be performed, but there was general agreement that a set of FITS
files that use the various WCS conventions should be created and made
available for testing purposes. Pence agreed to collect any suitable files
and make them available from the FITS Support Office web site
(fits.gsfc.nasa.gov).

6.  The SIP convention for representing image distortions

David Shupe and Richard Hook gave a presentation on the "SIP" convention for
representing image distortions that is being used in the images produced by
the Spitzer Science Center and may possibly be adopted for some HST images.
When the SIP convention was developed several years ago it more or less
conformed to the recommendations given in an early draft of the WCS paper on
representations of distortions in FITS world coordinate systems (the 4th in
the series of WCS papers).  Since then however the draft WCS paper has
evolved and recommends a convention that differs from the SIP convention.
There was some discussion on how this divergence could or should be
rectified.  No conclusions were reached, but Shupe agree to continue this
discussion on the fitswcs email list in the near future.

7. Status of the TDIMn and Variable Length Array Proposals

These 2 unofficial FITS conventions are currently defined in appendix B of
the FITS Standard.  A proposal to make them official by incorporating them
into the Standard itself is now in the Public Comment Period phase of the
formal review process.  Everyone was encouraged to carefully review the
proposals and submit any comments by 12 November.  Francois Ochsenbein
pointed out that the variable length array convention makes it more
difficult to stream FITS format files from one application to another
because in general the entire 'heap' of data (which could be very large)
must be transmitted before the receiving application can process a single
row of the table.  This is especially relevant for VO applications where
steaming of data is the norm. It was agreed that some cautionary statements
about the possible adverse consequences of using the variable length array
convention should be added to the proposal that is currently in review.

8.  The FITS User's Guide

The FITS User's Guide, last updated in 1997, is now in great need of
revision to reflect the many changes that have taken place of the past 7
years.  It currently contains many obsolete sections that would be confusing
or misleading to new FITS users.  Pence has begun updating some of the
sections,  and with the help of a few other contributors, hopes to have a
new draft available in early 2005.

9.  64-bit Integer support in FITS

Whether to add support in FITS for 64-bit integer images and table columns
has been a controversial issue for many years.  Technically, this would only
require a few relatively simple changes to the wording in the FITS Standard:
64-bit integer images would be denoted by "BITPIX = 64", and binary table
columns would have a "TFORM" type of 'K'.  In addition, a new 64-bit array
descriptor with a TFORM type of 'Q' could be defined (analogous to the
32-bit 'P' array descriptor.  The controversy about this has mainly centered
on the impact that this change would have on software developers and users.

Recently it has become apparent that there are now few practical obstacles
to prevent developers from writing software (or updating existing software)
to support 64-bit integers using most programming languages and computer
platforms in common use today.  Even Fortran-77 compilers widely support the
integer*8 data type.  There ensued a  general discussion on the need (or
lack of need) for 64-bit integers by FITS data providers and on the possible
impacts that this change to the FITS Standard would have.  Many people
expressed support for this proposal in general, but also recommended that
some cautionary statements be added to urge users to not abuse or
unnecessarily use this new feature if it is approved.

At the end of the discussion a straw vote was taken to gauge the amount of
support for this issue.  Out of the 60 to 70 people in attendance, about 80%
voted in favor of  seriously considering this proposal now, and only 1
person voted for deferring consideration to some unspecified later date.
The remaining 20% or so did not vote one way or the other.   Given this
strong support, it is likely that this issue will be brought up for further
discussion on the FITSBITS email list and possible consideration by the FITS
committees in the near future.

10.  A Registry of FITS Conventions

Pence briefly outlined a few ideas about establishing an official registry
of FITS conventions.  The general idea is that this would be a way to
recognize and document existing important keyword conventions that are in
use by at least some subset of the astronomical community.  Details on how
one could propose a convention to the registry, who would need to approve
it, and what information would need to be supplied to go into the registry
remain to be determined.  There was also some discussion on how formal or
detailed the registry should be, and ideas of how XML schema could be used,
as in the VO registry of services.  This will certainly be a topic of more
discussion on FITSBITS in the coming year.

11.  FITS and the VO

The last topic of the BoF session was on the role of FITS and the FITS
committees in the new Virtual Observatory era.  No one offered any
suggestions for technical changes to the FITS Standard that would make the
FITS format more supportive of VO applications, so the discussion quickly
moved on to the differences between the FITS and VO committee structures and
the standards approval procedures.   Some voiced the opinion that sooner or
later, there should be some rationalization or merging of the various
committees.  It was pointed out that IAU Commission 5 is planning to create
a VO oversight committee (at the same organizational level as the existing
FITS Working Group) which might help to improve communications between
developers in the 2 communities. If there are to be any major changes to
current FITS committee structure, it is unlikely to occur before the next
IAU General Assembly meeting in 2006.