Version 100-2.0 of NASA FITS Standard released
Bob Hanisch
hanisch at stsci.edu
Mon Apr 12 13:55:03 EDT 1999
12 April 1999
To the FITS Community:
I am pleased to announce that NASA's Office of Standards and Technology
(NOST) Accreditation Panel has reviewed and endorsed the revised FITS standard.
The new version of the standard, 100-2.0, is dated 29 March 1999 and is
available in Postscript, compressed (gzip) Postscript, and PDF formats at
http://fits.gsfc.nasa.gov/.
On behalf of the FITS Technical Panel, I would like to thank everyone who
submitted comments concerning the draft standard. These comments helped the
Technical Panel identify ambiguities in the document which (we hope) we have
been able to repair.
Attached below is a statement from Don Wells, chair of the IAU FITS
Working Group and member of the NOST Accreditation Panel. I encourage
everyone with an interest in FITS to read it carefully, as Don has done
an extremely thorough job of reviewing the new standard. He explains what
the Technical Panel has done and why, and why he believes that the regional
FITS Committees and IAU FITS Working Group should now move quickly to adopt
the new standard also.
Finally, as someone who has worked on the FITS Standard for many years
now, I note with some pride that FITS has just passed its 20th anniversary.
The adoption of V100-2.0 by NOST is a fitting birthday present for us all!
Please circulate this message to others in the astronomy community who
may have an interest in FITS.
Thanks,
Bob Hanisch
Chair, NASA FITS Technical Panel
Data Systems Division
Space Telescope Science Institute
-----
Notes on the NOST FITS Standard & Process
Don Wells <dwells at nrao.edu>
FITS community representative on NOST Accreditation Panel
Chair of IAU FITS Working Group [IAU-FWG]
1999-03-19
SUMMARY: The NOST [NASA Office of Standards and Technology]
process which produced the NOST_100-1.2 draft standard appears to
me to have been conducted properly, and to have produced a result
which is acceptable. I therefore vote 'yes' on the question of
whether this document should be accredited and become the FITS
standard for NASA. Furthermore, I conclude that NOST_100-1.2
contains no 'uncorrectable errors'. I therefore recommend that,
if no such errors are uncovered by the FITS committees, the
committees should approve NOST_100-1.2 as the new official FITS
standard of the International Astronomical Union [IAU].
-=- Introduction -=-
I was asked by NOST to act as the FITS community representative on
their Accreditation Panel [NOST-AP], in order to review the *process*
by which the NOST_100-1.2 document was produced. NOST specified that
"..the Accreditation Panel should ensure that,
- the proposed standard is of high general quality (well
organized, readable),
- the target community was adequately notified of the draft
standard and was afforded adequate chances to comment on the
draft,
- and comments were appropriately handled. In addressing
comments, consensus need not be reached, but the Technical Panel
needs to address all issues raised. There should not be
inordinate influence by any special interest group.."
In the discussion which follows I will reference two documents which
are available at http://ssdoo.gsfc.nasa.gov/nost/apanel, "FITS
Accreditation Panel Support Materials" [FAPSM, 1999-03-04, 69 pages]
and "Definition of FITS, version 1.2" [DoF1.2, 1999-01-22, 96 pages].
After addressing the three accreditation criteria listed above, I will
discuss the relationship of the NOST Technical Panel [NOST-TP] to the
International Astronomical Union [IAU] FITS Working Group [IAU-FWG]
and the special problem of 'uncorrectable errors'.
-=- Is NOST_100-1.2 of high general quality? -=-
My answer to this question is an emphatic 'yes!'. However, I also
reviewed the comments to see whether the FITS community is critical of
the general character of the document. For example, comment#1
[FAPSM:5.2,#1,p34] asks that deprecated features be highlighted in
bold font. Comment#19 [p37-38] wants the Random Groups discussion
reorganized. Comment#127 [p.58] says that a different form of
Backus-Naur notation should be used in Appendix A of DoF1.2. These are
rather minor criticisms of the style of the document. In each case
the NOST-TP declined to make the style change requested, and I
consider the justifications given for these decisions to be reasonable
enough.
-=- Were notification and opportunities to comment adequate? -=-
The April 1998 announcement [FAPSM:4.0,p7] that NOST_100-1.2 was
available for review was posted to a number of Email exploders and
newsgroups (see the 'To' line of the message header on p.7). In
particular, it was posted to newsgroup sci.astro.fits and its
associated exploder 'fitsbits' which are the official places on the
Internet where discussions of FITS occur. The announcement specified
that comments should be posted publicly, not sent privately to the
NOST-TP. This resulted in public followups by the community to the
comments, and followups to the followups. In effect, the opportunity
to comment was publicly announced repeatedly over a three month period.
Some controversial issues produced extensive threads of public
discussion, giving the Technical Panel a good sampling of the opinions
of the FITS community. Four notable cases can be seen in FAPSM: [1]
the proposal that OBJECT strings should conform to IAU Designations
recommendations [FAPSM:5.2,#77-91,p51-53], [2] the proposal that FITS
should support 64-bit integers [FAPSM:5.2,#142,p.61], [3] the proposal
that FITS should support unsigned integers [FAPSM:5.2,#143,p.61] and
[4] the strong demand that the NOST-TP delete or soften its
deprecation of Random Groups [FAPSM:5.2,#144,p.61]. In all four cases
the NOST-TP refused to take the action requested. I have reviewed the
four responses, and consider them to be reasonable enough. The
extensive public discussion of these controversial (and perennial)
issues is an exemplary aspect of the *process* of creation of this
standard.
Comments were received from a diverse group of people: 11 Europeans, 6
non-NASA US national center people, 9 US university people and 6
NASA-related people. Many of the 11 Europeans are members of the
European FITS committee, of course. Designers associated with most of
the major software packages of astronomy are in the group that offered
comments on DoF1.2. I conclude that the comments received by NOST-TP
were reasonably representative of opinion in the worldwide FITS
community.
-=- Were comments properly handled? -=-
In my opinion, the NOST-TP was quite responsive to the technical
comments they received. In many of the comments and responses there is
plenty of room for reasonable people to differ; a good example of such
differences is the manner of specifying the rules for 4-digit and
2-digit years in date strings [FAPSM:5.2,#71,p49-50].
The NOST-TP generally refused to establish new *policy* for FITS,
declaring policy issues to be the prerogative of IAU-FWG and 'out of
scope' for the Panel. The NOST-TP did make decisions on a variety of
technical (non-policy) issues which were previously ambiguous in the
FITS standards. The question of which issues are technical and which
are policy (political) matters is certainly a judgement call, and
reasonable people can differ in these matters. Items [1], [2] and [3]
cited in the previous section are examples where the NOST-TP ruled
that proposed changes were out-of-scope. In these three cases I think
the NOST-TP made the only ruling which they could properly make.
For example, consider item [1] above, in which I (Don Wells) asked the
NOST-TP to specify that value strings for OBJECT keywords should be in
conformance with recommendations of the IAU TG on Designations. The
NOST-TP refused. I not only accept this ruling against me, I applaud
it, because we should not attempt to short-circuit the political
process for making such decisions. In fact, I predicted that NOST-TP
would rule against me, but submitted my request anyway as a means to
get the issue before the community, to start the necessary process of
creating consensus which might lead, ultimately, to an IAU-FWG
agreement on the issue. This particular issue is still being debated
in the FITS community; only two weeks ago an astronomer asked about
OBJECT rules and got this reply from a member of the NOST-TP:
"The NOST panel decided it was out of scope of its charter to
introduce new restrictions or rules for the value of the OBJECT
keyword or to define a new reserved keyword such as IAUDESIG,
without the prior general agreement of the FITS community."
[W.Pence sci.astro.fits 1999-03-10]
Has there been 'inordinate influence by any special interest group' in
the responses of the NOST-TP to the comments they received? Some
members of the worldwide FITS community might worry that requirements
or desires of specific NASA flight missions or facilities might exert
an undue influence on the deliberations of the NOST-TP, due to the
large number of NASA-related people on the NOST-TP (see
[DoF1.2:Preface,p.'i'] for current and past members of NOST-TP). I
have reviewed this question, and have concluded that these people
represent multiple distinct missions with differing goals and
operating in different types of radiation (UV, optical, IR), and that
probably no single private agenda could dominate the processes of the
NOST-TP. Both science and computing people are represented on the
current and past NOST panels. I have also concluded that no single
large software system is over-represented on NOST-TP; for example, the
current panel includes the designer of AIPS (radio synthesis imaging),
the chair of the IRAF (optical) Technical Working Group and the
designer of the FITSIO package (high energy). Each of these people has
institutional and project ties to international partners, and can be
expected to consider the international aspects of their decisions. I
conclude that any FITS feature decision which is acceptable to this
set of people and the rest of the NOST-TP is likely to be acceptable
to the whole worldwide FITS community.
-=- Accreditation -=-
I conclude that the three criteria which NOST specified for the
NOST-AP to review have been satisfied, and therefore
+------------+
| I vote YES |
+------------+
on the question of whether DoF1.2 should advance from draft status to
become the NASA FITS standard.
In the remainder of this memorandum, I will review other criteria
which will be relevant in the process by which DoF1.2 might become the
official FITS standard of the IAU.
-=- NOST Technical Panel: history, membership, role -=-
The history of the versions of NOST_100-x.y produced by the NOST-TP is
tabulated at [DoF1.2:J.0,p91]. Five different 1.x versions of the
document have been publicly released over the past six years. These
documents have been in regular use throughout the FITS community as
the de facto FITS standards, in spite of the fact that the IAU-FWG has
not yet formally endorsed any of them. DoF1.2 differs from the 1.1
version by a set of incremental changes which are tabulated in
[FAPSM:6.2.I,pp62-64]. Some of the changes are merely rearrangements
(such as moving the BINTABLE definition from an unofficial appendix to
the main body of the formal standard). The point of mentioning this
history is to emphasize that DoF1.2 is the latest version of a
sequence of draft documents which have, for nearly a decade now, been
in regular use and regular review by the FITS community as the de
facto FITS standards. DoF1.2 therefore entered this final formal
review process with the knowledge that most parts of it have already
been reviewed and effectively approved by its end-user community many
times over many years.
Regarding the membership of NOST-TP, a key fact to note is that a
number of the members are also members of the North American
(AAS-WGAS) FITS Committee and/or the IAU-FWG. This is not a conflict
of interest if you view the NOST-TP and even the regional FITS
committees as acting somewhat like committees and subcommittees in the
typical legislative process: members of the full body are appointed to
develop proposals for the full body to consider.
What is the role of the NOST-TP with respect to the IAU-FWG and the
three regional FITS committees? The IAU-FWG never voted to charter the
NOST-TP activity, and so the NOST-TP has no official status from the
IAU-FWG point of view. However, it is a fact that ten years ago I and
other members of the FITS committees recommended that the NASA
Astrophysics Division fund the FITS standardization effort as a part
of the FITS Support Office activity under NOST. We understood that it
would not be practical to negotiate the myriad of details of an
official standards document by the usual FITS negotiation processes.
At that time (and still today) there was no other agency which could
fund such an effort. My opinion, as current Chair of IAU-FWG, is that
the NOST-TP has acted as an ad hoc task force which has done the hard
work of preparing a clean version of the FITS standards for the
IAU-FWG. The Chair of the NOST-TP recently commented of the role of
NOST-TP:
"My view has always been that the Technical Panel was put into
place to look at the options and select the best solution, rather
than simply turn the discussion loose in the community."
[Bob Hanisch, private communication 1999-03-10]
I consider the diversity and the technical skills of the NOST-TP to be
about as good as could be achieved in practice. If a similar but
different set of individuals had been picked for NOST-TP, I expect
that they would have produced a similar, but not identical result. I
therefore see no reason to revisit any of the many issues which the
NOST-TP has debated and settled in their many work sessions.
If it is accredited, the NOST FITS standard will be submitted to the
three regional FITS committees, and will thereby become an official
document in the IAU-FWG process.
+--------------------------------------------------------------+
| I recommend that the regional FITS committees and IAU-FWG |
| accept the DoF1.2 document as-is, without further changes, |
| assuming no 'uncorrectable errors' are discovered in review. |
+--------------------------------------------------------------+
-=- On 'Uncorrectable Errors' -=-
The FITS community has agreed that changes to FITS will always be
backwards compatible (often described as 'once FITS, always FITS').
The policy is explicitly specified in NOST_100-1.2 (DoF1.2:9.0,p.51).
The policy implies that FITS standards committees must be especially
careful when drafting new FITS standards to avoid making decisions
which they might later wish they had not made, because it will be
difficult if not impossible to correct those errors.
In particular, the committees should be cautious whenever they are
specifying rules which will preclude whole paths which the future
evolution of FITS might want to take. Furthermore, they should be
cautious whenever they are specifying rules beyond the minimum set
needed for interchange, because of the risk of precluding innovative
uses of 'holes' in the ruleset. FITS must always leave room for growth
and innovation. Therefore, I have reviewed the decisions of the
NOST-TP to check for cases where decisions have been made by the
NOST-TP which cannot be easily corrected by future IAU-FWG actions or
where decisions have been made which might limit growth or innovation.
The NOST-TP was sensitive to the future evolution of FITS; for
example, in [FAPSM:5.2,#26,p39] they note that 'special records' are a
key 'escape hatch' for the format and in [FAPSM:5.2,#28,p40] they
deliberately leave room for a definition which they do not specify.
I have reviewed the changes from DoF1.1 (1995-09) to DoF1.2
[FAPSM:6.2.I,p62]. In section 5.2, NOST-TP decided that keyword values
cannot be an array of values. This is an instance in which NOST-TP has
explicitly deleted functionality which was specified in the original
FITS Agreement of 1979. Eric Greisen, who negotiated that Agreement
with me in 1979, is a member of NOST-TP, and he tells me that the
NOST-TP believed that the functionality has never been used, and so no
harm is being done. I now believe that multivalued keywords are a bad
design idea, on grounds that they are an example of 'repeating groups'
in database design, and therefore I agree with this decision by
NOST-TP.
I reviewed the changes from DoF1.2draft to DoF1.2 [FAPSM:6.2.II,p64],
but saw no 'uncorrectable error' issues.
I reviewed my list of possible evolutionary paths for FITS (see
section 5 of http://www.cv.nrao.edu/adass/adassVI/wellsd.html), but
found nothing precluded by DoF1.2 rules.
I reviewed the list of differences from the IAU FITS papers
[DoF1.2:E,p71] but found no issues not discussed elsewhere in this
memorandum.
I am aware of three instances in which NOST-TP has definitely extended
the FITS standards, not just clarified or codified them:
* Section 5.1.2.3 [DoF1.2:p16] allows keyword value fields to be
blank, i.e. 'undefined'. This concept has been invented by NOST-TP.
* Section 5.2.1 [DoF1.2:p16-17] says: "..no length limit less than 68
is implied for character-valued keywords" (previously, FITS rules
said that some keywords should carry their information in 8 chars).
* Section 8.1.4 [DoF1.2:p38] says regarding the data arrays of TABLE
extensions: "There may be characters in a table row that are not
included in any field". I have searched, but have found no rule
which limits what those characters may be, other than the opening
sentence of 8.1.3 which says they must be "ASCII characters"
(defined on p.7 as 7-bit ASCII, which includes the 'ASCII Control'
column of Table G.1 on p.82). The TABLE extension paper
[Astron. Astrophys. Suppl. Ser. 73, 365-372 (1988)] makes it
absolutely clear that the TABLE data matrix is to be *printable*
ASCII codes, but the NOST-TP rule will allow *non-printable* codes
such as CR, LF and HT into characters which are not a part of table
entry fields.
In my opinion these three actions by NOST-TP are good things
('removing unneeded restrictions' as noted in [FAPSM:5.2,#135,p60]),
and the community should endorse them implicitly by adopting DoF1.2. I
believe they will not cause any uncorrectable error problems.
In the cases where NOST-TP declared that some proposed change would be
'out of scope' for NOST-TP, and declined to act, there is nothing to
prevent members of the FITS community submitting proposals to their
regional FITS committees, persuading the other FITS committees to
endorse them and then persuading IAU-FWG to adopt them. Future
versions of NOST's DoF will then be modified to incorporate such
rulings by IAU-FWG.
The case of the 'deprecation' of the Random Groups format is somewhat
troubling; it produced a major thread of discussion during the comment
period [FAPSM:5.2,#144,p61]. This problem arose because ten years ago,
when the BINTABLE extension became part of the FITS standards, there
was a general desire to replace Random Groups with BINTABLE in all
production applications, and it was generally believed that this would
naturally happen during the 90s. The NOST-TP therefore deprecated
Random Groups in early versions of DoF. It is still possible that
BINTABLE will replace Random Groups in production, but it can be
argued that the transition has failed for various sociological and
political reasons, and that Random Groups will be used forevermore.
The people who take the latter position strongly object to the
deprecation of Random Groups in DoF1.2, while the NOST-TP takes the
position that the issue was decided many years ago and they don't want
to change back. This is a question of exactly how the definition of
'deprecated' is worded in Section 3 [p8] of DoF1.2, and of exactly how
the deprecation is worded in the introductory paragraph of Section 7
[p31]. In any case, radio and optical interferometry instrument
systems will continue to write Random Groups as needed, all future
versions of the FITS standard will be obliged to continue reproducing
the rules of random groups as an official part of the FITS standard
*forever* and the relevant datasystems will be obliged to support the
reading of Random Groups *forever*, so this wording dispute has no
effect on astronomy research. In particular, it does not involve an
uncorrectable error.
+----------------------------------------------------------------+
| I conclude that decisions made by NOST-TP have not produced |
| any 'uncorrectable errors' which would be risky for IAU-FWG to |
| endorse by adopting DoF1.2 as the new IAU FITS standard. |
+----------------------------------------------------------------+
-=- Trivial Typographical Errorss -=-
* remove 'to' from 'Deprecated' definition on p.8 of DoF1.2.
* reword the last sentence of item 33 on p.78 of DoF1.2.
[Note: the above errors have been corrected, and the cover page, etc.,
have been updated to reflect the new version number and change in status
from draft to final form. RJH, 29 March 1999.]
More information about the fitsbits
mailing list