[fitsbits] further reopening of Public Comment Period on the CONTINUE convention]

[Rob Seaman]

Howdy,

Walter Jaffe wrote:

> ...or putting these strings in an extension as an ascii or binary table.

I have no comment on the CONTINUE convention, per se. Having recently changed institutions I have inherited (in the best sense of the word) a new archival data set with its own choices, long set. One of those choices was to archive a rich diversity of data products as text files, or to be more general, as non-FITS files. It is unremarkable that Planetary Science would adopt non-astronomical standards, but it is also unremarkable that Astronomy would use more than one standard itself. Both fields use XML and JSON, JPEG and PNG, etc and so forth.

That said, the Catalina Sky Survey (my new project) had recently adopted FITS tile compression before I came onboard, and as we commission new, larger cameras, we will be adding various keywords (mostly numeric and boolean). CSS had been archiving SExtractor output in both ASCII and FITS formats (different thresholds, different purposes) and I see no reason to change this choice just for ideological reasons. On the other hand, the nature of the Survey is such that many interesting time domain queries can be done against a MySQL DB of FITS headers, and the output of joins against astrometry or asteroid ephemerides might well find a useful home as FITS BINTABLEs.

FITS has a long future. Non-FITS obviously also. Recent discussions at ADASS have been predicated on the assumption that FITS is near death and that something better will come along. Well, it hasn’t come along yet, and there are copious reasons to be skeptical of rapid uptake should it do so. The entire notion would be rejected out of hand by the many, many projects in astronomy and related fields – or for that matter, unrelated fields, as with the Vatican manuscripts projects – who have made a long, expensive, and evolving investment in FITS.

Rather, new features (or established conventions) like CONTINUE should be evaluated with an eye to whether they make the current FITS stronger, more flexible, more “complete”. (And don’t forget our bumper sticker: “Once FITS, always FITS”.)

Simultaneously, we should consider what FITS should look like decades and centuries hence. The likeliest “something better” that will succeed at replacing current FITS is FITS itself, as with numerous projects (including both my old and new employers) replacing IMAGE arrays with tile-compressed BINTABLEs.

It would be straightforward to copy the current FITS keyword schema (if it can be called such) into a FITS BINTABLE. It would be straightforward to add long strings and many of the features of other metadata-related conventions to the resulting BINTABLE schema. It would be straightforward to layer a BINTABLE access layer on top of the current library routines for header keyword access. And it would be straightforward to come up with a few rules of usage such as when BINTABLE keywords should take precedence over current headers. Such a metadata BINTABLE is easily updateable, sortable, queriable, etc. It can be joined in various useful ways against data BINTABLEs. It is compressible, or alternately, can be extracted back into current header structures (with loss of new features) for backwards compatibility. Current FITS files can be reformatted with minimal fuss.

BINTABLEs populated with header keywords are already legal FITS. Projects that find legacy header keywords problematic could reformat their data legally, and with good data hygiene, before their morning coffee. It would be desirable, but not required, to settle on a common schema.

Reports of the death of FITS are greatly exaggerated. FITS CONTINUEs.

Rob Seaman
Catalina Sky Survey
Lunar and Planetary Laboratory
University of Arizona