[fitsbits] FW: FITS future

[Robert J. Hanisch]

I share the general reluctance to make significant changes to FITS at this point in time.  As others have already noted, the reasons people are starting to use other formats has less to do with the constraints of FITS headers and more to do with file sizes and encoding complex relationships between data objects.

I read the Thomas et al. poster, plus a longer version of that work that Brian sent to me.  I suggested to Brian that the criticisms of FITS need to be turned around into a set of goals or requirements for a successor.  I think it's worth sharing that message with the rest of you.

Bob

From: "Robert J. Hanisch" <hanisch at stsci.edu<mailto:hanisch at stsci.edu>>
Date: Tuesday, 8 October 2013 3:05 PM
To: "bthomas at noao.edu<mailto:bthomas at noao.edu>" <bthomas at noao.edu<mailto:bthomas at noao.edu>>
Subject: FITS future

Hi Brian.  OK, I've read the two anti-FITS papers…points made.  I think the longer one can be put to a better purpose by rewriting it as a set of goals or requirements for a FITS successor.  It's mostly all there, just in the form of negatives about FITS rather than desired attributes for a new format.

I take away these points as goals:

o  Support for versioning
o  Support for the encoding of and/or mapping to external data models, including support for name spaces
-  For core data types:  image, spectrum, time series, table
-  For errors, masks, coverage maps, etc.
o  Support for full ASCII and Unicode
o  Provision of robust validation tools (easier if industry standards like XML or JSON are part of the solution)
o  Rich representations of world coordinate systems, accommodating complex distortions
o  Standard representations of provenance and data processing histories
o  Standard metrics and representations of data quality
o  Flexible and extensible representations of physical units
o  Flexible and extensible metadata encoding
o  Robust mechanisms for describing relationships among data objects
o  Ability to use either big- or little-endian byte order (or perhaps some other low-level representation?)
o  Support for real-time streaming
o  Support for large, distributed data (where a "file" may be striped across multiple storage systems)

There is a lot going on in the IVOA context addressing these concerns, where the "data format" ends up being metadata encoded in XML (VOTable) and FITS remains a serialization for pixel values.

Anyway, I think it would be ever so much more helpful at this point to talk in terms of goals, and just take it out of the context of FITS.  One can then ask what extant formats, if any, support the above goals?  What industry standards can be utilized to support these goals?

Also, I think there are some important goals missing:

o  A fully documented format that can be understood, over time, through its documentation rather than an API.
o  A format that serves equally well in real-time analysis and processing and long-term archiving.

cheers,
Bob

(Please feel free to forward this to your co-authors.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20131025/2e76e716/attachment.html>