[fitsbits] start of Public Comment Period on compressed FITS image and tables

[Mark Taylor]

Dear FITS,

regarding the tiled image/table compression proposal, in the spirit
of this clarification from Lucio:

On Wed, 24 Jun 2015, Lucio Chiappetti wrote:

> First of all I clarify that the purpose of the Public Comment Period is just
> to assess the feeling of the community.

I'd like to express my feeling that: it looks pretty complicated.

One of the attractions of the FITS format (as opposed to formats
such as HDF5 or HDS that are essentially defined by their data
access libraries) is that access to the basic numeric data is very
straightforward in implementation terms.  When I needed to
write a data access library for BINTABLE from scratch I could
do it without a great deal of effort.  Supporting tiled table
compression would complicate the implementation task considerably.
The same goes even more so (complication by an order of magnitude?)
for tiled images, though those are less close to my current
personal concerns.  That is particularly true if the implementation
language does not have existing libraries for the various defined
compression algorithms.

The content of the proposed text seems to be reasonably clear,
but three weeks is not enough time to attempt an implementation
to check it does actually provide all the description required
to implement these conventions.  The text looks like what it
presumably is, a post-hoc codification of a series of experiments
in compression (various different compression algorithms,
dithering options), rather than a designed proposal for how to
specify compression in a clean way.  Doing it that way is cheap
for the implementation that served as the experimental testbed,
but expensive for other implementations.

Yes the implementation of this proposed addition to the standard
is all there in CFITSIO, but that can't be used directly by
non-C-friendly languages such as java, javascript, and who knows
what future platforms might arise.  That means that for instance
browser-based FITS image viewers which currently can display any
legal FITS image would likely, following incorporation of this
convention to the standard, find themselves unable to deal with
some standard FITS image data if they are unable to afford
considerable extra implementation effort.

I am aware there are some committed advocates and strong arguments
for use of this convention.  This message is not itself a call to block
incorporation of this text in the standard.  But since it's a Public
Comment Period, I wanted to make a public comment noting my
reservations.

Mark

On Tue, 16 Jun 2015, Lucio Chiappetti wrote:

> ANNOUNCEMENT:  START OF FORMAL PUBLIC COMMENT PERIOD
> 
> This is to announce the official start of a 3-week formal Public Comment
> Period on the incorporation of the Tiled Image Compression and Tiled Table
> Compression conventions in the FITS Standard.
> 
> This is part of a process to incorporate the most useful and widely used
> registered conventions (which are valid FITS constructs) into the official
> definition of the standard.
> 
> Among these the two compression conventions benefit of a common handling.
> Given their relative complexity they are better discussed first, before other
> easier conventions.
> 
> The proposed text consists
> 
> - in the ADDITION of an entire new chapter (10)  to the FITS Standard
>   Document which describes the two conventions in a common prescriptive
>   framework.
> - It also includes the ADDITION of a new non-prescriptive Appendix I,
> - plus the addition of the necessary bibliographic references,
> 
>   and has been prepared by a technical team including L.Chiappetti,
>   W.Pence, A.Dobrzycki, R.A.Shaw and W.Thompson (main editor Dick Shaw).
> 
> - If the proposal is approved also Appendix C will be updated listing the
>   new keywords, and a section H.3 will be added to Appendix H describing
>   the updates and the differences with the registered convention.
> 
>   All the updates are shown in blue colour in their current context (with
>   the exception of the NEW chapter 10 which is black)
> 
> The proposed draft text is available at 
> http://sax.iasf-milano.inaf.it/~lucio/FITS/Conventions/compression-upd2.pdf
> 
> Supporting material is provided in the FITS Convention Registry at the
> following URLs
> http://fits.gsfc.nasa.gov/registry/tilecompression.html
> http://fits.gsfc.nasa.gov/registry/tiletablecompression.html
> 
> Considering that the convention(s) have been in use since several years, are
> legal FITS, were discussed on FITSBITS when the conventions were entered in
> the Registry and therefore their usage is well proven (also for what
> interoperability is concerned), the Public Comment Period is reduced to 3
> weeks.
> 
> Also the review by FITS Working Group Executive can be speeded up and handled
> in parallel or quickly after the conclusion of the Public Comment Period.
> 
> Please review the text carefully and post any comments, criticisms, or
> suggestions on the FITSBITS mailing list (not on iauwfg or elsewhere)
> ==================================================================
> 
> The Public Comment Period starts today 16 June 2015 and will last formally for
> 3 weeks until July 6
> 
> ================================================================== Background
> information on the FITS approval process
> 
> Under the "Rules and Procedures" of the IAU FITS Working Group, 
> http://fits.gsfc.nasa.gov/iaufwg/iaufwg_rules.html, the first step in the 
> official approval process of any FITS proposal will be a formal Public Comment
> Period to take place on the FITSBITS mailing list. After that the IAU FITS
> Working Group Executive will review the results. Following that the IAU FITS
> Working Group will then conduct a final vote to approve or disapprove the
> proposal.
> 
> -- 
> Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy)
> Current IAU FWG chair
> 
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
> 
> 

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/