[fitsbits] start of Public Comment Period on compressed FITS image and tables

Wed Jun 24 11:07:38 EDT 2015

>
> Yes the implementation of this proposed addition to the standard
> is all there in CFITSIO, but that can't be used directly by
> non-C-friendly languages such as java, javascript, and who knows
> what future platforms might arise.  That means that for instance
> browser-based FITS image viewers which currently can display any
> legal FITS image would likely, following incorporation of this
> convention to the standard, find themselves unable to deal with
> some standard FITS image data if they are unable to afford

considerable extra implementation effort.

For the record, this is not strictly true. The JS9 project (DS9 in the
browser, see http://js9.si.edu) can read any FITS file (subject to browser
memory limitations, along with my coding skills), because we implement FITS
support by compiling cfitsio to optimized javascript using emscripten (
http://kripken.github.io/emscripten-site/). One should be able to drag and
drop any FITS data file onto the JS9 web site to display the image. In
fact, there is a lot of work going on in the Web world to make C available
inside the browser (cf. the latest developments at
http://www.2ality.com/2015/06/web-assembly.html). So I think the Web world
will be able to follow whatever FITS standards we agree on, so long as
there are C implementations to support those standards.

But as Mark points out, it's a different question whether one wants to make
the FITS standard so complicated that only a relatively few C/C++ library
implementations can claim completeness. One big reason we moved to
emscripten and cfitsio is that it was way too much work to implement tiled
compression in the javascript FITS library we wrote for initial JS9
implementation.

Eric

On Wed, Jun 24, 2015 at 6:44 AM, Mark Taylor <m.b.taylor at bristol.ac.uk>
wrote:

> Dear FITS,
>
> regarding the tiled image/table compression proposal, in the spirit
> of this clarification from Lucio:
>
> On Wed, 24 Jun 2015, Lucio Chiappetti wrote:
>
> > First of all I clarify that the purpose of the Public Comment Period is
> just
> > to assess the feeling of the community.
>
> I'd like to express my feeling that: it looks pretty complicated.
>
> One of the attractions of the FITS format (as opposed to formats
> such as HDF5 or HDS that are essentially defined by their data
> access libraries) is that access to the basic numeric data is very
> straightforward in implementation terms.  When I needed to
> write a data access library for BINTABLE from scratch I could
> do it without a great deal of effort.  Supporting tiled table
> compression would complicate the implementation task considerably.
> The same goes even more so (complication by an order of magnitude?)
> for tiled images, though those are less close to my current
> personal concerns.  That is particularly true if the implementation
> language does not have existing libraries for the various defined
> compression algorithms.
>
> The content of the proposed text seems to be reasonably clear,
> but three weeks is not enough time to attempt an implementation
> to check it does actually provide all the description required
> to implement these conventions.  The text looks like what it
> presumably is, a post-hoc codification of a series of experiments
> in compression (various different compression algorithms,
> dithering options), rather than a designed proposal for how to
> specify compression in a clean way.  Doing it that way is cheap
> for the implementation that served as the experimental testbed,
> but expensive for other implementations.
>
> Yes the implementation of this proposed addition to the standard
> is all there in CFITSIO, but that can't be used directly by
> non-C-friendly languages such as java, javascript, and who knows
> what future platforms might arise.  That means that for instance
> browser-based FITS image viewers which currently can display any
> legal FITS image would likely, following incorporation of this
> convention to the standard, find themselves unable to deal with
> some standard FITS image data if they are unable to afford
> considerable extra implementation effort.
>
> I am aware there are some committed advocates and strong arguments
> for use of this convention.  This message is not itself a call to block
> incorporation of this text in the standard.  But since it's a Public
> Comment Period, I wanted to make a public comment noting my
> reservations.
>
> Mark
>
> On Tue, 16 Jun 2015, Lucio Chiappetti wrote:
>
> > ANNOUNCEMENT:  START OF FORMAL PUBLIC COMMENT PERIOD
> >
> > This is to announce the official start of a 3-week formal Public Comment
> > Period on the incorporation of the Tiled Image Compression and Tiled
> Table
> > Compression conventions in the FITS Standard.
> >
> > This is part of a process to incorporate the most useful and widely used
> > registered conventions (which are valid FITS constructs) into the
> official
> > definition of the standard.
> >
> > Among these the two compression conventions benefit of a common handling.
> > Given their relative complexity they are better discussed first, before
> other
> > easier conventions.
> >
> > The proposed text consists
> >
> > - in the ADDITION of an entire new chapter (10)  to the FITS Standard
> >   Document which describes the two conventions in a common prescriptive
> >   framework.
> > - It also includes the ADDITION of a new non-prescriptive Appendix I,
> > - plus the addition of the necessary bibliographic references,
> >
> >   and has been prepared by a technical team including L.Chiappetti,
> >   W.Pence, A.Dobrzycki, R.A.Shaw and W.Thompson (main editor Dick Shaw).
> >
> > - If the proposal is approved also Appendix C will be updated listing the
> >   new keywords, and a section H.3 will be added to Appendix H describing
> >   the updates and the differences with the registered convention.
> >
> >   All the updates are shown in blue colour in their current context (with
> >   the exception of the NEW chapter 10 which is black)
> >
> > The proposed draft text is available at
> >
> http://sax.iasf-milano.inaf.it/~lucio/FITS/Conventions/compression-upd2.pdf
> >
> > Supporting material is provided in the FITS Convention Registry at the
> > following URLs
> > http://fits.gsfc.nasa.gov/registry/tilecompression.html
> > http://fits.gsfc.nasa.gov/registry/tiletablecompression.html
> >
> > Considering that the convention(s) have been in use since several years,
> are
> > legal FITS, were discussed on FITSBITS when the conventions were entered
> in
> > the Registry and therefore their usage is well proven (also for what
> > interoperability is concerned), the Public Comment Period is reduced to 3
> > weeks.
> >
> > Also the review by FITS Working Group Executive can be speeded up and
> handled
> > in parallel or quickly after the conclusion of the Public Comment Period.
> >
> > Please review the text carefully and post any comments, criticisms, or
> > suggestions on the FITSBITS mailing list (not on iauwfg or elsewhere)
> > ==================================================================
> >
> > The Public Comment Period starts today 16 June 2015 and will last
> formally for
> > 3 weeks until July 6
> >
> > ==================================================================
> Background
> > information on the FITS approval process
> >
> > Under the "Rules and Procedures" of the IAU FITS Working Group,
> > http://fits.gsfc.nasa.gov/iaufwg/iaufwg_rules.html, the first step in
> the
> > official approval process of any FITS proposal will be a formal Public
> Comment
> > Period to take place on the FITSBITS mailing list. After that the IAU
> FITS
> > Working Group Executive will review the results. Following that the IAU
> FITS
> > Working Group will then conduct a final vote to approve or disapprove the
> > proposal.
> >
> > --
> > Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy)
> > Current IAU FWG chair
> >
> > _______________________________________________
> > fitsbits mailing list
> > fitsbits at listmgr.nrao.edu
> > https://listmgr.nrao.edu/mailman/listinfo/fitsbits
> >
> >
>
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20150624/7288c104/attachment-0001.html>