[fitsbits] reopening of Public Comment Period on the compression conventions

Mon Jan 11 11:44:14 EST 2016

In the interval since the last discussion, I've had a bit of an epiphany 
about this proposal.  I still believe that there should be two distinct 
proposals, but changed the basis set I'm thinking in.

Consider the name 'tiled image compression'.  This says we're doing  two 
different things:

First we're defining a new way of representing an image: as a set of 
tiles. Second we're compressing those tiles.

I would suggest that it would be much cleaner to have this as two 
separate proposals:

  1. "Tiled images"  defines a new representation of images as tiles 
within a binary table. These tiles represented in this standard are not 
compressed.  They are just fields in a binary table.

2. "Table compression" defines a general standard for compression of 
binary tables.  In this context the tiled images are simply one kind of 
binary table.

A file that uses both of these conventions would look like our current 
compressed tile images.

This seems much cleaner to me than the current approach.  Each proposal 
does one thing completely.

I don't know how difficult it would be to recast the existing proposal 
into these terms, but it seems liike it should be possible to do this in 
a way which is fully consistent with the existing datasets.

Were this approach adopted, then it would raise some questions as to 
whether we need to have  non-compressed implementations of tiled tables, 
but would conversely mean that we have lots of implementations of the 
table compression code since all of the existing compressed images would 
qualify.

Absent such a restructuring, I'd reiterate my comments from June: 
Supporting image tiling seems important, but it's less clear that there 
much push from the community for supporting non-image compressed binary 
tables and I would prefer to leave that out until have a few 
instantiated use-cases.

    Tom McGlynn

Mark Taylor wrote:
> Dear FITS,
>
> I have no objection to tiled compression for image data, which
> I understand to have been in wide and successful use.
>
> However, I would like to reiterate the concerns I aired in the
> previous public comment period about tiled compression for *table*
> data (the proposed Section 10.3).  In response to my query
> about where this convention has been used to date, only one example
> was given; Bill Pence cited its use by the FACT Cherenkov Telescope
> project (see message
> https://listmgr.nrao.edu/pipermail/fitsbits/2015-July/002803.html).
> As Bill noted, in that case a custom compression algorithm was used,
> which as I understand it means that although they are using substantial
> parts of the proposed compression convention, they are not using
> it in quite its proposed form, so that a FITS reader implemented
> according to the current proposal would not recover the FACT data.
>
> My feeling is that this level of usage of the tiled table compression
> convention does not justify its addition into the standard,
> especially in view of the proposal's complexity, and hence the
> implementation burden it imposes upon implementors (e.g. me) who
> wish to be able to parse all standard-compliant FITS table data.
>
> Tom McGlynn made a similar point:
>
> On Thu, 25 Jun 2015, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
>
>> I've almost no insight into table compression.  Given that no one seems to be
>> using this convention, my suggestion would be that it's premature to add to
>> the standard.
> Mark
>
> On Mon, 11 Jan 2016, Lucio Chiappetti wrote:
>
>> ANNOUNCEMENT:  START OF FORMAL PUBLIC COMMENT PERIOD
>>
>> This is to announce the official reopening of a formal Public Comment Period
>> on the incorporation of the Tiled Image Compression and Tiled Table
>> Compression conventions in the FITS Standard.
>>
>> A previous PCP was held in June-July (started on 16 Jun 2015).
>>
>> We are now re-opening the Public Comment Period to collect comments on the
>> modified text, ONE CONVENTION AT A TIME. When the PCP is over, IAUFWG will
>> vote on one convention in turn, while the next one will be offered for public
>> comments, and so on.
>>
>> The second convention if the list, to be discussed NOW defines the
>> Tiled Image Compression and Tiled Table Compression conventions.
>>
>> The proposed text consists
>>
>> - in the ADDITION of an entire new chapter (10)  to the FITS Standard
>>    Document which describes the two conventions in a common prescriptive
>>    framework.
>> - It also includes the ADDITION of a new non-prescriptive Appendix I,
>>
>> Only minor re-wording were applied w.r.t. the earlier version. Considering
>> that, despite its complexity, the proposal did not receive any substantial
>> objections, I am reopening the PCP for TWO WEEKS (in order to speed up the
>> cumulative prcoess dealing with all conventions), but this can be extended if
>> requested and motivated.
>>
>> The updated draft text is available at
>> http://sax.iasf-milano.inaf.it/~lucio/FITS/Conventions/compression-rev.pdf
>>
>> - the relevant bibliographic references, and appendices C and H.3 will be
>>    updated accordingly at the end of the convention incorporation procedure
>>    (this text is NOT shown but a cumulative draft is available at
>>    http://sax.iasf-milano.inaf.it/~lucio/FITS/Conventions/commonapps-rev.pdf
>>
>> Please review the text carefully and post any comments, criticisms, or
>> suggestions on the FITSBITS mailing list.
>>
>> Supporting material for the PREVIOUS public comment period announcement
>> can be retrieved from the FITSBITS archives
>> https://listmgr.nrao.edu/pipermail/fitsbits/2015-June/002615.html
>>
>> ==================================================================
>>
>> The reopened Public Comment Period starts today 11 January 2015 and will last
>> until 24 January 2016, unless extensions are requested and motivated.
>>
>> -- 
>> Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy)
>> for IAU FWG (prorogated)
>>
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>>
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits