[fitsbits] reopening of Public Comment Period on the compression conventions

Mon Jan 11 18:01:57 EST 2016

While I don't see much benefit in totally separating the tiling 
methodology from the compression techniques into separate proposals, I 
do think it would be good to add another compression technique to Table 
36: 'NOCOMPRESS' which means that the tile of data are stored verbatim 
without any compression.  This is mainly useful for debugging purposes 
and is already supported by CFITSIO and fpack.

-Bill Pence

On 1/11/2016 11:44 AM, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
> In the interval since the last discussion, I've had a bit of an epiphany
> about this proposal.  I still believe that there should be two distinct
> proposals, but changed the basis set I'm thinking in.
>
> Consider the name 'tiled image compression'.  This says we're doing  two
> different things:
>
> First we're defining a new way of representing an image: as a set of
> tiles. Second we're compressing those tiles.
>
> I would suggest that it would be much cleaner to have this as two
> separate proposals:
>
>   1. "Tiled images"  defines a new representation of images as tiles
> within a binary table. These tiles represented in this standard are not
> compressed.  They are just fields in a binary table.
>
> 2. "Table compression" defines a general standard for compression of
> binary tables.  In this context the tiled images are simply one kind of
> binary table.
>
> A file that uses both of these conventions would look like our current
> compressed tile images.
>
> This seems much cleaner to me than the current approach.  Each proposal
> does one thing completely.
>
> I don't know how difficult it would be to recast the existing proposal
> into these terms, but it seems liike it should be possible to do this in
> a way which is fully consistent with the existing datasets.
>
> Were this approach adopted, then it would raise some questions as to
> whether we need to have  non-compressed implementations of tiled tables,
> but would conversely mean that we have lots of implementations of the
> table compression code since all of the existing compressed images would
> qualify.
>
> Absent such a restructuring, I'd reiterate my comments from June:
> Supporting image tiling seems important, but it's less clear that there
> much push from the community for supporting non-image compressed binary
> tables and I would prefer to leave that out until have a few
> instantiated use-cases.
>
>     Tom McGlynn
>
>
> Mark Taylor wrote:
>> Dear FITS,
>>
>> I have no objection to tiled compression for image data, which
>> I understand to have been in wide and successful use.
>>
>> However, I would like to reiterate the concerns I aired in the
>> previous public comment period about tiled compression for *table*
>> data (the proposed Section 10.3).  In response to my query
>> about where this convention has been used to date, only one example
>> was given; Bill Pence cited its use by the FACT Cherenkov Telescope
>> project (see message
>> https://listmgr.nrao.edu/pipermail/fitsbits/2015-July/002803.html).
>> As Bill noted, in that case a custom compression algorithm was used,
>> which as I understand it means that although they are using substantial
>> parts of the proposed compression convention, they are not using
>> it in quite its proposed form, so that a FITS reader implemented
>> according to the current proposal would not recover the FACT data.
>>
>> My feeling is that this level of usage of the tiled table compression
>> convention does not justify its addition into the standard,
>> especially in view of the proposal's complexity, and hence the
>> implementation burden it imposes upon implementors (e.g. me) who
>> wish to be able to parse all standard-compliant FITS table data.
>>
>> Tom McGlynn made a similar point:
>>
>> On Thu, 25 Jun 2015, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
>>
>>> I've almost no insight into table compression.  Given that no one
>>> seems to be
>>> using this convention, my suggestion would be that it's premature to
>>> add to
>>> the standard.
>> Mark
>>
>> On Mon, 11 Jan 2016, Lucio Chiappetti wrote:
>>
>>> ANNOUNCEMENT:  START OF FORMAL PUBLIC COMMENT PERIOD
>>>
>>> This is to announce the official reopening of a formal Public Comment
>>> Period
>>> on the incorporation of the Tiled Image Compression and Tiled Table
>>> Compression conventions in the FITS Standard.
>>>
>>> A previous PCP was held in June-July (started on 16 Jun 2015).
>>>
>>> We are now re-opening the Public Comment Period to collect comments
>>> on the
>>> modified text, ONE CONVENTION AT A TIME. When the PCP is over, IAUFWG
>>> will
>>> vote on one convention in turn, while the next one will be offered
>>> for public
>>> comments, and so on.
>>>
>>> The second convention if the list, to be discussed NOW defines the
>>> Tiled Image Compression and Tiled Table Compression conventions.
>>>
>>> The proposed text consists
>>>
>>> - in the ADDITION of an entire new chapter (10)  to the FITS Standard
>>>    Document which describes the two conventions in a common prescriptive
>>>    framework.
>>> - It also includes the ADDITION of a new non-prescriptive Appendix I,
>>>
>>> Only minor re-wording were applied w.r.t. the earlier version.
>>> Considering
>>> that, despite its complexity, the proposal did not receive any
>>> substantial
>>> objections, I am reopening the PCP for TWO WEEKS (in order to speed
>>> up the
>>> cumulative prcoess dealing with all conventions), but this can be
>>> extended if
>>> requested and motivated.
>>>
>>> The updated draft text is available at
>>> http://sax.iasf-milano.inaf.it/~lucio/FITS/Conventions/compression-rev.pdf
>>>
>>>
>>> - the relevant bibliographic references, and appendices C and H.3
>>> will be
>>>    updated accordingly at the end of the convention incorporation
>>> procedure
>>>    (this text is NOT shown but a cumulative draft is available at
>>>
>>> http://sax.iasf-milano.inaf.it/~lucio/FITS/Conventions/commonapps-rev.pdf
>>>
>>>
>>> Please review the text carefully and post any comments, criticisms, or
>>> suggestions on the FITSBITS mailing list.
>>>
>>> Supporting material for the PREVIOUS public comment period announcement
>>> can be retrieved from the FITSBITS archives
>>> https://listmgr.nrao.edu/pipermail/fitsbits/2015-June/002615.html
>>>
>>> ==================================================================
>>>
>>> The reopened Public Comment Period starts today 11 January 2015 and
>>> will last
>>> until 24 January 2016, unless extensions are requested and motivated.
>>>
>>> --
>>> Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy)
>>> for IAU FWG (prorogated)
>>>
>>> _______________________________________________
>>> fitsbits mailing list
>>> fitsbits at listmgr.nrao.edu
>>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>>>
>> --
>> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
>> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
>>
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits