[fitsbits] start of Public Comment Period on compressed FITS image and tables

[Tom McGlynn (NASA/GSFC Code 660.1)]

While I'm generally supportive of the compression proposal (at least for 
images), I feel that the current text reflects the sense of this being a 
convention rather than part of the standard.  By this I mean that if we 
are going to support compressed images and tables then they should be 
incorporated into the standard as first class objects.  The current text 
makes it clear that these compressed HDU's are compressed 
representations of virtual uncompressed images and tables.  Implicitly 
the idea is the the user converts from the compressed image to the 
uncompressed version and then processes that.  Instead we should 
recognize that a compressed image is just one of the ways that FITS 
allows one to store an image just like a pimary image array, an 
extension image or vector value in a table.

So I would suggest that the ZSIMPLE, ZEXTEND, ZBLOCKED and such keywords 
be made optional with wording something like:  "If a compressed image is 
being used to compress an existing FITS image extension, the ZXTENSION 
keyword MAY be used to contain the value of the original extension."     
I'd suggest that in future use the use of these keywords be discouraged.

The recommended practice would be that users treat the compressed image 
as the image and not worry about some
intermediate image representation.


My second major concern with with this convention is that it does seem 
rather ad hoc.  I think that it would be much better if the proposal was 
rigorously separated the algorithmic aspects from the non-algorithmic 
elements.  A mechanism for how additional compression techniques could 
be added should be notedE.g., the discussion of quantization should part 
of the implementation of the lossy compression algorithms and the 
ZQUANTIZ parameter should  probably be  one of the ZVALn, ZNAMEn 
elememts.  Table 36 should titled something like:
   Supported Compression Algorithms
with the first column being the name of the compression algorithm, the 
second the value of ZCMPTYPE, and also including the ZNAMEs using 
(flagging the critical ones).

I've almost no insight into table compression.  Given that no one seems 
to be using this convention, my suggestion would be that it's premature 
to add to the standard.

Overall I suspect that the tiling capabilities are going to be 
increasing essential for handling large images, so that at least that 
much needs to be made part of the standard.  However I don't feel this 
text is ready to be finalized.

     Regards
     Tom McGlynn

Lucio Chiappetti wrote:
> ANNOUNCEMENT:  START OF FORMAL PUBLIC COMMENT PERIOD
>
> This is to announce the official start of a 3-week formal Public 
> Comment Period on the incorporation of the Tiled Image Compression and 
> Tiled Table Compression conventions in the FITS Standard.
>
> This is part of a process to incorporate the most useful and widely 
> used registered conventions (which are valid FITS constructs) into the 
> official definition of the standard.
>
> Among these the two compression conventions benefit of a common 
> handling. Given their relative complexity they are better discussed 
> first, before other easier conventions.
>
> The proposed text consists
>
> - in the ADDITION of an entire new chapter (10)  to the FITS Standard
>   Document which describes the two conventions in a common prescriptive
>   framework.
> - It also includes the ADDITION of a new non-prescriptive Appendix I,
> - plus the addition of the necessary bibliographic references,
>
>   and has been prepared by a technical team including L.Chiappetti,
>   W.Pence, A.Dobrzycki, R.A.Shaw and W.Thompson (main editor Dick Shaw).
>
> - If the proposal is approved also Appendix C will be updated listing the
>   new keywords, and a section H.3 will be added to Appendix H describing
>   the updates and the differences with the registered convention.
>
>   All the updates are shown in blue colour in their current context (with
>   the exception of the NEW chapter 10 which is black)
>
> The proposed draft text is available at 
> http://sax.iasf-milano.inaf.it/~lucio/FITS/Conventions/compression-upd2.pdf
>
> Supporting material is provided in the FITS Convention Registry at the 
> following URLs
> http://fits.gsfc.nasa.gov/registry/tilecompression.html 
> http://fits.gsfc.nasa.gov/registry/tiletablecompression.html
>
> Considering that the convention(s) have been in use since several 
> years, are legal FITS, were discussed on FITSBITS when the conventions 
> were entered in the Registry and therefore their usage is well proven 
> (also for what interoperability is concerned), the Public Comment 
> Period is reduced to 3 weeks.
>
> Also the review by FITS Working Group Executive can be speeded up and 
> handled in parallel or quickly after the conclusion of the Public 
> Comment Period.
>
> Please review the text carefully and post any comments, criticisms, or 
> suggestions on the FITSBITS mailing list (not on iauwfg or elsewhere) 
> ==================================================================
>
> The Public Comment Period starts today 16 June 2015 and will last 
> formally for 3 weeks until July 6
>
> ================================================================== 
> Background information on the FITS approval process
>
> Under the "Rules and Procedures" of the IAU FITS Working Group, 
> http://fits.gsfc.nasa.gov/iaufwg/iaufwg_rules.html, the first step in 
> the official approval process of any FITS proposal will be a formal 
> Public Comment Period to take place on the FITSBITS mailing list. 
> After that the IAU FITS Working Group Executive will review the 
> results. Following that the IAU FITS Working Group will then conduct a 
> final vote to approve or disapprove the proposal.
>