[fitsbits] start of Public Comment Period on compressed FITS image and tables

[Dick Shaw]

Forgive me for my late reply, but it's important to respond to some commentary posted a few weeks 
ago:

On Thu, 25 Jun 2015 23:14:47 -0400
>From: "Tom McGlynn (NASA/GSFC Code 660.1)" <tom.mcglynn at nasa.gov>
> To: FITSBITS <fitsbits at nrao.edu>
> Subject: Re: [fitsbits] start of Public Comment Period on compressed
> 	FITS image and tables
> Message-ID: <558C1253.5000203 at nasa.gov>
> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
> 
> While I'm generally supportive of the compression proposal (at least for 
> images), I feel that the current text reflects the sense of this being a 
> convention rather than part of the standard.  By this I mean that if we 
> are going to support compressed images and tables then they should be 
> incorporated into the standard as first class objects.  The current text 
> makes it clear that these compressed HDU's are compressed 
> representations of virtual uncompressed images and tables. 

Yes, that is exactly what it means.

> Implicitly 
> the idea is [that] the user converts from the compressed image to the 
> uncompressed version and then processes that. 

No, it means some software (the FITS reader) must uncompress the bytestream for the data to be 
interpreted as the original HDU.

> Instead we should 
> recognize that a compressed image is just one of the ways that FITS 
> allows one to store an image just like a pimary image array, an 
> extension image or vector value in a table.

There are important differences, but yes, that's the idea.

> So I would suggest that the ZSIMPLE, ZEXTEND, ZBLOCKED and such keywords 
> be made optional 

The text states:

"The following keywords are reserved....
These optional keywords, when present, shall be used when reconstructing
an identical copy of the original FITS HDU"

They are clearly optional. However, if used, these keywords must be used only to enable an 
identical copy of the image to be reconstructed.

> with wording something like:  "If a compressed image is 
> being used to compress an existing FITS image extension, the ZXTENSION 
> keyword MAY be used to contain the value of the original extension."     
> I'd suggest that in future use the use of these keywords be discouraged.

The point was to provide a uniform mechanism to make an identical copy of the original image. 
Discouraging their use is counterproductive, and allowing their use is helpful in one 
circumstance, and benign in others.

> The recommended practice would be that users treat the compressed image 
> as the image and not worry about some
> intermediate image representation.

The point of this chapter is to allow users (or software) to treat compressed images either as 
such (preserving direct access to metadata) or to restore the original pixels and treat it as an 
ordinary image, depending upon the need.

> My second major concern with with this convention is that it does seem 
> rather ad hoc.  I think that it would be much better if the proposal was 
> rigorously separated the algorithmic aspects from the non-algorithmic 
> elements. 

I thought they were separated to the extent they can be, but specific suggestions would be 
welcome.

> A mechanism for how additional compression techniques could 
> be added should be noted

This is described in Sect. 10.4, and references appear to that section from a few points in the 
Chapter.

> E.g., the discussion of quantization should part be 
> of the implementation of the lossy compression algorithms and the 
> ZQUANTIZ parameter should  probably be  one of the ZVALn, ZNAMEn 
> elememts.  

Quantization is a separate, optional step that may be performed to optimize the compression. This 
is discussed at length in Sect. 10.2.

> Table 36 should titled something like:
>   Supported Compression Algorithms
> with the first column being the name of the compression algorithm, the 
> second the value of ZCMPTYPE, and also including the ZNAMEs using 
> (flagging the critical ones).
> 
> I've almost no insight into table compression.  Given that no one seems 
> to be using this convention, my suggestion would be that it's premature 
> to add to the standard.

These compression techniques are used extensively at NOAO and (I believe) at STScI, though not as 
the only way to package data for transport. I know of some popular applications and systems (e.g., 
DS9, IRAF) that treat FITS-compressed data transparently. I do not know the extent of its use 
elsewhere, but I suspect it will be used more widely once this capability is part of the Standard.

> Overall I suspect that the tiling capabilities are going to be 
> increasing essential for handling large images, so that at least that 
> much needs to be made part of the standard.  However I don't feel this 
> text is ready to be finalized.
> 
>     Regards
>     Tom McGlynn

Again, specific suggestions for improving the text would be welcome.

Regards,
Dick Shaw