[fitsbits] reopening of Public Comment Period on the compression conventions

[Dick Shaw]

The main idea behind combining the image and table compression conventions into a single Chapter 
for inclusion in the Standard is that they draw upon the same technology and the same approach 
w.r.t. making floating-point numbers, NaNs, etc. compressible. Thus, many of the tables and other 
material apply to both. I attempted (as the original authors did) to craft the narrative in a way 
that could be extended in the future, should other techniques be incorporated, or other kinds of 
data objects be the target of compression. What's mostly different between compressing images vs. 
tables is the way the data are chunked prior to compression, and the wrangling of the relevant 
metadata.

So to me it doesn't make a lot of sense to split the proposed update into two. I'm not sure I'd 
object, but I don't know what would be gained by doing so. Yes it's true that table tile 
compression is not very widely used today (and many apps may not choose to implement table 
compression soon), but that may change in the future. Anyway, the machinery clearly works, and the 
chapter as proposed makes very clear the intention of supporting multiple types of compressed data 
objects in FITS.

-Dick

On Tue, 12 Jan 2016 12:00:05 -0500
  fitsbits-request at listmgr.nrao.edu wrote:
> Message: 1
> Date: Tue, 12 Jan 2016 12:21:39 +0100 (CET)
>From: Lucio Chiappetti <lucio at lambrate.inaf.it>
> To: FITS List <fitsbits at nrao.edu>
> Subject: Re: [fitsbits] reopening of Public Comment Period on the
> 	compression conventions
> Message-ID:
> 	<alpine.LSU.2.00.1601121152580.5638 at cbfrvqba.ynzoengr.vans.vg>
> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
> 
> On Mon, 11 Jan 2016, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
> 
>> 1. "Tiled images"  defines a new representation of images as tiles
>>    within a binary table. These tiles represented in this standard are
>>    not compressed.  They are just fields in a binary table.
> 
> Personally I do not see much reason in having these tiled images (well, I 
> do recall "Exosat small maps", which were somehow optimized for random 
> access, but that was in 1980s), but apparently they are implicity already 
> allowed by the secret option Bill kept in his pocket "'NOCOMPRESS' ... 
> already supported by CFITSIO and fpack"
> 
>> 2. "Table compression" defines a general standard for compression of
>>    binary tables.  In this context the tiled images are simply one kind
>>    of binary table.
> 
>> A file that uses both of these conventions would look like our current
>> compressed tile images.
> 
> I am not deep enough in the inner working of compression. Is it true that
> 
>  - ZIMAGE compressing (storing) an image in NOCOMPRESS tiles
>  - then ZTABLE compressing the resulting bintable
> 
>  gives the same result as
> 
>  - ZIMAGE compressing the image with one particular algorithm ?
> 
> If the operations are not associative and the resulting files will be 
> different, this should be against the separation. given that there are 
> plenty of image viewers supporting current ZIMAGE-compressed images.
> 
>> Supporting image tiling seems important, but it's less clear that there 
>> much push from the community for supporting non-image compressed binary 
>> tables and I would prefer to leave that out until have a few 
>> instantiated use-cases.
> 
> Essentially I see a benefit of elegance in having both conventions 
> described in the same chapter and voted at the same time.
> 
> And since FITS has a fame of being slow in its approval process, I would 
> avoid things (like we did in the past for BINTABLE) as having some parts 
> listed in an appendix as "not part of the standard" and being voted as 
> such a few years later. Let's be daring, and pass all in one go.
> 
> On Mon, 11 Jan 2016, William Pence wrote:
>> would suggest a simpler solution in this case:  Just pipe the compressed 
>> FITS file to an external uncompression program (funpack in this case)
> 
> This I surely second.
> 
> Personally I would never write a line of code about compressing and 
> uncompressing, but since I recently received (actually I had to ftp) some 
> tens of bulky catalogues which I'll have to store and redistribute, I was 
> contemplating to fpack them before storing !
> 
> It takes just a couple of minutes to build fpack/funpack after all !
> 
> On Mon, 11 Jan 2016, William Pence wrote:
>> I support the suggestion by Mark Taylor and Tom McGlynn to decouple the 
>> Image compression and Table compression proposals ... the proposals can 
>> be voted on separately.
> 
> This is a possibility. I note however that some of the IAUFWG voters are 
> requesting to move on and speed up things. So I suggest we hold separate 
> votes but simultaneously.
> 
> It looks to me that Images correspond to 10.1, 10.2 and 10.4, while Tables 
> correspond just to 10.3 with a cross-reference to a subset of the 
> algorithms decribed in 10.4
> 
> Apart from the numbering (if 10.3 is not approved, current 10.4 has to be 
> renumbered, OR the order of sections has to be changed beforehand) the 
> relevant text is enough segregated.
> 
> I'd like to hear the opinion of Dick Shaw which spent a lot of time on 
> writing chapter 10 combining the two conventions.
> 
> Lucio
[...]