[fitsbits] reopening of Public Comment Period on the compression conventions

[Rob Seaman]

Lucio Chiappetti wrote:

> And since FITS has a fame of being slow in its approval process, I would avoid things (like we did in the past for BINTABLE) as having some parts listed in an appendix as "not part of the standard" and being voted as such a few years later. Let's be daring, and pass all in one go.

Hear, hear!

Tile-compression for images and for tables build on similar concepts of plugging multiple compression algorithms into the same simple tiling scheme. Neither the list of algorithms nor the rectangular tiling is exhaustively complete, and nothing about tile-compression suggests, for instance, that HEALPix or HTM tilings won’t be used for other purposes.

Tiling for tables is a chicken-and-egg problem; it came along later than for images and simply hasn’t had the chance for similar uptake. That said, both images and tables are rectangular grids stored in separate HDUs and tile-compressing a table is a well-defined operation.

Both should be described together and both should be adopted together.


William Pence wrote:

> While I don't see much benefit in totally separating the tiling methodology from the compression techniques into separate proposals, I do think it would be good to add another compression technique to Table 36: 'NOCOMPRESS' which means that the tile of data are stored verbatim without any compression.  This is mainly useful for debugging purposes and is already supported by CFITSIO and fpack.

NOCOMPRESS entirely satisfies this suggestion. The larger compression literature always has to address the question of packaging, e.g., jpeg. The two ideas go hand-in-hand and should be described together. “No compression” is a special case of data represention, just as compression is. The question is how efficiently the data are represented.


Mark Taylor wrote:

> As Bill noted, in that case a custom compression algorithm was used

Per section 10.4, shouldn't we simply be discussing reserving a type identifier for the algorithm used by FACT:

	"The name of the permitted algorithms for compressing FITS HDUs, as recorded in the ZCMPTYPE keyword, are listed in Table 36; if other types are later supported, they must be registered with the IAUFWG to reserve the keyword values."

Or perhaps we need to define what is required of the IAUFWG for such registration?

I support the current draft. The concept of tile-compression is quite mature and widely used. It has now been extended to include binary tables in a straightforward application of the same concepts. That projects may identify new algorithms in the future is a strength of the tile-compression framework. The FPACK defaults of RICE_1 for images and GZIP_2 for tables are sensible application-level choices, but new applications may reveal new requirements; the framework needs to be flexible. And the concepts behind tile compression, both lossless and lossy, are easily applicable to non-FITS formats, and to alternate tiling schemes; incorporating them into the standard has broader implications.

On the other hand, rewriting the text to artificially separate tiling from compression, or tables from images, would only confuse the issues involved in what is otherwise a coherent whole.

Rob