[fitsbits] start of Public Comment Period on compressed FITS image and tables

[van Nieuwenhoven, Richard]

Hi,

I must agree with Bill that the "stuff" is less complicated than it
sounds when reading the text. Like in many cases the algorithm is not so
complicated but the language is not capable to keep the text simple ;-)

That's exactly why a set of low level samples that build on each other
till they become complete fits files would help a lot, in explaining
understanding and in testing

   Ritchie

Am 2015-06-25 um 06:31 schrieb William Pence:
> On 6/24/2015 6:44 AM, Mark Taylor wrote:
>> Dear FITS,
>>
>> regarding the tiled image/table compression proposal, in the spirit
>> of this clarification from Lucio:
>>
>> On Wed, 24 Jun 2015, Lucio Chiappetti wrote:
>>
>>> First of all I clarify that the purpose of the Public Comment Period
>>> is just
>>> to assess the feeling of the community.
>>
>> I'd like to express my feeling that: it looks pretty complicated.
> 
> Yes, and no.  The basic convention is really quite simple: divide the
> image into a rectangular grid of tiles, like a checkerboard, and then
> copy each tile into the corresponding row of a vector column of a binary
> table, where the first tile is stored in the first row, ..., and the
> last tile goes in the last row.  It only takes simple book keeping to
> figure out which tile contains a particular pixel of interest, or to put
> all the tiles back together to recreate the original image.
> 
> The complications come from supporting several different options for
> compressing the tile of pixels before storing them in the binary table.
> Image compression is still an evolving art, and some improved
> compression methods (such as dithering) have been added to this
> convention since it was first developed in 2001.  It is also possible
> that yet more compression options will be added in the future in
> response to user requests and needs.
> 
>>
>> One of the attractions of the FITS format (as opposed to formats
>> such as HDF5 or HDS that are essentially defined by their data
>> access libraries) is that access to the basic numeric data is very
>> straightforward in implementation terms.  When I needed to
>> write a data access library for BINTABLE from scratch I could
>> do it without a great deal of effort.  Supporting tiled table
>> compression would complicate the implementation task considerably.
>> The same goes even more so (complication by an order of magnitude?)
>> for tiled images, though those are less close to my current
>> personal concerns.  That is particularly true if the implementation
>> language does not have existing libraries for the various defined
>> compression algorithms.
> 
> I would guess that there is already a suitable implementation of the
> gzip algorithm for just about every language.  One would then just need
> implementations of the Rice, H-compress, and PLIO algorithms to complete
> the set of currently supported algorithms.  Rice and PLIO are actually
> pretty simple, so it would likely not be difficult to transcribe the
> existing C code into any other language that supports bit-level
> manipulations of arrays of bytes  (for Rice, in particular).  The
> quantizing and dithering algorithms are also pretty simple.
> 
>> The content of the proposed text seems to be reasonably clear,
>> but three weeks is not enough time to attempt an implementation
>> to check it does actually provide all the description required
>> to implement these conventions.  The text looks like what it
>> presumably is, a post-hoc codification of a series of experiments
>> in compression (various different compression algorithms,
>> dithering options), rather than a designed proposal for how to
>> specify compression in a clean way.  Doing it that way is cheap
>> for the implementation that served as the experimental testbed,
>> but expensive for other implementations.
> 
> As mentioned above, the original design of the basic tiled image format
> has not changed, but a few new features have been added over the years
> to support some newer compression methods
> 
>> Yes the implementation of this proposed addition to the standard
>> is all there in CFITSIO, but that can't be used directly by
>> non-C-friendly languages such as java, javascript, and who knows
>> what future platforms might arise.  That means that for instance
>> browser-based FITS image viewers which currently can display any
>> legal FITS image would likely, following incorporation of this
>> convention to the standard, find themselves unable to deal with
>> some standard FITS image data if they are unable to afford
>> considerable extra implementation effort.
> 
> This is beyond my area of expertise, but according to Eric Mandel's
> message earlier today, it may in fact be possible to incorporate C
> libraries into the languages that are used for Web applications.
> 
>> I am aware there are some committed advocates and strong arguments
>> for use of this convention.  This message is not itself a call to block
>> incorporation of this text in the standard.  But since it's a Public
>> Comment Period, I wanted to make a public comment noting my
>> reservations.
>>
>> Mark
>>
> 
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits


-- 
BSc Richard van Nieuwenhoven
Software Architekt

adesso Austria GmbH
floridotower 26. Stock              T +43 1 2198790-0
Foridsdorfer Hauptstr. 1            F +43 1 2198790-13
A-1210 Wien                         H +43 664 88614710
                                    E richard.vannieuwenhoven at adesso.at
                                    www.adesso.at
-------------------------------------------------------------
         >>> business. people. technology. <<<
-------------------------------------------------------------
adesso Austria GmbH mit Sitz in Wien
Handelsgericht Wien FN231467v