[fitsbits] start of Public Comment Period on compressed FITS image and tables

[Mark Taylor]

On Wed, 24 Jun 2015, Lucio Chiappetti wrote:

> On Wed, 24 Jun 2015, Mark Taylor wrote:
> 
> > but three weeks is not enough time to attempt an implementation to check it
> > does actually provide all the description required to implement these
> > conventions.
> 
> That statement reminds me interoperability testing is part of the criteria to
> be considered by the EC when going to votes. I assumed the fact the registered
> conventions (in general) have been around for quite a while could waive for
> most of the interoperability demonstration.

That does prove that it can be done and work well, but it doesn't prove
that the way it's described in the text correctly and fully describes
how it's done.

> Nevertheless, while nobody is asking that topcat supports ZIMAGE and ZTABLE
> now, in 3 weeks, 3 months or 3 years :-) it would be helpful if the original
> authors and supporters of the convention may recall the status of
> interoperability (multiple implementations), and also any other author of
> popular s/w (e.g. ds9) may state whether they support it already.

Yes, I'd be very interested to hear if there is more than one
independent implementation out there.
 
> > The text looks like what it presumably is, a post-hoc codification of a
> > series of experiments in compression
> 
> This seems a bit unfair or un-kind, but ...

I apologise if that is or appears to be true.  The text itself looks
well written and complete, but my point was that there is quite a
large number of algorithmic options described, and my suspicion
is that some of them may be there because they got implemented
and then superceded by a better idea rather than because they
are all useful alternatives.

> > is all there in CFITSIO, but that can't be used directly by non-C-friendly
> > languages such as java, javascript, and who knows
> 
> is Fortran C-friendly ? or idl ?

I would class IDL and FORTRAN as "C-friendly" on the grounds that
they are (probably) implemented in C and can generally be linked
cleanly to C routines.  Java is capable of interfacing with C
via the Java Native Interface, but this breaks the 'pure java' model
and greatly complicates distribution of java executables which
are otherwise completely platform-independent, so I'd call it
C-hostile.  In browser-based javascript I think it's generally
impossible for security reasons to use C-based libraries or
run C-based utilities.

>                                  I would not engage myself in supporting
> ZIMAGE or ZTABLE directly, not even in java ... I guess it won't be very
> efficient.

You might not, but I probably should.  If this becomes part of the
standard (rather than an associated convention) I would hope to
be able to support tile compressed tables in the java library STIL
(hence STILTS/TOPCAT), though I'm not sure on what timescale.
(There is no reason by the way that java cannot do this processing
with comparable efficiency to C.)  At the moment this java software
supports any table encoded according to the current FITS standard,
and I'd like to be able to say the same about future versions.

>            Although if somebody writes a class for a Rice (or PLIO or
> HCOMPRESS) stream (gzip *is* supported) the problem can be nicely isolated.

Yes, if they do.

> (well javascript is not a serious language really suited to deal with binary
> data, is it ?)

I'm not a javascript/HTML5 expert, but there are increasingly complex
web applications appearing for visualising and manipulating
astronomical data within the browser.  Some of these do the data
access on the server side, in which case use of CFITSIO can be used,
but not all.  While investigating this, I found that the client-side
javascript FITS I/O library fitsjs (https://github.com/astrojs/fitsjs)
actually already offers partial support for tile-compressed images -
though it looks like only the Rice algorithm is supported.
I have CC'd Amit Kapadia, the author of that package, in case he
wants to comment on the difficulties or otherwise of implementation
(and doesn't already read this list).

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/