[fitsbits] start of Public Comment Period on compressed FITS image and tables

Wed Jun 24 13:06:28 EDT 2015

At nom-tam-fits we wan't to implement the compression's and have done so partly already. The 
biggest problem is that the standard does not provide test data, and in my opinion this is a  prerequisite for a standard like this.
In fact a good standard delivers a reference implementation or a compliance test. 

I can see that a reference implementation is not possible in a community like this, but the delivery of
test data would make the live of a developer 1000 fold easyer. Especially if it is available in levels, like separate data files for the different kinds of compression. 

Sure the compression makes fits more comple, but that's what access library's like ours are for. Especially when the fits files start to get as big as e few gigabytes...

       Ritchie
________________________________________
From: fitsbits [fitsbits-bounces at listmgr.nrao.edu] on behalf of Mark Taylor [m.b.taylor at bristol.ac.uk]
Sent: Wednesday, June 24, 2015 5:45 PM
To: FITS List
Cc: Amit Kapadia
Subject: Re: [fitsbits] start of Public Comment Period on compressed FITS image and tables

On Wed, 24 Jun 2015, Lucio Chiappetti wrote:

> On Wed, 24 Jun 2015, Mark Taylor wrote:
>
> > but three weeks is not enough time to attempt an implementation to check it
> > does actually provide all the description required to implement these
> > conventions.
>
> That statement reminds me interoperability testing is part of the criteria to
> be considered by the EC when going to votes. I assumed the fact the registered
> conventions (in general) have been around for quite a while could waive for
> most of the interoperability demonstration.

That does prove that it can be done and work well, but it doesn't prove
that the way it's described in the text correctly and fully describes
how it's done.

> Nevertheless, while nobody is asking that topcat supports ZIMAGE and ZTABLE
> now, in 3 weeks, 3 months or 3 years :-) it would be helpful if the original
> authors and supporters of the convention may recall the status of
> interoperability (multiple implementations), and also any other author of
> popular s/w (e.g. ds9) may state whether they support it already.

Yes, I'd be very interested to hear if there is more than one
independent implementation out there.

> > The text looks like what it presumably is, a post-hoc codification of a
> > series of experiments in compression
>
> This seems a bit unfair or un-kind, but ...

I apologise if that is or appears to be true.  The text itself looks
well written and complete, but my point was that there is quite a
large number of algorithmic options described, and my suspicion
is that some of them may be there because they got implemented
and then superceded by a better idea rather than because they
are all useful alternatives.

> > is all there in CFITSIO, but that can't be used directly by non-C-friendly
> > languages such as java, javascript, and who knows
>
> is Fortran C-friendly ? or idl ?

I would class IDL and FORTRAN as "C-friendly" on the grounds that
they are (probably) implemented in C and can generally be linked
cleanly to C routines.  Java is capable of interfacing with C
via the Java Native Interface, but this breaks the 'pure java' model
and greatly complicates distribution of java executables which
are otherwise completely platform-independent, so I'd call it
C-hostile.  In browser-based javascript I think it's generally
impossible for security reasons to use C-based libraries or
run C-based utilities.

>                                  I would not engage myself in supporting
> ZIMAGE or ZTABLE directly, not even in java ... I guess it won't be very
> efficient.

You might not, but I probably should.  If this becomes part of the
standard (rather than an associated convention) I would hope to
be able to support tile compressed tables in the java library STIL
(hence STILTS/TOPCAT), though I'm not sure on what timescale.
(There is no reason by the way that java cannot do this processing
with comparable efficiency to C.)  At the moment this java software
supports any table encoded according to the current FITS standard,
and I'd like to be able to say the same about future versions.

>            Although if somebody writes a class for a Rice (or PLIO or
> HCOMPRESS) stream (gzip *is* supported) the problem can be nicely isolated.

Yes, if they do.

> (well javascript is not a serious language really suited to deal with binary
> data, is it ?)

I'm not a javascript/HTML5 expert, but there are increasingly complex
web applications appearing for visualising and manipulating
astronomical data within the browser.  Some of these do the data
access on the server side, in which case use of CFITSIO can be used,
but not all.  While investigating this, I found that the client-side
javascript FITS I/O library fitsjs (https://github.com/astrojs/fitsjs)
actually already offers partial support for tile-compressed images -
though it looks like only the Rice algorithm is supported.
I have CC'd Amit Kapadia, the author of that package, in case he
wants to comment on the difficulties or otherwise of implementation
(and doesn't already read this list).

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/

_______________________________________________
fitsbits mailing list
fitsbits at listmgr.nrao.edu
https://listmgr.nrao.edu/mailman/listinfo/fitsbits