<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; "><DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">On Jul 18, 2006, at 9:05 PM, Mark Calabretta wrote:</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR></SPAN></FONT></DIV> <BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">For the FITS binary table, 7zip is costly in CPU time for compression</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">but beats gzip and bzip2 handsomely in compression ratio.</SPAN></FONT><SPAN class="Apple-converted-space"><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> </SPAN></FONT></SPAN><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">However, 7zip</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">is not nearly so costly in elapsed time for decompression.</SPAN></FONT><SPAN class="Apple-converted-space"><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> </SPAN></FONT></SPAN><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">If these</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">results are typical then 7zip would have to be the compressor of choice</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">for FITS data distributed on the web.</SPAN></FONT></DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">Which raises the general question of constructing a figure of merit for data compression.</SPAN></FONT><SPAN class="Apple-converted-space"><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> Discussions like this usually focus on c</SPAN></FONT></SPAN><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">ompression ratio, the speed to compress and the speed to decompress, but there are a number of important, less quantifiable, parameters:</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">1) market penetration - gzip is a clear leader here</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">2) openness of software - Both ends of the spectrum may have issues. Patents held by some multi-national can quell our access (and interest) if there is no loophole for educational licensing, but navigating the intricacies of some extreme copyleft can do the same.</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">3) applicability to a particular purpose - tiled Rice and PLIO are very attractive, tiled gzip much less so (with default parameters)</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">4) tailoring to data - a tile compressed FITS file is still a FITS file</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">5) stability across a range of data sets - Even good ol' gzip varies quite a bit in compression ratio from one file to the next. For example, the average gzip compression ratio over two years of NOAO Mosaic II data is <SPAN style="">0.586 +/- 0.0449. Four and a half percent (1-sigma) may not seem like a very wide distribution, but it's all in the meaning of "average". This is from 170 nights selected from 304 total. All nights with binned data were rejected. All multi-instrument nights were rejected. All nights with fewer than 10 object exposures were rejected. And more to the point, average here means "the mean of nightly means". Picking a random recent night, the compression ratio varies between 0.33 and 0.79 across several dozen overtly identical 140 MB files. Calibrations at the low end, of course, and object frames at the top. Obviously there are issues of information theory here and one could use the incompressibility of the "science" data to gauge the skill of the observer :-)</SPAN></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">6) availability of software - if God hadn't created cfitsio, it would have had to be invented. (Those who might be thinking that the same applies for the Devil and IRAF - shame on you!)</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">7) community support - after 7 years one might have hoped that more projects and software would support tile compression.</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">8) <your feature here></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">In general, we often get bound up in theoretical discussions about things like lossy compression, rather than focusing on pragmatic issues of usability and suitability. Meanwhile the LSST tidal wave approaches, but there are going to be several smaller waves impacting astronomy's shores first, including Pan-STARRS (however many telescopes) and next generation instruments like the One-Degree Imager and the Dark Energy Camera.</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">Features like #'s 1-8 can all be addressed through coordinated community action - it might as well be the FITS community. On the other hand, the best way to understand the figure of merit parameters of compression ratio, speed in, and speed out may be to focus not on static archival holdings, but rather on the costs of bandwidth and latency encountered when moving the data around. After all, isn't the point of the emerging Virtual Observatory to keep the pixels in play, ever moving and interacting? Even if we co-locate processing with data, the data have to shuttle from a SAN across gigabit or fiber channel to the Beowulf next door. As Arnold just pointed out, customer satisfaction (and thus our job security, I might add) depend on the aggregate response of our systems.</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">I s</SPAN></FONT><SPAN class="Apple-style-span" style="font-size: 13px;">tumbled across a very interesting, very recent, paper on lossless floating point compression:</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN class="Apple-tab-span" style="white-space:pre"> </SPAN><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><A href="http://www-static.cc.gatech.edu/~lindstro/papers/floatzip/paper.pdf">http://www-static.cc.gatech.edu/~lindstro/papers/floatzip/paper.pdf</A></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">...so recent it has yet to appear in either author's online publication list.</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">As far as I can tell, there is nothing about any of the algorithms referenced that would keep them from being used with astronomical data. The real question is how to turn academic advances into useful tools for our community. The FITS tile compression convention is one step toward greasing the rails.<BR></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">Bill Pence wants to add Hcompress to the cfitsio support for tile compression. Imagine, rather, supporting any and all of the algorithms mentioned above - perhaps using some sort of plug-in/component architecture. We're never going to identify a single best compression scheme for all our data. This was the subtext of the tile compression proposal in the first place. It's time to follow through to the logical conclusion. If any application could transparently access data compressed a dozen different ways (perhaps HDU by HDU in the same MEF), there would be no reason not to store such heterogeneous representations or to convert the data on-the-fly for task-specific purposes. A suite of layered benchmark applications would provide the tools to make these decisions. Those tools could even be automated to operate in adaptive ways within the data handling components of our archives, pipelines, web services and portals.</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">Sounds like a nifty ADASS abstract to me :-) I'd already asked Bill if he wanted to work on such a paper - anybody else want to pile on?</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">Rob</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR></SPAN></FONT></DIV></DIV><FONT class="Apple-style-span" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR></SPAN></FONT></BODY></HTML>