<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; "><DIV></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">On Jul 12, 2006, at 9:32 PM, Mark Calabretta wrote:</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">I now mostly use bzip2 (GPL, bzip.org), it usually outperforms gzip by</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">about 10% for binary data.</SPAN></FONT></DIV></BLOCKQUOTE><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">It's quite a bit slower, however.</SPAN></FONT><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR style=""></SPAN></FONT><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">I have also recently started to use 7zip, an archiving compressor that</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">mostly seems to outperform bzip2. The unix GPL distro (p7zip.org) also</SPAN></FONT></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">supports gzip and bzip2 and various archive formats. [...] wikipedia</SPAN></FONT></DIV></BLOCKQUOTE><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">has excellent articles on all of these. It also mentions PPM which</SPAN></FONT></DIV></BLOCKQUOTE><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">is supposed to be the best, at least for text.</SPAN></FONT></DIV></BLOCKQUOTE><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">One needs to distinguish between the compression algorithm and the FITS tile-compression convention. The latter should allow output from most any algorithm to be embedded in a FITS file complete with readable headers and transparent access to the pixels. A Rice compressed image expressed as a bintable looks like the original image array to cfitsio, for instance.</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">In the absence of further suggestions for tile-compression options beyond the cfitsio example imcopy program, I've decided to proceed to roll my own. Bill Pence is making corresponding improvements to cfitsio to address some of the issues mentioned in my prior message.</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">I've appended the envisioned command line help for a new cfitsio layered program called "fpack". There will also be "funpack" which will share the capital letter arguments below. Fpack will be able to compress a list of FITS files in place, or to create a single large output compressed MEF. Funpack will be able to uncompress in place, or to restore compressed HDUs to the original list of individual files.</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">I see these commands as combining features from the FITS tile-compression, checksum and foreign encapsulation conventions, and benefiting from various cfitsio functionality such as HDU validation. The final product will likely differ in (hopefully) small ways from the concept below, but the idea should remain to provide rich functionality in a simple package (e.g., this command won't descend directory trees - if you need that, use FITS encapsulation first).</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">It has also come to the notice of its authors that the foreign file encapsulation proposal is not widely known. Somehow it was never published after it appeared as an ADASS IX poster (same year as tile-compression). We intend to remedy this via the new registry of FITS conventions after having a chance to review our latest (very mature) draft. Conforming writer and reader have been available since 1999 as host callable tasks in the IRAF fitsutil external package.</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">Comments about the fpack concept, or FITS compression in general, would be quite welcome. Folks intending to add tile-compression support to their own FITS libraries, image processing environments, or web services would be deliriously so.</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR style=""></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">Rob Seaman</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">NOAO</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">------</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">fpack, a FITS tile-compression engine. Version 0.0.2 (18 July 2006)</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">usage: fpack [-r|-g|-p] [-w|-t <axes>] [-n <bits>] [-FLAGS] <FITS></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -r Rice compression [default], or</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -p PLIO compression, or</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -g GZIP (per-tile) compression</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -n <bits> noise bits to preserve for real pixels [default=4]</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -t <axes> CSV list of tile sizes [default=one tile/image row], or</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -w override tile size to be whole image</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -F clobber output file(s) (overwrites input in-place by default)</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -K keep (don't delete or overwrite) input files</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -A <file> append all output to single <file>, create if needed, or</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -P <pre> prepend <pre> to create separate output filenames</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -U|-u unpack files, instead of pack (same as funpack)</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -Z when unpacking, do not restore original names/times</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -L list-only file contents</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -Q silent operation</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -H print this message</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> -V print version number</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"> <FITS> FITS files to (un)pack</SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;"><BR class="khtml-block-placeholder"></SPAN></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier" size="3"><SPAN class="Apple-style-span" style="font-size: 13px;">All flags must appear (separately) before all filenames.</SPAN></FONT></DIV><DIV><BR class="khtml-block-placeholder"></DIV></BODY></HTML>