<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:338390908;
mso-list-type:hybrid;
mso-list-template-ids:187874016 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Josh and all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Long time no chat. It’s good to meet you, Rithwik. I’m not sure who else is copied on this thread, but it should reach various FITS folk.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As the original author of FPACK, my best decision was to work through the FITS standard, and especially through a reference implementation using CFITSIO. I thoroughly support your apparent interest in doing the same. Many people, both before
and after FPACK, have contributed to FITS tile compression, and there are manifold advantages to creating conforming FITS files. If astropy is one reference implementation for your project, CFITSIO could be the other, which would automatically also generate
standalone FPACK support. Such a utility program is useful for innumerable purposes.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">By design, FITS tile compression supports multiple codecs. The main reason RICE is FPACK’s default is speed, but also record-by-record simplicity. Back in the day, I could review an octal dump of a tile-compressed file and interpret the
records one by one from first principles. That said, it was always envisioned that other algorithms could be added for either special or general purposes. Depending on the trade-offs between speed and size, FPACK’s default might even be updated.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have only quickly reviewed your document. I see you cite both FPACK and our Paper I (“Lossless Astronomical Image Compression and the Effects of Noise”, PASP, 121(878):414), but not Paper II (“Optimal Compression of Floating-Point Astronomical
Images Without Significant Loss of Information”, PASP 122(895):1065). Note that deploying the codec through CFITSIO would allow comparing any built-in JPEG-XL lossy mode with native lossy FITS tile compression using the same algorithm. (That is, if I understand
correctly what you mean by “lossy context”.)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regarding your benchmarking efforts, putting on my day job hat as the Catalina Sky Survey archivist, I’ll direct you to our many terabytes of imaging data (<a href="https://sbn.psi.edu/pds/resource/css.html">https://sbn.psi.edu/pds/resource/css.html</a>).
The raw images are in the *.fits.fz files, and calibrated images in *.arch.fz files. Please let me know if there’s any interest in extending your benchmarks.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I see your datasets don’t include DECam, which has been serving as a proxy for Rubin for many years. FPACK was positioned to support numerous ground-based projects during its early days, and there are several archives worth mining for benchmarks.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There are a couple of additional options to explore:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Astronomical data (including from cubesats) can be both imaging data (including data cubes) but also binary tables. Both may be packaged as multi-extension formats. FPACK also supports
the FITS tiled table compression convention, and I’d recommend you consider doing the same. See the link in the third paragraph of
<a href="https://heasarc.gsfc.nasa.gov/fitsio/fpack/">https://heasarc.gsfc.nasa.gov/fitsio/fpack/</a><o:p></o:p></li></ol>
<p class="MsoListParagraph"><o:p> </o:p></p>
<ol style="margin-top:0in" start="2" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">You say (section 4.6): “<i>we defer the nuanced practical considerations of space-compatible hardware and runtime optimization to future investigations, noting that all presented algorithms
leverage unoptimized reference implementations</i>”. I would suggest that the use cases here extend to ground-based facilities, as well. Hardware-accelerated compression (that is, “efficient data representation”) could very productively be moved closer to
the detector controllers. This is similar to cameras with built-in GPS modules for occultation work, or hardware time capture (e.g.,
<a href="https://arxiv.org/pdf/1807.01370">https://arxiv.org/pdf/1807.01370</a>). Which is to say, I wouldn’t defer such practicalities for very long.<o:p></o:p></li></ol>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m available to chat most weeks, and we should see who else can be looped in. This should be a simple addition to the FITS standard, though exactly what process needs to be followed these days would need to be settled through the IAU (Comm
B2) FITS WG.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best wishes!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Rob<o:p></o:p></p>
<p class="MsoNormal"><a href="mailto:rseaman@arizona.edu">rseaman@arizona.edu</a><o:p></o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 5/9/25, 8:31 AM, "William Pence" wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Helvetica"><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="margin-left:.5in;text-align:center">
<span style="font-family:Helvetica">
<hr size="0" width="100%" align="center">
</span></div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<br>
FYI<br>
<br>
Begin forwarded message:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<b>From:</b> "Jaffe, Tess (GSFC-6601)" <tess.jaffe@nasa.gov><br>
<b>Date:</b> May 9, 2025 at 10:42:13 AM EDT<br>
<b>To:</b> FTOOLS DIST <ftoolsdis@bigbang.gsfc.nasa.gov><br>
<b>Subject:</b> <b>Fw: [EXTERNAL] Re: new FITS Compression value?</b><o:p></o:p></p>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black">Anybody want to take on this discussion? My name is on the site as responsible, but I'm not qualified to comment on the FITS standard, much
less modify it. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="margin-left:.5in;text-align:center">
<hr size="0" width="75%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="color:black">From:</span></b><span style="color:black"> Perry Greenfield <perry@stsci.edu><br>
<b>Sent:</b> Friday, May 9, 2025 7:40 AM<br>
<b>To:</b> Joshua S Bloom <joshbloom@berkeley.edu><br>
<b>Cc:</b> Jaffe, Tess (GSFC-6601) <tess.jaffe@nasa.gov>; Rithwik Sudharsan <rithwik@berkeley.edu><br>
<b>Subject:</b> [EXTERNAL] Re: new FITS Compression value?</span> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.<br>
<br>
<br>
<br>
<br>
Hi Joshua,<br>
<br>
When you speak of lossy compression, are you retaining the current FITS scheme of tiling and quantizing, and then using JPEG-XL to compress the quantized tiles, or instead just applying the the compression to the whole image?<br>
<br>
Rick White was heavily involved in this, more than I was, so if you don’t mind I will pass this along to him.<br>
<br>
Thanks, Perry<br>
<br>
> On May 8, 2025, at 6:21 PM, Joshua S Bloom <joshbloom@berkeley.edu> wrote:<br>
><br>
> External Email - Use Caution<br>
><br>
> Dear Tess (cc Perry),<br>
><br>
> This is an inquiry to start a conversation about a potential addition to the FITS standard reserved value for compression.<br>
><br>
> Background: My student Rithwik (cc’ed here) has been leading an effort to improve image compression (focused on space applications) and we’re launching a cubesat later this year using his results.<br>
><br>
> After extensive testing we’ve found that the JPEG-XL codec offers significant improvements over RICE and HCOMPRESS for astronomical imaging both in the lossless and lossy contexts. (The RICE algorithm was mostly developed by Solomon Golomb in 1966, and later
analyzed for astronomy purposes by William Pence in 2009. Two decades later, there have been few to no changes.)<br>
><br>
> As part of our work, we recently published AstroCompress, a large benchmarking effort evaluating various compression codecs on 300 GB of diverse astronomy imagery:
<a href="https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopenreview.net%2Fpdf%3Fid%3DkQCHCkNk7s&data=05%7C02%7Cftoolsdis%40bigbang.gsfc.nasa.gov%7Cbf1d02b12f5b432c56b108dd8f07aa71%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638823985258528905%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Mr1wTCUYCOOac1pMHdaPA43n7HX7nYg9xU80p8MC%2FHA%3D&reserved=0">
https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopenreview.net%2Fpdf%3Fid%3DkQCHCkNk7s&data=05%7C02%7Ctess.jaffe%40nasa.gov%7C35cfb3af30e1479012c108dd8eee53b7%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638823876444759243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=l1xJzCNPPqvrJmwROEQ2ENIk%2FWFnMenFNYKv0sR6SV4%3D&reserved=0</a><br>
><br>
> We have conducted extensive testing across four datasets (James Webb, Hubble, SDSS, Keck) and across four codecs (JPEG-XL, JPEG-LS, JPEG-2000, RICE) and find JPEG-XL to be a clear ideal choice for the next generation of astronomy codecs.<br>
><br>
> We show that JPEG-XL can regularly achieve 10-15% better lossless compression ratios relative to RICE. More astoundingly, when quantizing data (under the FITS astropy default settings for compressing floating-point data), JPEG-XL code-lengths are 20% superior
to RICE. Finally, by passing a maxed-out “effort setting” to the JPEG-XL algorithm, these numbers increase by another 10%.<br>
><br>
> During the pull request process to astropy to extend its fitsio compression capability, Rithwik encountered some hesitance to do this extension unless the official FITS header reserved values were also extended.<br>
><br>
> Proposal: we’d like to propose reserving the ‘JPEGXL’ keyword value as an option for ‘ZCMPTYPE’ as outlined in table 36 of the FITS 4.0 standard:
<a href="https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf">
https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf</a><br>
> We would also add a parameter that denotes the ‘effort’ level of the JPEG-XL algorithm, which ranges from 0 to 9, and higher levels indicate improved compression performance.<br>
><br>
><br>
> Keyword<br>
> Newly Permitted Value(s)<br>
> Default<br>
> ZCMPTYPE<br>
> JPEGXL<br>
> -<br>
> ZNAME1<br>
> EFFORT<br>
> -<br>
> ZVAL1<br>
> {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}<br>
> 7<br>
><br>
> Would you be willing to have a chat about this, so we could present some further evidence and see if incorporating this into the FITS standard could be a possibility?<br>
><br>
> Thanks in advance,<br>
> Josh<br>
><br>
> ________________________________________________________<br>
><br>
> Joshua Bloom | Astronomy Dept | Professor | UC Berkeley | @profjsb<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>