<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<span style="font-family:monospace"><span style="color:#000000;background-color:#ffffff;">Thanks for all the very useful information posted so far.
</span><br>
I also had the opportunity to talk to Preben Grosbol here at ESO. <br>
I will try to hand on all this information to the right people at ALMA. <br>
Fortunately, the Plan A of ALMA so far is to stay with FITS. <br>
<br>
And if we (the participants in this email conversation) manage to <br>
restart a group of FITS standard keepers either in the existing form <br>
at NASA and the IAU, or in a new form, I think we would make <br>
a big contribution to the health of the astronomical community. <br>
A common image data format is fundamental. <br>
<br>
Maybe some kind of campaign is needed to inform the community about <br>
the advanced FITS features and about what the structure of a modern <br>
FITS image should look like. A training session could be offered<br>
at ADASS in 2024 and elsewhere.<br>
<br>
Then the next generation of data analysis programmers can make <br>
sure that the images coming out of the upcoming big observatories <br>
are optimally formatted and a good advertisement for FITS. <br>
<br>
Best, <br>
<br>
Dirk<br>
</span><br>
<br>
<div class="moz-cite-prefix">On 14.12.23 16:44, Seaman, Robert Lewis - (rseaman) wrote:<br>
</div>
<blockquote type="cite" cite="mid:BYAPR19MB2263D77C0573841A920669B2A78CA@BYAPR19MB2263.namprd19.prod.outlook.com">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}div.WordSection1
{page:WordSection1;}</style>
<table style="font-family: Arial, Helvetica, sans-serif; font-size:x-small" width="100%" cellspacing="0" cellpadding="10">
<tbody>
<tr>
<td style="border: 1px solid orange; border-left: 10px solid orange;" bgcolor="white">
This email was sent from outside of ESO from the address <strong>[<a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:rseaman@arizona.edu" moz-do-not-send="true">rseaman@arizona.edu</a>]</strong>. If it looks suspicious, please report it
to <a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:phishing@eso.org" moz-do-not-send="true">
phishing@eso.org</a>.<br>
</td>
</tr>
</tbody>
</table>
<table cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="font-size: 10px; line-height: 10px;" width="100%" height="10"> </td>
</tr>
</tbody>
</table>
<div class="WordSection1">
<p class="MsoNormal">Hi Dirk and all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The underlying governance of the FITS standard flows from the IAU (<a href="https://iau.org/science/scientific_bodies/working_groups/314/" moz-do-not-send="true" class="moz-txt-link-freetext">https://iau.org/science/scientific_bodies/working_groups/314/</a>),
and at least in the U.S., the infrastructure is founded on NASA’s FITS Support Office (<a href="https://fits.gsfc.nasa.gov" moz-do-not-send="true" class="moz-txt-link-freetext">https://fits.gsfc.nasa.gov</a>). It’s fair to say that activities surrounding FITS
have been quiet for a few years. Partially this is due to the retirement of various FITS notables, but more is due to the FITS community having addressed diverse concerns over the previous decade. As you say it is a mature format that works well.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It is hard to find another format as well defined as FITS, and certainly not as well defined in the peer-reviewed literature. FITS is controlled by the astronomical community. Other formats may expose projects to technological politics
from external sources. A number of astronomical community standards issues beyond FITS have been impacted by the COVID pandemic since these sorts of issues are the first to be mothballed.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It may well be time for the community to ramp up FITS and other data standards issues. I had to miss the recent FITS BoF at ADASS (<a href="https://adass2023.lpl.arizona.edu/events/bof-b301" moz-do-not-send="true" class="moz-txt-link-freetext">https://adass2023.lpl.arizona.edu/events/bof-b301</a>)
since I was in the session next door. There is perennial community interest in efficient data representation, whether for huge data cubes or tiny binary tables.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I suggest ALMA look into FITS tile compression, a binary table format (<a href="https://heasarc.gsfc.nasa.gov/fitsio/fpack/" moz-do-not-send="true" class="moz-txt-link-freetext">https://heasarc.gsfc.nasa.gov/fitsio/fpack/</a>), as an efficient
option and conceivably as a starting point to define a hierarchical storage strategy. Some of the tile compression usual suspects have already been discussing evaluating new algorithms, and conceivably, a hierarchical tiling scheme could be introduced at the
same time.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">FITS binary tables offer very generalized support. I built new infrastructure for my own project using CFITSIO binary table support during COVID that is in daily (well, nightly) used by my own survey. Indeed, binary tables are my own preferred
solution to the long-standing issue of FITS ASCII image headers: just define a bintable extension to hold the metadata with all the benefits of long keyword names, data-typing, etc.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In addition to standalone FITS-aware tools, an effort to provide the broad FITS standard support that CFITSIO has for other languages and programming frameworks could be a productive step forward for ALMA and other major projects.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Rob Seaman<o:p></o:p></p>
<p class="MsoNormal">Catalina Sky Survey<o:p></o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 12/13/23, 11:02 PM, "fitsbits" wrote:</p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
Hi,<br>
I am member of the ALMA team looking into the design the the "next generation"<br>
ALMA data processing and on behalf of that group (which contains many of your NRAO colleagues),<br>
I have been trying to learn more about the plans for the evolution of FITS.<br>
After first contacting Mark Calabretta and learning that he is retired since 10 years<br>
and then contacting the FITS working group at NASA and learning from Bill Pence<br>
that also the FITS working group has been mostly inactive since almost 10 years,<br>
I am now contacting you.<br>
<br>
ALMA will produce huge image cubes in the future with 10000 channels or more, possibly<br>
Terabytes in size. So people are wondering about the image storage format, and if FITS will<br>
be the right choice in the future.<br>
<br>
FITS is a very mature format which is well supported inside and even outside<br>
the astronomy world. Tests by the ALMA Archive subsystem scientist with FITS image cubes<br>
of up to 42 TB in size together with the CARTA visualization package,<br>
have shown that such a cube can still be loaded and viewed with reasonable response times.<br>
So in principle, FITS still seems to be up to the task.<br>
<br>
And having an archival format which is different from the format served to users<br>
requires translation which is expensive. Storing and serving in FITS might be the best<br>
also in the future.<br>
<br>
Do you know if there are plans to update/extend the FITS format to support some sort of<br>
hierarchical storage format like HDF5 under the hood?<br>
Could you point me to web pages where such developments are discussed?<br>
<br>
Thanks in advance for your help.<br>
<br>
Best regards and season's greetings!<br>
<br>
Dirk<br>
<br>
-- <br>
Dr. Dirk Petry, ALMA Regional Centre,<br>
ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,<br>
Germany, Phone: +49 89 3200-6511, Fax: -6358<br>
<br>
_______________________________________________<br>
fitsbits mailing list<br>
<a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:fitsbits@listmgr.nrao.edu" moz-do-not-send="true">fitsbits@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits" moz-do-not-send="true" class="moz-txt-link-freetext">https://listmgr.nrao.edu/mailman/listinfo/fitsbits</a></p>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Dr. Dirk Petry, ALMA Regional Centre,
ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,
Germany, Phone: +49 89 3200-6511, Fax: -6358</pre>
</body>
</html>