<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Dirk and all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The underlying governance of the FITS standard flows from the IAU (<a href="https://iau.org/science/scientific_bodies/working_groups/314/">https://iau.org/science/scientific_bodies/working_groups/314/</a>), and at least in the U.S., the
infrastructure is founded on NASA’s FITS Support Office (<a href="https://fits.gsfc.nasa.gov">https://fits.gsfc.nasa.gov</a>). It’s fair to say that activities surrounding FITS have been quiet for a few years. Partially this is due to the retirement of various
FITS notables, but more is due to the FITS community having addressed diverse concerns over the previous decade. As you say it is a mature format that works well.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It is hard to find another format as well defined as FITS, and certainly not as well defined in the peer-reviewed literature. FITS is controlled by the astronomical community. Other formats may expose projects to technological politics
from external sources. A number of astronomical community standards issues beyond FITS have been impacted by the COVID pandemic since these sorts of issues are the first to be mothballed.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It may well be time for the community to ramp up FITS and other data standards issues. I had to miss the recent FITS BoF at ADASS (<a href="https://adass2023.lpl.arizona.edu/events/bof-b301">https://adass2023.lpl.arizona.edu/events/bof-b301</a>)
since I was in the session next door. There is perennial community interest in efficient data representation, whether for huge data cubes or tiny binary tables.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I suggest ALMA look into FITS tile compression, a binary table format (<a href="https://heasarc.gsfc.nasa.gov/fitsio/fpack/">https://heasarc.gsfc.nasa.gov/fitsio/fpack/</a>), as an efficient option and conceivably as a starting point to
define a hierarchical storage strategy. Some of the tile compression usual suspects have already been discussing evaluating new algorithms, and conceivably, a hierarchical tiling scheme could be introduced at the same time.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">FITS binary tables offer very generalized support. I built new infrastructure for my own project using CFITSIO binary table support during COVID that is in daily (well, nightly) used by my own survey. Indeed, binary tables are my own preferred
solution to the long-standing issue of FITS ASCII image headers: just define a bintable extension to hold the metadata with all the benefits of long keyword names, data-typing, etc.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In addition to standalone FITS-aware tools, an effort to provide the broad FITS standard support that CFITSIO has for other languages and programming frameworks could be a productive step forward for ALMA and other major projects.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Rob Seaman<o:p></o:p></p>
<p class="MsoNormal">Catalina Sky Survey<o:p></o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 12/13/23, 11:02 PM, "fitsbits" wrote:</p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
Hi,<br>
I am member of the ALMA team looking into the design the the "next generation"<br>
ALMA data processing and on behalf of that group (which contains many of your NRAO colleagues),<br>
I have been trying to learn more about the plans for the evolution of FITS.<br>
After first contacting Mark Calabretta and learning that he is retired since 10 years<br>
and then contacting the FITS working group at NASA and learning from Bill Pence<br>
that also the FITS working group has been mostly inactive since almost 10 years,<br>
I am now contacting you.<br>
<br>
ALMA will produce huge image cubes in the future with 10000 channels or more, possibly<br>
Terabytes in size. So people are wondering about the image storage format, and if FITS will<br>
be the right choice in the future.<br>
<br>
FITS is a very mature format which is well supported inside and even outside<br>
the astronomy world. Tests by the ALMA Archive subsystem scientist with FITS image cubes<br>
of up to 42 TB in size together with the CARTA visualization package,<br>
have shown that such a cube can still be loaded and viewed with reasonable response times.<br>
So in principle, FITS still seems to be up to the task.<br>
<br>
And having an archival format which is different from the format served to users<br>
requires translation which is expensive. Storing and serving in FITS might be the best<br>
also in the future.<br>
<br>
Do you know if there are plans to update/extend the FITS format to support some sort of<br>
hierarchical storage format like HDF5 under the hood?<br>
Could you point me to web pages where such developments are discussed?<br>
<br>
Thanks in advance for your help.<br>
<br>
Best regards and season's greetings!<br>
<br>
Dirk<br>
<br>
-- <br>
Dr. Dirk Petry, ALMA Regional Centre,<br>
ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,<br>
Germany, Phone: +49 89 3200-6511, Fax: -6358<br>
<br>
_______________________________________________<br>
fitsbits mailing list<br>
fitsbits@listmgr.nrao.edu<br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits">https://listmgr.nrao.edu/mailman/listinfo/fitsbits</a></p>
</div>
</div>
</div>
</div>
</body>
</html>