<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 2 1 2 1 8 4 8 7 8;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:581723222;
mso-list-type:hybrid;
mso-list-template-ids:617266224 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word;line-break:after-white-space">
<div class="WordSection1">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">By all means, have this discussion here on fitsbits. These questions/comments are meant as conversation starters:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">What distinguishes a real-world astronomer from an ADASS attendee? Astronomical projects have been able to choose different data formats since FITS was created. Somehow this rarely
happens. Hint: it’s not because the astronomers pay too much attention to programmers’ opinions.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Fix what? Whatever else is true, archival FITS data sets will need to continue to be supported. FITS binary tables will continue to provide sufficient room for evolution, but if you
think otherwise, use a different format. You may then find out what real-world astronomers think about that choice.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">2012? We had the same conversation in 1992.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">The old guard wants to hand off responsibility for the format(s) to earlier career folk. Will your management support a significant fraction of your time supporting FITS or any other
standards? How many FTEs are required on your team for such efforts?<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">The old guard invested quite significantly in time and “career points” in evolving the original standard through multi-extension FITS for mosaics, binary tables, tile compression,
etc. Some of us continue to be interested in evolving the standard. Can folks suggest a new round of data challenges to drive such efforts?<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">One can already implement all the changes suggested for FITS headers using binary tables. It is already perfectly legal FITS. Why has nobody simply defined a binary table to convey
such metadata? Pair it with minimal ASCII headers, a single 2880 record per extension, or suggest some SIMPLE=F variation to omit the headers.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">When evaluating fitness of purpose of FITS for some project, don’t start with something that resembles an early 1980’s single image format, rather a mid-2010’s binary table, e.g.,
see references at <a href="https://heasarc.gsfc.nasa.gov/fitsio/fpack/">https://heasarc.gsfc.nasa.gov/fitsio/fpack/</a><o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Several, perhaps all, of the IAU WG members must be on this list. The point isn’t about IAU policies; it’s that HDF and other competing formats (of which there are few, especially
for binary tables) are also owned by such organizations, and they are not ultimately responsible to the astronomical community. Ideas come from the communit(ies), sure, but standards come from boring committees.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">If there are minimal funds available at HEASARC, are there more available in your project? If your project or segment of the community wants to use something other than FITS, what
is stopping you from doing so? Whatever it is may reveal undiscovered engineering requirements or funding requirements. Are you in a position to pay the teams responsible for maintaining your favorite tools to support a new (or evolved) data format?<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Not all of us, perhaps not the majority, have ever embraced once FITS, always FITS, other than to suggest that archival data sets need to be supported. Nothing stops your project from
implementing tools that rely on different data formats. Whether the community later embraces them is not a function of cutting the legs out from under FITS, but of simply making your tools and alternate data formats useful on their own.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">There are a lot of papers from previous meetings, e.g.:
<a href="https://arxiv.org/pdf/1411.0996.pdf">https://arxiv.org/pdf/1411.0996.pdf</a> I don’t see this as a shortcoming.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">There are any number of perfectly legal ways to version FITS and build apps and libraries that are version-aware. There are no FITS police.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Evolving data formats will require evolution of infrastructure:
<a href="https://arxiv.org/pdf/2311.04272.pdf">https://arxiv.org/pdf/2311.04272.pdf</a><o:p></o:p></li></ul>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoNormal">Unlike biological evolution, there is no requirement for natural selection to kill off one format before another can arise. Build a better mouse. Where do you want to start, and how are you going to pay for it? What standards, tools, and
libraries will be needed to drive community adoption? What are the diverse use cases? How will such an effort be funded at your institution or in the community?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Rob<o:p></o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
</div>
<div id="mail-editor-reference-message-container">
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Hello,</p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I attended the FITS BoF at ADASS (actually, both of them) so I will take the opportunity to relay some thoughts from there and some of my own.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">The name of the session was "The Future of FITS and Other Standardized Astronomical Data Formats” and the sentiment in the room was definitely not “everything’s great, let’s grab some drinks.” Representatives from
observatories and archives addressed specific deficiencies in the FITS format and were unhappy about how some aspects really showed their age (headers, anyone?). Many (though I didn’t count hands at the time) were actively considering other formats, and many
would like to see updates to the FITS format. I would boil it down to this:</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">1. There are many real world needs astronomers have that the FITS format is not addressing.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">2. While many suggestions were made on how to fix this, all were dismissed as “that won’t work” or “we tried that” and the meeting ended with no path forward or solution.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">My favorite comment was at the very end of the meeting: “We had this exact conversation at ADASS in 2012”, so:</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">3. There is, and has been, virtually no movement to address these problems.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">So my takeaway is, yes, FITS is and has been great, but if you consider it bronzed don’t be surprised when people put it into a museum.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I’d like to address some of Rob’s points in his reply, but to be clear, these are by no means directed at Rob as I’ve heard these arguments in other places.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I don’t know anything about the IAU governance. Is there anyone on this mailing list in that working group? I would love to hear a statement on addressing the problems the community is having with FITS. Are there
plans to update things? Or is the working group tasked with keeping a specification? The same applies to HEASARC. It is my understanding that there are minimal funds available to managing the FITS format there.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">But regardless, changes come from the community, not standards bodies. It’s time to poll the community, find out what they need, and address the problem. Or task the community with solving the problems. Astropy
shows it can work. Saying “FITS is controlled by the astronomical community” may technically be true, but it really depends on how you define “community”.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Regarding the “FITS notables”, one question I’ve asked people in person and long toyed with asking on this mailing list is: “Of those in charge of steering the FITS format, what is the average age of this group
and does it increase by one every year?” (And to update it, “Does this number only ever decrease with retirement?”)</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Another thing I want to address is the “once FITS, always FITS” mantra. It seems like it could be interpreted as one of two ways:</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">A. Any program written that reads FITS files should always be able to read any FITS file, past and future.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">B. Any program that reads FITS files must be able to read all previously created FITS files.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">“A” means that a program written in 1985 that reads FITS files must be able to read all FITS files made after it, even if that program has long been abandoned. The B interpretation means that the format can evolve
and be versioned while ensuring that no data are orphaned. It is the correct interpretation. However, it’s clear that the steering committees/working groups have chosen the A interpretation, which means, well:</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:.5in">From our investigation, it is clear that FITS suffers from a lack of sufficient evolution. Original design decisions, such as the header byte layout and fixed character encoding made a certain sense at the time
FITS was founded. The later enshrinement of the FITS “Once FITS, always FITS” doctrine, which has been utilized to effectively freeze the format, was an mistake in our opinion. Adherence to the doctrine, and lack of any means to version the format in a machine-readable
manner, has stifled necessary change of FITS.</p>
</blockquote>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">From "Learning from FITS: Limitations in use in modern astronomical research”, <a href="https://arxiv.org/abs/1502.00996">https://arxiv.org/abs/1502.00996</a></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">So to directly answer your question Dirk, no, there are no plans to update/extend the FITS format to support, well, X. However it’s clear that there is a need. I’m happy to continue discussing how that need can
be addressed, whether it’s on this list or off.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Cheers,</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Demitri Muna</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Office of the Chief Science Data Officer</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">NASA HQ</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">(Opinions are my own.)</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
</p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:.5in">On Dec 13, 2023, at 03:50, Dirk Petry via fitsbits <fitsbits@listmgr.nrao.edu> wrote:</p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi,<br>
I am member of the ALMA team looking into the design the the "next generation"<br>
ALMA data processing and on behalf of that group (which contains many of your NRAO colleagues),<br>
I have been trying to learn more about the plans for the evolution of FITS.<br>
After first contacting Mark Calabretta and learning that he is retired since 10 years<br>
and then contacting the FITS working group at NASA and learning from Bill Pence<br>
that also the FITS working group has been mostly inactive since almost 10 years,<br>
I am now contacting you.<br>
<br>
ALMA will produce huge image cubes in the future with 10000 channels or more, possibly<br>
Terabytes in size. So people are wondering about the image storage format, and if FITS will<br>
be the right choice in the future.<br>
<br>
FITS is a very mature format which is well supported inside and even outside<br>
the astronomy world. Tests by the ALMA Archive subsystem scientist with FITS image cubes<br>
of up to 42 TB in size together with the CARTA visualization package,<br>
have shown that such a cube can still be loaded and viewed with reasonable response times.<br>
So in principle, FITS still seems to be up to the task.<br>
<br>
And having an archival format which is different from the format served to users<br>
requires translation which is expensive. Storing and serving in FITS might be the best<br>
also in the future.<br>
<br>
Do you know if there are plans to update/extend the FITS format to support some sort of<br>
hierarchical storage format like HDF5 under the hood?<br>
Could you point me to web pages where such developments are discussed?<br>
<br>
Thanks in advance for your help.<br>
<br>
Best regards and season's greetings!<br>
<br>
Dirk<br>
<br>
-- <br>
Dr. Dirk Petry, ALMA Regional Centre,<br>
ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,<br>
Germany, Phone: +49 89 3200-6511, Fax: -6358<br>
<br>
_______________________________________________<br>
fitsbits mailing list<br>
fitsbits@listmgr.nrao.edu<br>
https://listmgr.nrao.edu/mailman/listinfo/fitsbits</p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>