[fitsbits] FITS evolution {External}

[Demitri Muna]

Hello,

I attended the FITS BoF at ADASS (actually, both of them) so I will take the opportunity to relay some thoughts from there and some of my own.

The name of the session was "The Future of FITS and Other Standardized Astronomical Data Formats” and the sentiment in the room was definitely not “everything’s great, let’s grab some drinks.” Representatives from observatories and archives addressed specific deficiencies in the FITS format and were unhappy about how some aspects really showed their age (headers, anyone?). Many (though I didn’t count hands at the time) were actively considering other formats, and many would like to see updates to the FITS format. I would boil it down to this:

1. There are many real world needs astronomers have that the FITS format is not addressing.
2. While many suggestions were made on how to fix this, all were dismissed as “that won’t work” or “we tried that” and the meeting ended with no path forward or solution.

My favorite comment was at the very end of the meeting: “We had this exact conversation at ADASS in 2012”, so:

3. There is, and has been, virtually no movement to address these problems.

So my takeaway is, yes, FITS is and has been great, but if you consider it bronzed don’t be surprised when people put it into a museum.

I’d like to address some of Rob’s points in his reply, but to be clear, these are by no means directed at Rob as I’ve heard these arguments in other places.

I don’t know anything about the IAU governance. Is there anyone on this mailing list in that working group? I would love to hear a statement on addressing the problems the community is having with FITS. Are there plans to update things? Or is the working group tasked with keeping a specification? The same applies to HEASARC. It is my understanding that there are minimal funds available to managing the FITS format there.

But regardless, changes come from the community, not standards bodies. It’s time to poll the community, find out what they need, and address the problem. Or task the community with solving the problems. Astropy shows it can work. Saying “FITS is controlled by the astronomical community” may technically be true, but it really depends on how you define “community”.

Regarding the “FITS notables”, one question I’ve asked people in person and long toyed with asking on this mailing list is: “Of those in charge of steering the FITS format, what is the average age of this group and does it increase by one every year?” (And to update it, “Does this number only ever decrease with retirement?”)

Another thing I want to address is the “once FITS, always FITS” mantra. It seems like it could be interpreted as one of two ways:

A. Any program written that reads FITS files should always be able to read any FITS file, past and future.
B. Any program that reads FITS files must be able to read all previously created FITS files.

“A” means that a program written in 1985 that reads FITS files must be able to read all FITS files made after it, even if that program has long been abandoned. The B interpretation means that the format can evolve and be versioned while ensuring that no data are orphaned. It is the correct interpretation. However, it’s clear that the steering committees/working groups have chosen the A interpretation, which means, well:

> From our investigation, it is clear that FITS suffers from a lack of sufficient evolution. Original design decisions, such as the header byte layout and fixed character encoding made a certain sense at the time FITS was founded. The later enshrinement of the FITS “Once FITS, always FITS” doctrine, which has been utilized to effectively freeze the format, was an mistake in our opinion. Adherence to the doctrine, and lack of any means to version the format in a machine-readable manner, has stifled necessary change of FITS.

From "Learning from FITS: Limitations in use in modern astronomical research”, https://arxiv.org/abs/1502.00996

So to directly answer your question Dirk, no, there are no plans to update/extend the FITS format to support, well, X. However it’s clear that there is a need. I’m happy to continue discussing how that need can be addressed, whether it’s on this list or off.

Cheers,

Demitri Muna
Office of the Chief Science Data Officer
NASA HQ

(Opinions are my own.)



> On Dec 13, 2023, at 03:50, Dirk Petry via fitsbits <fitsbits at listmgr.nrao.edu> wrote:
> 
> Hi,
> I am member of the ALMA team looking into the design the the "next generation"
> ALMA data processing and on behalf of that group (which contains many of your NRAO colleagues),
> I have been trying to learn more about the plans for the evolution of FITS.
> After first contacting Mark Calabretta and learning that he is retired since 10 years
> and then contacting the FITS working group at NASA and learning from Bill Pence
> that also the FITS working group has been mostly inactive since almost 10 years,
> I am now contacting you.
> 
> ALMA will produce huge image cubes in the future with 10000 channels or more, possibly
> Terabytes in size. So people are wondering about the image storage format, and if FITS will
> be the right choice in the future.
> 
> FITS is a very mature format which is well supported inside and even outside
> the astronomy world. Tests by the ALMA Archive subsystem scientist with FITS image cubes
> of up to 42 TB in size together with the CARTA visualization package,
> have shown that such a cube can still be loaded and viewed with reasonable response times.
> So in principle, FITS still seems to be up to the task.
> 
> And having an archival format which is different from the format served to users
> requires translation which is expensive. Storing and serving in FITS might be the best
> also in the future.
> 
> Do you know if there are plans to update/extend the FITS format to support some sort of
> hierarchical storage format like HDF5 under the hood?
> Could you point me to web pages where such developments are discussed?
> 
> Thanks in advance for your help.
> 
> Best regards and season's greetings!
> 
> Dirk
> 
> -- 
> Dr. Dirk Petry, ALMA Regional Centre,
> ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,
> Germany, Phone: +49 89 3200-6511, Fax: -6358
> 
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20231215/7c5956c4/attachment.html>