[fitsbits] FITS evolution {External}

Fri Dec 15 14:02:30 EST 2023

Hi all,

By all means, have this discussion here on fitsbits. These questions/comments are meant as conversation starters:

  *   What distinguishes a real-world astronomer from an ADASS attendee? Astronomical projects have been able to choose different data formats since FITS was created. Somehow this rarely happens. Hint: it’s not because the astronomers pay too much attention to programmers’ opinions.
  *   Fix what? Whatever else is true, archival FITS data sets will need to continue to be supported. FITS binary tables will continue to provide sufficient room for evolution, but if you think otherwise, use a different format. You may then find out what real-world astronomers think about that choice.
  *   2012? We had the same conversation in 1992.
  *   The old guard wants to hand off responsibility for the format(s) to earlier career folk. Will your management support a significant fraction of your time supporting FITS or any other standards? How many FTEs are required on your team for such efforts?
  *   The old guard invested quite significantly in time and “career points” in evolving the original standard through multi-extension FITS for mosaics, binary tables, tile compression, etc. Some of us continue to be interested in evolving the standard. Can folks suggest a new round of data challenges to drive such efforts?
  *   One can already implement all the changes suggested for FITS headers using binary tables. It is already perfectly legal FITS. Why has nobody simply defined a binary table to convey such metadata? Pair it with minimal ASCII headers, a single 2880 record per extension, or suggest some SIMPLE=F variation to omit the headers.
  *   When evaluating fitness of purpose of FITS for some project, don’t start with something that resembles an early 1980’s single image format, rather a mid-2010’s binary table, e.g., see references at https://heasarc.gsfc.nasa.gov/fitsio/fpack/
  *   Several, perhaps all, of the IAU WG members must be on this list. The point isn’t about IAU policies; it’s that HDF and other competing formats (of which there are few, especially for binary tables) are also owned by such organizations, and they are not ultimately responsible to the astronomical community. Ideas come from the communit(ies), sure, but standards come from boring committees.
  *   If there are minimal funds available at HEASARC, are there more available in your project? If your project or segment of the community wants to use something other than FITS, what is stopping you from doing so? Whatever it is may reveal undiscovered engineering requirements or funding requirements. Are you in a position to pay the teams responsible for maintaining your favorite tools to support a new (or evolved) data format?
  *   Not all of us, perhaps not the majority, have ever embraced once FITS, always FITS, other than to suggest that archival data sets need to be supported. Nothing stops your project from implementing tools that rely on different data formats. Whether the community later embraces them is not a function of cutting the legs out from under FITS, but of simply making your tools and alternate data formats useful on their own.
  *   There are a lot of papers from previous meetings, e.g.: https://arxiv.org/pdf/1411.0996.pdf  I don’t see this as a shortcoming.
  *   There are any number of perfectly legal ways to version FITS and build apps and libraries that are version-aware. There are no FITS police.
  *   Evolving data formats will require evolution of infrastructure: https://arxiv.org/pdf/2311.04272.pdf

Unlike biological evolution, there is no requirement for natural selection to kill off one format before another can arise. Build a better mouse. Where do you want to start, and how are you going to pay for it? What standards, tools, and libraries will be needed to drive community adoption? What are the diverse use cases? How will such an effort be funded at your institution or in the community?

Rob

Hello,

I attended the FITS BoF at ADASS (actually, both of them) so I will take the opportunity to relay some thoughts from there and some of my own.

The name of the session was "The Future of FITS and Other Standardized Astronomical Data Formats” and the sentiment in the room was definitely not “everything’s great, let’s grab some drinks.” Representatives from observatories and archives addressed specific deficiencies in the FITS format and were unhappy about how some aspects really showed their age (headers, anyone?). Many (though I didn’t count hands at the time) were actively considering other formats, and many would like to see updates to the FITS format. I would boil it down to this:

1. There are many real world needs astronomers have that the FITS format is not addressing.
2. While many suggestions were made on how to fix this, all were dismissed as “that won’t work” or “we tried that” and the meeting ended with no path forward or solution.

My favorite comment was at the very end of the meeting: “We had this exact conversation at ADASS in 2012”, so:

3. There is, and has been, virtually no movement to address these problems.

So my takeaway is, yes, FITS is and has been great, but if you consider it bronzed don’t be surprised when people put it into a museum.

I’d like to address some of Rob’s points in his reply, but to be clear, these are by no means directed at Rob as I’ve heard these arguments in other places.

I don’t know anything about the IAU governance. Is there anyone on this mailing list in that working group? I would love to hear a statement on addressing the problems the community is having with FITS. Are there plans to update things? Or is the working group tasked with keeping a specification? The same applies to HEASARC. It is my understanding that there are minimal funds available to managing the FITS format there.

But regardless, changes come from the community, not standards bodies. It’s time to poll the community, find out what they need, and address the problem. Or task the community with solving the problems. Astropy shows it can work. Saying “FITS is controlled by the astronomical community” may technically be true, but it really depends on how you define “community”.

Regarding the “FITS notables”, one question I’ve asked people in person and long toyed with asking on this mailing list is: “Of those in charge of steering the FITS format, what is the average age of this group and does it increase by one every year?” (And to update it, “Does this number only ever decrease with retirement?”)

Another thing I want to address is the “once FITS, always FITS” mantra. It seems like it could be interpreted as one of two ways:

A. Any program written that reads FITS files should always be able to read any FITS file, past and future.
B. Any program that reads FITS files must be able to read all previously created FITS files.

“A” means that a program written in 1985 that reads FITS files must be able to read all FITS files made after it, even if that program has long been abandoned. The B interpretation means that the format can evolve and be versioned while ensuring that no data are orphaned. It is the correct interpretation. However, it’s clear that the steering committees/working groups have chosen the A interpretation, which means, well:

>From our investigation, it is clear that FITS suffers from a lack of sufficient evolution. Original design decisions, such as the header byte layout and fixed character encoding made a certain sense at the time FITS was founded. The later enshrinement of the FITS “Once FITS, always FITS” doctrine, which has been utilized to effectively freeze the format, was an mistake in our opinion. Adherence to the doctrine, and lack of any means to version the format in a machine-readable manner, has stifled necessary change of FITS.

>From "Learning from FITS: Limitations in use in modern astronomical research”, https://arxiv.org/abs/1502.00996

So to directly answer your question Dirk, no, there are no plans to update/extend the FITS format to support, well, X. However it’s clear that there is a need. I’m happy to continue discussing how that need can be addressed, whether it’s on this list or off.

Cheers,

Demitri Muna
Office of the Chief Science Data Officer
NASA HQ

(Opinions are my own.)

On Dec 13, 2023, at 03:50, Dirk Petry via fitsbits <fitsbits at listmgr.nrao.edu> wrote:

Hi,
I am member of the ALMA team looking into the design the the "next generation"
ALMA data processing and on behalf of that group (which contains many of your NRAO colleagues),
I have been trying to learn more about the plans for the evolution of FITS.
After first contacting Mark Calabretta and learning that he is retired since 10 years
and then contacting the FITS working group at NASA and learning from Bill Pence
that also the FITS working group has been mostly inactive since almost 10 years,
I am now contacting you.

ALMA will produce huge image cubes in the future with 10000 channels or more, possibly
Terabytes in size. So people are wondering about the image storage format, and if FITS will
be the right choice in the future.

FITS is a very mature format which is well supported inside and even outside
the astronomy world. Tests by the ALMA Archive subsystem scientist with FITS image cubes
of up to 42 TB in size together with the CARTA visualization package,
have shown that such a cube can still be loaded and viewed with reasonable response times.
So in principle, FITS still seems to be up to the task.

And having an archival format which is different from the format served to users
requires translation which is expensive. Storing and serving in FITS might be the best
also in the future.

Do you know if there are plans to update/extend the FITS format to support some sort of
hierarchical storage format like HDF5 under the hood?
Could you point me to web pages where such developments are discussed?

Thanks in advance for your help.

Best regards and season's greetings!

Dirk

--
Dr. Dirk Petry, ALMA Regional Centre,
ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,
Germany, Phone: +49 89 3200-6511, Fax: -6358

_______________________________________________
fitsbits mailing list
fitsbits at listmgr.nrao.edu
https://listmgr.nrao.edu/mailman/listinfo/fitsbits

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20231215/76912913/attachment-0001.html>