[fitsbits] FITS evolution {External}

Sun Dec 17 15:13:53 EST 2023

Demitri,

I wanted to comment on a few of the issues you raised:

1.  “some aspects [of FITS] really show their age (headers, anyone?)”

The FITS header design is based on the 80-character IBM punched cards that we used back in the 1980s, which certainly qualifies as archaic.  However, FITS conventions have been developed to work around the most serious limitations. Now one can create keywords whose names are  40 characters or more long, and string-valued keywords can have arbitrarily long character string values that are continued over multiple header records.  Another commonly heard complaint about FITS headers is that if there is no room in the header to append another keyword, then the following data unit and any following extensions must be recopied in order to insert another empty FITS block in the header.  However, this should not be necessary if one simply preallocates enough scratch space in the header when it is first created to hold the anticipated total number of keyword that will eventually be written.

2.  “I don’t know anything about the IAU governance.”

Since the FITS  community has been largely inactive since 2016, it is understandable that many people don’t know or have forgotten this. Astronomers started using the FITS format in about 1980 and the IAU created the FITS Working Group in 1988 to “maintain the existing FITS standards and to review, approve and maintain future extensions to FITS”.   The FITS WG is fairly large with 20+ members with expertise in a broad range of astronomical sub fields to be able to carefully review any proposed changes to the FITS Standard document. Note that the FITS WG is too big and unwieldy to develop new proposals for improving FITS by itself;  it relies on smaller groups of technical experts to submit detailed proposals to improve the FITS format. 

The last significant work by the FITS WG occurred in 2016. Since then it has not received any new proposals to consider. Now, in 2023, only about a third of the previous WG members are still actively working.  So, if and when any new proposal to improve the FITS format is developed, the IAU (specifically the Data Representation Working Group under Commission B2) would need to appoint new members to the FITS WG to review that proposal. 

3. “It is my understanding that there are minimal funds available to managing the FITS format [at the HEASARC].”

As a continuing emeritus at the HEASARC at the NASA Goddard Space Flight Center, I can assure you that this is not true.  The HEASARC continues to fully fund the maintenance and enhancement of its large collection of FITS software (CFITSIO, FV, FPACK, FVERIFY, FTOOLS, etc.) and hosting the FITS Support Office website. Incidentally, CFITSIO is now on GitHub where anyone can submit bug fixes or enhancements.

4. “Once FITS always FITS”

This often misunderstood mantra comes directly from the following sentence in section 3.7 of the FITS standard: “Any structure that is a valid FITS structure shall remain a valid FITS structure at all future times.”  Simply put, this just means that the FITS format cannot be changed in any way that would cause existing FITS files to no longer be valid. The FITS Standard places no requirements at all on what types of FITS files that software programs should support. Thus both of your A and B interpretations of this mantra are false.

Regards,

Bill Pence
Emeritus, HEASARC at NASA/GSFC

> On Dec 15, 2023, at 11:37 AM, Demitri Muna via fitsbits <fitsbits at listmgr.nrao.edu> wrote:
> Hello,
> 
> I attended the FITS BoF at ADASS (actually, both of them) so I will take the opportunity to relay some thoughts from there and some of my own.
> 
> The name of the session was "The Future of FITS and Other Standardized Astronomical Data Formats” and the sentiment in the room was definitely not “everything’s great, let’s grab some drinks.” Representatives from observatories and archives addressed specific deficiencies in the FITS format and were unhappy about how some aspects really showed their age (headers, anyone?). Many (though I didn’t count hands at the time) were actively considering other formats, and many would like to see updates to the FITS format. I would boil it down to this:
> 
> 1. There are many real world needs astronomers have that the FITS format is not addressing.
> 2. While many suggestions were made on how to fix this, all were dismissed as “that won’t work” or “we tried that” and the meeting ended with no path forward or solution.
> 
> My favorite comment was at the very end of the meeting: “We had this exact conversation at ADASS in 2012”, so:
> 
> 3. There is, and has been, virtually no movement to address these problems.
> 
> So my takeaway is, yes, FITS is and has been great, but if you consider it bronzed don’t be surprised when people put it into a museum.
> 
> I’d like to address some of Rob’s points in his reply, but to be clear, these are by no means directed at Rob as I’ve heard these arguments in other places.
> 
> I don’t know anything about the IAU governance. Is there anyone on this mailing list in that working group? I would love to hear a statement on addressing the problems the community is having with FITS. Are there plans to update things? Or is the working group tasked with keeping a specification? The same applies to HEASARC. It is my understanding that there are minimal funds available to managing the FITS format there.
> 
> But regardless, changes come from the community, not standards bodies. It’s time to poll the community, find out what they need, and address the problem. Or task the community with solving the problems. Astropy shows it can work. Saying “FITS is controlled by the astronomical community” may technically be true, but it really depends on how you define “community”.
> 
> Regarding the “FITS notables”, one question I’ve asked people in person and long toyed with asking on this mailing list is: “Of those in charge of steering the FITS format, what is the average age of this group and does it increase by one every year?” (And to update it, “Does this number only ever decrease with retirement?”)
> 
> Another thing I want to address is the “once FITS, always FITS” mantra. It seems like it could be interpreted as one of two ways:
> 
> A. Any program written that reads FITS files should always be able to read any FITS file, past and future.
> B. Any program that reads FITS files must be able to read all previously created FITS files.
> 
> “A” means that a program written in 1985 that reads FITS files must be able to read all FITS files made after it, even if that program has long been abandoned. The B interpretation means that the format can evolve and be versioned while ensuring that no data are orphaned. It is the correct interpretation. However, it’s clear that the steering committees/working groups have chosen the A interpretation, which means, well:
> 
>> From our investigation, it is clear that FITS suffers from a lack of sufficient evolution. Original design decisions, such as the header byte layout and fixed character encoding made a certain sense at the time FITS was founded. The later enshrinement of the FITS “Once FITS, always FITS” doctrine, which has been utilized to effectively freeze the format, was an mistake in our opinion. Adherence to the doctrine, and lack of any means to version the format in a machine-readable manner, has stifled necessary change of FITS.
> 
> From "Learning from FITS: Limitations in use in modern astronomical research”, https://arxiv.org/abs/1502.00996
> 
> So to directly answer your question Dirk, no, there are no plans to update/extend the FITS format to support, well, X. However it’s clear that there is a need. I’m happy to continue discussing how that need can be addressed, whether it’s on this list or off.
> 
> Cheers,
> 
> Demitri Muna
> Office of the Chief Science Data Officer
> NASA HQ
> 
> (Opinions are my own.)
> 
> 
> 
>> On Dec 13, 2023, at 03:50, Dirk Petry via fitsbits <fitsbits at listmgr.nrao.edu> wrote:
>> 
>> Hi,
>> I am member of the ALMA team looking into the design the the "next generation"
>> ALMA data processing and on behalf of that group (which contains many of your NRAO colleagues),
>> I have been trying to learn more about the plans for the evolution of FITS.
>> After first contacting Mark Calabretta and learning that he is retired since 10 years
>> and then contacting the FITS working group at NASA and learning from Bill Pence
>> that also the FITS working group has been mostly inactive since almost 10 years,
>> I am now contacting you.
>> 
>> ALMA will produce huge image cubes in the future with 10000 channels or more, possibly
>> Terabytes in size. So people are wondering about the image storage format, and if FITS will
>> be the right choice in the future.
>> 
>> FITS is a very mature format which is well supported inside and even outside
>> the astronomy world. Tests by the ALMA Archive subsystem scientist with FITS image cubes
>> of up to 42 TB in size together with the CARTA visualization package,
>> have shown that such a cube can still be loaded and viewed with reasonable response times.
>> So in principle, FITS still seems to be up to the task.
>> 
>> And having an archival format which is different from the format served to users
>> requires translation which is expensive. Storing and serving in FITS might be the best
>> also in the future.
>> 
>> Do you know if there are plans to update/extend the FITS format to support some sort of
>> hierarchical storage format like HDF5 under the hood?
>> Could you point me to web pages where such developments are discussed?
>> 
>> Thanks in advance for your help.
>> 
>> Best regards and season's greetings!
>> 
>> Dirk
>> 
>> --
>> Dr. Dirk Petry, ALMA Regional Centre,
>> ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,
>> Germany, Phone: +49 89 3200-6511, Fax: -6358
>> 
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
> 
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20231217/edfd051c/attachment.html>