[fitsbits] FITS evolution {External}

[Dirk Petry]

Thanks for all the very useful information posted so far.
I also had the opportunity to talk to Preben Grosbol here at ESO.
I will try to hand on all this information to the right people at ALMA.
Fortunately, the Plan A of ALMA so far is to stay with FITS.

And if we (the participants in this email conversation) manage to
restart a group of FITS standard keepers either in the existing form
at NASA and the IAU, or in a new form, I think we would make
a big contribution to the health of the astronomical community.
A common image data format is fundamental.

Maybe some kind of campaign is needed to inform the community about
the advanced FITS features and about what the structure of a modern
FITS image should look like. A training session could be offered
at ADASS in 2024 and elsewhere.

Then the next generation of data analysis programmers can make
sure that the images coming out of the upcoming big observatories
are optimally formatted and a good advertisement for FITS.

Best,

Dirk


On 14.12.23 16:44, Seaman, Robert Lewis - (rseaman) wrote:
This email was sent from outside of ESO from the address [rseaman at arizona.edu<mailto:rseaman at arizona.edu>]. If it looks suspicious, please report it to phishing at eso.org<mailto:phishing at eso.org>.


Hi Dirk and all,

The underlying governance of the FITS standard flows from the IAU (https://iau.org/science/scientific_bodies/working_groups/314/), and at least in the U.S., the infrastructure is founded on NASA’s FITS Support Office (https://fits.gsfc.nasa.gov). It’s fair to say that activities surrounding FITS have been quiet for a few years. Partially this is due to the retirement of various FITS notables, but more is due to the FITS community having addressed diverse concerns over the previous decade. As you say it is a mature format that works well.

It is hard to find another format as well defined as FITS, and certainly not as well defined in the peer-reviewed literature. FITS is controlled by the astronomical community. Other formats may expose projects to technological politics from external sources. A number of astronomical community standards issues beyond FITS have been impacted by the COVID pandemic since these sorts of issues are the first to be mothballed.

It may well be time for the community to ramp up FITS and other data standards issues. I had to miss the recent FITS BoF at ADASS (https://adass2023.lpl.arizona.edu/events/bof-b301) since I was in the session next door. There is perennial community interest in efficient data representation, whether for huge data cubes or tiny binary tables.

I suggest ALMA look into FITS tile compression, a binary table format (https://heasarc.gsfc.nasa.gov/fitsio/fpack/), as an efficient option and conceivably as a starting point to define a hierarchical storage strategy. Some of the tile compression usual suspects have already been discussing evaluating new algorithms, and conceivably, a hierarchical tiling scheme could be introduced at the same time.

FITS binary tables offer very generalized support. I built new infrastructure for my own project using CFITSIO binary table support during COVID that is in daily (well, nightly) used by my own survey. Indeed, binary tables are my own preferred solution to the long-standing issue of FITS ASCII image headers: just define a bintable extension to hold the metadata with all the benefits of long keyword names, data-typing, etc.

In addition to standalone FITS-aware tools, an effort to provide the broad FITS standard support that CFITSIO has for other languages and programming frameworks could be a productive step forward for ALMA and other major projects.

Rob Seaman
Catalina Sky Survey



On 12/13/23, 11:02 PM, "fitsbits" wrote:

Hi,
I am member of the ALMA team looking into the design the the "next generation"
ALMA data processing and on behalf of that group (which contains many of your NRAO colleagues),
I have been trying to learn more about the plans for the evolution of FITS.
After first contacting Mark Calabretta and learning that he is retired since 10 years
and then contacting the FITS working group at NASA and learning from Bill Pence
that also the FITS working group has been mostly inactive since almost 10 years,
I am now contacting you.

ALMA will produce huge image cubes in the future with 10000 channels or more, possibly
Terabytes in size. So people are wondering about the image storage format, and if FITS will
be the right choice in the future.

FITS is a very mature format which is well supported inside and even outside
the astronomy world. Tests by the ALMA Archive subsystem scientist with FITS image cubes
of up to 42 TB in size together with the CARTA visualization package,
have shown that such a cube can still be loaded and viewed with reasonable response times.
So in principle, FITS still seems to be up to the task.

And having an archival format which is different from the format served to users
requires translation which is expensive. Storing and serving in FITS might be the best
also in the future.

Do you know if there are plans to update/extend the FITS format to support some sort of
hierarchical storage format like HDF5 under the hood?
Could you point me to web pages where such developments are discussed?

Thanks in advance for your help.

Best regards and season's greetings!

Dirk

--
Dr. Dirk Petry, ALMA Regional Centre,
ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,
Germany, Phone: +49 89 3200-6511, Fax: -6358

_______________________________________________
fitsbits mailing list
fitsbits at listmgr.nrao.edu<mailto:fitsbits at listmgr.nrao.edu>
https://listmgr.nrao.edu/mailman/listinfo/fitsbits


--
Dr. Dirk Petry, ALMA Regional Centre,
ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,
Germany, Phone: +49 89 3200-6511, Fax: -6358
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20231217/7a7fc89b/attachment.html>