[fitsbits] FITS evolution {External}

[Seaman, Robert Lewis - (rseaman)]

Hi Dirk and all,

The underlying governance of the FITS standard flows from the IAU (https://iau.org/science/scientific_bodies/working_groups/314/), and at least in the U.S., the infrastructure is founded on NASA’s FITS Support Office (https://fits.gsfc.nasa.gov). It’s fair to say that activities surrounding FITS have been quiet for a few years. Partially this is due to the retirement of various FITS notables, but more is due to the FITS community having addressed diverse concerns over the previous decade. As you say it is a mature format that works well.

It is hard to find another format as well defined as FITS, and certainly not as well defined in the peer-reviewed literature. FITS is controlled by the astronomical community. Other formats may expose projects to technological politics from external sources. A number of astronomical community standards issues beyond FITS have been impacted by the COVID pandemic since these sorts of issues are the first to be mothballed.

It may well be time for the community to ramp up FITS and other data standards issues. I had to miss the recent FITS BoF at ADASS (https://adass2023.lpl.arizona.edu/events/bof-b301) since I was in the session next door. There is perennial community interest in efficient data representation, whether for huge data cubes or tiny binary tables.

I suggest ALMA look into FITS tile compression, a binary table format (https://heasarc.gsfc.nasa.gov/fitsio/fpack/), as an efficient option and conceivably as a starting point to define a hierarchical storage strategy. Some of the tile compression usual suspects have already been discussing evaluating new algorithms, and conceivably, a hierarchical tiling scheme could be introduced at the same time.

FITS binary tables offer very generalized support. I built new infrastructure for my own project using CFITSIO binary table support during COVID that is in daily (well, nightly) used by my own survey. Indeed, binary tables are my own preferred solution to the long-standing issue of FITS ASCII image headers: just define a bintable extension to hold the metadata with all the benefits of long keyword names, data-typing, etc.

In addition to standalone FITS-aware tools, an effort to provide the broad FITS standard support that CFITSIO has for other languages and programming frameworks could be a productive step forward for ALMA and other major projects.

Rob Seaman
Catalina Sky Survey



On 12/13/23, 11:02 PM, "fitsbits" wrote:

Hi,
I am member of the ALMA team looking into the design the the "next generation"
ALMA data processing and on behalf of that group (which contains many of your NRAO colleagues),
I have been trying to learn more about the plans for the evolution of FITS.
After first contacting Mark Calabretta and learning that he is retired since 10 years
and then contacting the FITS working group at NASA and learning from Bill Pence
that also the FITS working group has been mostly inactive since almost 10 years,
I am now contacting you.

ALMA will produce huge image cubes in the future with 10000 channels or more, possibly
Terabytes in size. So people are wondering about the image storage format, and if FITS will
be the right choice in the future.

FITS is a very mature format which is well supported inside and even outside
the astronomy world. Tests by the ALMA Archive subsystem scientist with FITS image cubes
of up to 42 TB in size together with the CARTA visualization package,
have shown that such a cube can still be loaded and viewed with reasonable response times.
So in principle, FITS still seems to be up to the task.

And having an archival format which is different from the format served to users
requires translation which is expensive. Storing and serving in FITS might be the best
also in the future.

Do you know if there are plans to update/extend the FITS format to support some sort of
hierarchical storage format like HDF5 under the hood?
Could you point me to web pages where such developments are discussed?

Thanks in advance for your help.

Best regards and season's greetings!

Dirk

--
Dr. Dirk Petry, ALMA Regional Centre,
ESO, Karl-Schwarzschild-Strasse 2, 85748 Garching,
Germany, Phone: +49 89 3200-6511, Fax: -6358

_______________________________________________
fitsbits mailing list
fitsbits at listmgr.nrao.edu
https://listmgr.nrao.edu/mailman/listinfo/fitsbits
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20231214/c197b434/attachment-0001.html>