From pebarrett at email.gwu.edu Mon Oct 27 11:13:00 2025 From: pebarrett at email.gwu.edu (Barrett, Paul) Date: Mon, 27 Oct 2025 11:13:00 -0400 Subject: [fitsbits] Adding fpzip floating point compression {External} Message-ID: I would like to suggest that FITS add the fpzip floating point compression algorithm to the FITS standard. fpzip is primarily designed for lossless compression but also has a provision for lossy compression. I will be adding it to FITSFiles.jl, a pure Julia implementation of the FITS standard. -- Paul -- Paul Barrett, PhD Department of Physics The George Washington University Washington, DC 20052 -------------- next part -------------- An HTML attachment was scrubbed... URL: From teuben at umd.edu Mon Oct 27 11:25:53 2025 From: teuben at umd.edu (peter teuben) Date: Mon, 27 Oct 2025 11:25:53 -0400 Subject: [fitsbits] Adding fpzip floating point compression {External} In-Reply-To: References: Message-ID: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> I'm not quite understanding how this would be implemented. Is this a new BITPIX ??? Or does it only apply to a new extension. like how BINTABLE was done? I get worried if we encode (compression) algorithms in order to decode FITS files. ?- peter On 10/27/25 11:13, Barrett, Paul via fitsbits wrote: > I would like to suggest that FITS add the fpzip > ?floating point compression > algorithm to the FITS standard. fpzip is primarily designed for > lossless compression but also has a provision for lossy compression. > > I will be adding it to FITSFiles.jl, a pure Julia implementation?of > the FITS standard. > > ?-- Paul > > > -- > Paul Barrett, PhD > Department of Physics > The George Washington University > Washington, DC 20052 > > > _______________________________________________ > fitsbits mailing list > fitsbits at listmgr.nrao.edu > https://listmgr.nrao.edu/mailman/listinfo/fitsbits -------------- next part -------------- An HTML attachment was scrubbed... URL: From rseaman at arizona.edu Mon Oct 27 11:37:04 2025 From: rseaman at arizona.edu (Seaman, Robert Lewis - (rseaman)) Date: Mon, 27 Oct 2025 15:37:04 +0000 Subject: [fitsbits] Adding fpzip floating point compression {External} In-Reply-To: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> References: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> Message-ID: Hi Paul, You do mean adding it as an option to FITS tile compression, right? This may include bintable compression, as well as image arrays. There have been a number of suggestions lately related to enabling additional algorithms for tile compression. It was always designed to support this. Any such that are adopted into FITS should (at least) be enabled for FPACK. It would be good if such algorithms could be evaluated similarly to as described in papers I and II linked at: https://heasarc.gsfc.nasa.gov/docs/software/fitsio/fpack/ (The website is still accessible, if not being maintained, during the gov?t shutdown.) Speed as well as compression ratio are important in efficient data representations. Decompression may be faster or slower than compression, etc. Rob Seaman Lunar and Planetary Laboratory University of Arizona ? On 10/27/25, 8:27?AM, "fitsbits" wrote: External Email ________________________________ I'm not quite understanding how this would be implemented. Is this a new BITPIX ? Or does it only apply to a new extension. like how BINTABLE was done? I get worried if we encode (compression) algorithms in order to decode FITS files. - peter On 10/27/25 11:13, Barrett, Paul via fitsbits wrote: I would like to suggest that FITS add the fpzip floating point compression algorithm to the FITS standard. fpzip is primarily designed for lossless compression but also has a provision for lossy compression. I will be adding it to FITSFiles.jl, a pure Julia implementation of the FITS standard. -- Paul -- Paul Barrett, PhD Department of Physics The George Washington University Washington, DC 20052 _______________________________________________ fitsbits mailing list fitsbits at listmgr.nrao.edu https://listmgr.nrao.edu/mailman/listinfo/fitsbits -------------- next part -------------- An HTML attachment was scrubbed... URL: From gpdf at ipac.caltech.edu Mon Oct 27 12:35:25 2025 From: gpdf at ipac.caltech.edu (Dubois-Felsmann, Gregory P.) Date: Mon, 27 Oct 2025 16:35:25 +0000 Subject: [fitsbits] Adding fpzip floating point compression {External} In-Reply-To: References: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> Message-ID: Paul, The "tile" in tile compression arises, in part, from the idea that for quantized compression there are advantages to aligning the quantization with local conditions across an image. It also facilitates accessing only part of a large image if all that's needed is a cutout. I'm not familiar with fpzip; can you comment on whether there are overhead issues that would push one toward compressing images as a whole or whether the "tile" approach matches up with how fpzip performs? I've found references that say that LLNL's "zfp" is designed for streaming processing, which would likely mean it would be well-matched to tiling as well. I wasn't able to find as clear a statement on their "fpzip", but that might be because I'm having (temporary, I assume) trouble accessing the original paper. Thanks, Gregory ________________________________________ From: fitsbits on behalf of Seaman, Robert Lewis - (rseaman) via fitsbits Sent: Monday, October 27, 2025 08:37 To: fitsbits at listmgr.nrao.edu Subject: Re: [fitsbits] Adding fpzip floating point compression {External} Hi Paul, You do mean adding it as an option to FITS tile compression, right? This may include bintable compression, as well as image arrays. There have been a number of suggestions lately related to enabling additional algorithms for tile compression. It was always designed to support this. Any such that are adopted into FITS should (at least) be enabled for FPACK. It would be good if such algorithms could be evaluated similarly to as described in papers I and II linked at: https://heasarc.gsfc.nasa.gov/docs/software/fitsio/fpack/ (The website is still accessible, if not being maintained, during the gov?t shutdown.) Speed as well as compression ratio are important in efficient data representations. Decompression may be faster or slower than compression, etc. Rob Seaman Lunar and Planetary Laboratory University of Arizona ? On 10/27/25, 8:27?AM, "fitsbits" wrote: External Email ________________________________ I'm not quite understanding how this would be implemented. Is this a new BITPIX ? Or does it only apply to a new extension. like how BINTABLE was done? I get worried if we encode (compression) algorithms in order to decode FITS files. - peter On 10/27/25 11:13, Barrett, Paul via fitsbits wrote: I would like to suggest that FITS add the fpzip floating point compression algorithm to the FITS standard. fpzip is primarily designed for lossless compression but also has a provision for lossy compression. I will be adding it to FITSFiles.jl, a pure Julia implementation of the FITS standard. -- Paul -- Paul Barrett, PhD Department of Physics The George Washington University Washington, DC 20052 _______________________________________________ fitsbits mailing list fitsbits at listmgr.nrao.edu https://listmgr.nrao.edu/mailman/listinfo/fitsbits From gpdf at ipac.caltech.edu Mon Oct 27 12:45:53 2025 From: gpdf at ipac.caltech.edu (Dubois-Felsmann, Gregory P.) Date: Mon, 27 Oct 2025 16:45:53 +0000 Subject: [fitsbits] Adding fpzip floating point compression {External} In-Reply-To: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> References: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> Message-ID: Hi, Peter, As Rob has also noted, we're so far assuming that Paul is suggesting extending the set of algorithms already supported by FITS. Have a look at section 10.4 (p. 50) in the FITS 4.00 document https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf for the currently standardized list. Gregory ________________________________________ From: fitsbits on behalf of peter teuben via fitsbits Sent: Monday, October 27, 2025 08:25 To: fitsbits at listmgr.nrao.edu Cc: peter teuben Subject: Re: [fitsbits] Adding fpzip floating point compression {External} I'm not quite understanding how this would be implemented. Is this a new BITPIX ? Or does it only apply to a new extension. like how BINTABLE was done? I get worried if we encode (compression) algorithms in order to decode FITS files. - peter On 10/27/25 11:13, Barrett, Paul via fitsbits wrote: I would like to suggest that FITS add the fpzip floating point compression algorithm to the FITS standard. fpzip is primarily designed for lossless compression but also has a provision for lossy compression. I will be adding it to FITSFiles.jl, a pure Julia implementation of the FITS standard. -- Paul -- Paul Barrett, PhD Department of Physics The George Washington University Washington, DC 20052 _______________________________________________ fitsbits mailing list fitsbits at listmgr.nrao.edu https://listmgr.nrao.edu/mailman/listinfo/fitsbits From pebarrett at email.gwu.edu Mon Oct 27 12:48:42 2025 From: pebarrett at email.gwu.edu (Barrett, Paul) Date: Mon, 27 Oct 2025 12:48:42 -0400 Subject: [fitsbits] Adding fpzip floating point compression {External} In-Reply-To: References: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> Message-ID: >From the fpzip website, the lossless (fpzip) algorithm achieves compression ratios of 1.5 - 4x depending on the data. It is also designed for streaming with little to no latency when decompressing. The lossy (zfp) algorithm achieves compression ratios of ~100x. The C/C++ library is freely available. -- Paul On Mon, Oct 27, 2025 at 12:35?PM Dubois-Felsmann, Gregory P. < gpdf at ipac.caltech.edu> wrote: > Paul, > > The "tile" in tile compression arises, in part, from the idea that for > quantized compression there are advantages to aligning the quantization > with local conditions across an image. It also facilitates accessing only > part of a large image if all that's needed is a cutout. > > I'm not familiar with fpzip; can you comment on whether there are overhead > issues that would push one toward compressing images as a whole or whether > the "tile" approach matches up with how fpzip performs? > > I've found references that say that LLNL's "zfp" is designed for streaming > processing, which would likely mean it would be well-matched to tiling as > well. I wasn't able to find as clear a statement on their "fpzip", but > that might be because I'm having (temporary, I assume) trouble accessing > the original paper. > > Thanks, > Gregory > > ________________________________________ > From: fitsbits on behalf of Seaman, > Robert Lewis - (rseaman) via fitsbits > Sent: Monday, October 27, 2025 08:37 > To: fitsbits at listmgr.nrao.edu > Subject: Re: [fitsbits] Adding fpzip floating point compression {External} > > Hi Paul, > > You do mean adding it as an option to FITS tile compression, right? This > may include bintable compression, as well as image arrays. > > There have been a number of suggestions lately related to enabling > additional algorithms for tile compression. It was always designed to > support this. Any such that are adopted into FITS should (at least) be > enabled for FPACK. It would be good if such algorithms could be evaluated > similarly to as described in papers I and II linked at: > https://heasarc.gsfc.nasa.gov/docs/software/fitsio/fpack/ > > (The website is still accessible, if not being maintained, during the > gov?t shutdown.) > > Speed as well as compression ratio are important in efficient data > representations. Decompression may be faster or slower than compression, > etc. > > Rob Seaman > Lunar and Planetary Laboratory > University of Arizona > > ? > > > On 10/27/25, 8:27?AM, "fitsbits" wrote: > > > External Email > > ________________________________ > > I'm not quite understanding how this would be implemented. Is this a new > BITPIX ? Or does it only apply to a new extension. like how BINTABLE was > done? > > I get worried if we encode (compression) algorithms in order to decode > FITS files. > > - peter > > On 10/27/25 11:13, Barrett, Paul via fitsbits wrote: > I would like to suggest that FITS add the fpzip< > https://computing.llnl.gov/projects/fpzip> floating point compression > algorithm to the FITS standard. fpzip is primarily designed for lossless > compression but also has a provision for lossy compression. > > I will be adding it to FITSFiles.jl, a pure Julia implementation of the > FITS standard. > > -- Paul > > > -- > Paul Barrett, PhD > Department of Physics > The George Washington University > Washington, DC 20052 > > > > _______________________________________________ > fitsbits mailing list > fitsbits at listmgr.nrao.edu > https://listmgr.nrao.edu/mailman/listinfo/fitsbits > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pebarrett at email.gwu.edu Mon Oct 27 12:51:05 2025 From: pebarrett at email.gwu.edu (Barrett, Paul) Date: Mon, 27 Oct 2025 12:51:05 -0400 Subject: [fitsbits] Adding fpzip floating point compression {External} In-Reply-To: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> References: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> Message-ID: Yes, the suggestion is to extend the set of algorithms to include the lossless (fpzip) and lossy (fpz) algorithms. -- Paul On Mon, Oct 27, 2025 at 11:34?AM peter teuben via fitsbits < fitsbits at listmgr.nrao.edu> wrote: > I'm not quite understanding how this would be implemented. Is this a new > BITPIX ? Or does it only apply to a new extension. like how BINTABLE was > done? > > I get worried if we encode (compression) algorithms in order to decode > FITS files. > > - peter > On 10/27/25 11:13, Barrett, Paul via fitsbits wrote: > > I would like to suggest that FITS add the fpzip > floating point compression > algorithm to the FITS standard. fpzip is primarily designed for lossless > compression but also has a provision for lossy compression. > > I will be adding it to FITSFiles.jl, a pure Julia implementation of the > FITS standard. > > -- Paul > > > -- > Paul Barrett, PhD > Department of Physics > The George Washington University > Washington, DC 20052 > > > _______________________________________________ > fitsbits mailing listfitsbits at listmgr.nrao.eduhttps://listmgr.nrao.edu/mailman/listinfo/fitsbits > > _______________________________________________ > fitsbits mailing list > fitsbits at listmgr.nrao.edu > https://listmgr.nrao.edu/mailman/listinfo/fitsbits > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rseaman at arizona.edu Mon Oct 27 13:18:52 2025 From: rseaman at arizona.edu (Seaman, Robert Lewis - (rseaman)) Date: Mon, 27 Oct 2025 17:18:52 +0000 Subject: [fitsbits] Adding fpzip floating point compression {External} In-Reply-To: References: <95929c10-8ddc-4083-ac06-dd4af76e40da@astro.umd.edu> Message-ID: Hi Paul, A way you can justify your suggestion is to repeat the various tests against astronomical data as described in http://adsabs.harvard.edu/abs/2010PASP..122.1065P and http://adsabs.harvard.edu/abs/2009PASP..121..414P No matter the algorithm and the data type, Shannon?s entropy will always govern the compression ratio that can be realized. Speed is at least as important as the compression ratio. Many lossy algorithms are tuned for appearance, not to preserve derived photometry, etc. And, I suppose, if we are still considering implementing half-precision floating point, you might want to evaluate the algorithms for those data in advance. Rob ? On 10/27/25, 9:52?AM, "fitsbits" wrote: External Email ________________________________ Yes, the suggestion is to extend the set of algorithms to include the lossless (fpzip) and lossy (fpz) algorithms. -- Paul On Mon, Oct 27, 2025 at 11:34?AM peter teuben via fitsbits > wrote: I'm not quite understanding how this would be implemented. Is this a new BITPIX ? Or does it only apply to a new extension. like how BINTABLE was done? I get worried if we encode (compression) algorithms in order to decode FITS files. - peter On 10/27/25 11:13, Barrett, Paul via fitsbits wrote: I would like to suggest that FITS add the fpzip floating point compression algorithm to the FITS standard. fpzip is primarily designed for lossless compression but also has a provision for lossy compression. I will be adding it to FITSFiles.jl, a pure Julia implementation of the FITS standard. -- Paul -- Paul Barrett, PhD Department of Physics The George Washington University Washington, DC 20052 _______________________________________________ fitsbits mailing list fitsbits at listmgr.nrao.edu https://listmgr.nrao.edu/mailman/listinfo/fitsbits _______________________________________________ fitsbits mailing list fitsbits at listmgr.nrao.edu https://listmgr.nrao.edu/mailman/listinfo/fitsbits -------------- next part -------------- An HTML attachment was scrubbed... URL: