[fitsbits] 16-bit floats {External}

[Pierre Fernique]

Hi all,

In the context of the IVOA HiPS methodology 
(https://www.ivoa.net/documents/HiPS/), these arguments are particularly 
relevant. HiPS hierarchical method uses a tiling system in FITS format, 
and the right balance between volume and pixel access time is a key 
element in the success of this technology 
(https://www.ivoa.net/documents/HiPS/). The ability to code in 16-bit 
floats would bring undeniable additional flexibility. Both compact and 
fast to "decode". In this context, the lack of programming languages 
supporting natively this coding is not really a problem, as compactness 
is especially important for storage and transfer.

Cheers,
Pierre Fernique

Le 23/07/2025 à 16:48, Mohammad Akhlaghi via fitsbits a écrit :
> Hi Rob,
>
> Indeed, the tiled compression is very elegant and useful in many 
> contexts and will indeed decrease the file size without loss of 
> information if the extra 16 bits of a 32-bit floating point image are 
> intentionally set to zero. I am currently advocating tiled compression 
> for the data model of our ARRAKIHS mission (by ESA). I really 
> appreciate the wonderful work that has gone into into it and how 
> useful it is!
>
> The main problem with using tiled compression to replace the need for 
> a 16-bit floating point is the extra 16 bits. By default, they will 
> get filled with floating-point errors of the compiler/CPU. So to have 
> a good lossless compression, the writing library will have to ensure 
> to set them to zero before using the tiled compression.
>
> For long term archival, this extra effort makes sense and is 
> reasonable. But when you just want to write a temporary table as part 
> of a larger pipeline and read it in a later phase, all the overhead 
> (of making and then unpacking) can be non-negligible.
>
> So if there is a native 16-bit float type, it would help both in terms 
> of intermediate file products (storage), in-memory operations (less 
> RAM while the program is running), all without any overhead.
>
> Cheers,
> Mohammad
>
> On 7/23/25 4:30 PM, Seaman, Robert Lewis - (rseaman) via fitsbits wrote:
>> Hi all,
>>
>> Any such changes should be coordinated with the recent suggestion of 
>> adding JPEGXL support to tile compression.
>>
>> If the IAU FITS WG is going through a period of rebuilding, the 
>> responsibility for maintaining FITS falls to Commission B2.
>>
>> We looked into support for 16-bit floats when working on FPACK and 
>> tile compression. As I recall, it may not be as standardized as 
>> 32-bit and 64-bit IEEE floats. There were also some video (?) 
>> standards that used real-number representations that were 
>> fixed-point, not floating-point.
>>
>> 128-bit formats may need to coexist with complex number 
>> representations. The motivations for both shorter than usual or 
>> longer than usual floating-point representations in FITS should flow 
>> from explicit science and engineering use cases.
>>
>> Tiled 32-bit integer compression is actually quite elegant, and the 
>> noise-sensitive floating-point tile compression quite powerful, see 
>> Paper I and Paper II linked at 
>> https://heasarc.gsfc.nasa.gov/fitsio/fpack/. There are also a number 
>> of interesting issues associated with efficient table compression, 
>> such as the benefit of transposing tables and shuffling the bytes. 
>> Simply adding another data type won’t address all the issues.
>>
>> I am unaware of any coherent benchmarks comparing the speed of using 
>> 16-bit floats versus FPACK-style floating point compression. This 
>> should also be a prerequisite. Are 16-bit floats widely used in other 
>> scientific disciplines?
>>
>> Rob Seaman
>> Lunar and Planetary Laboratory
>>
>>
>>
>>
>>
>> On 7/23/25, 6:55 AM, "fitsbits" wrote:
>>
>> External Email
>>
>> On Wed 2025-07-23T13:34:54+0100 Thomas Robitaille via fitsbits hath 
>> writ:
>>> I am aware of some modern projects that would benefit from having 
>>> 16-bit
>>> floats, since they consider it to be sufficient in precision to 
>>> store very
>>> large datasets, and using 16-bit floats would perform a lot better than
>>> using compression on 32-bit floats for example, and 16-bit floats would
>>> allow a larger dynamic range than using 16-bit ints with BSCALE/BZERO.
>>
>> This is not a breaking change, and with little more discussion than we
>> already see today an informal agreement cold quickly go into use and
>> then be adopted.
>> I think the harder part about making this change is establishing
>> the auspices under which the IAU FITS working group currently acts.
>>
>> -- 
>> Steve Allen <sla at ucolick.org> WGS-84 (GPS)
>> UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat 
>> +36.99855
>> 1156 High Street Voice: +1 831 459 3046 Lng -122.06015
>> Santa Cruz, CA 95064 
>> https://www.ucolick.org/~sla/<https://www.ucolick.org/~sla> Hgt +250 m
>>
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits<https://listmgr.nrao.edu/mailman/listinfo/fitsbits> 
>>
>>
>>
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits