[fitsbits] 16-bit floats {External}

[Francois-Xavier PINEAU]

Rust supports both f16 and f128 in its nightly (thus "experimental)" 
version:
* https://doc.rust-lang.org/std/primitive.f16.html
* https://doc.rust-lang.org/std/primitive.f128.html
See also the RFC book
https://rust-lang.github.io/rfcs/3453-f16-and-f128.html
pointing to some discussions
https://hackmd.io/@rust-lang-team/rkSQNidkp
It should only be a matter of time before both become standard.

But, I wonder if adding new BITPIX/TFORM values in HDUs
already described in the standard (Primary, BINTABLE, ...) does fit
with the FITS philosophy as it is described here:
https://fits.gsfc.nasa.gov/users_guide/users_guide/node7.html

First paragraph:
   "Any FITS reader should be able to cope with any FITS formatted file,
    skipping over portions (extensions) and ignoring keywords that the 
reader
    does not and need not understand"

Knowing how to interpret BITPIX and TFORM is mandatory to compute the 
byte size
of the data part of a HDU, and thus to be able to skip it.
True that the BITPIX description contains "[t]he absolute value is used in
computing the sizes of data structures", but there is a list of "valid" 
values
and I would expect a lot of implementations to raise an error when 
encountering
an undefined BITPIX value.
In addition, there is no way to know the byte size of an unknown TFORM 
value.

Then, true:
   "Changes in the FITS rules may add new structures that old software 
cannot
    handle. Revised software will be required for new standard 
extensions [...]"

but, followed by:
   "As far as is possible, however, FITS should be expanded in such a 
way that
    the old software will still be able to process those parts of the 
file which
    it is capable of handling.
    In such a case, software should not fail or give incorrect results when
    confronted with the new extension or conventions; it should simply 
ignore
    them and continue to process those parts of the file that it can
    understand."

"As far as is possible" is open to interpretation, but "new structures" and
"required for new standard extensions" does not seem to be compatible 
(to me)
with *current standard extension* and *new values of already defined 
keywords*.
Does the "FITS philosophy" cover the case of breaking changes in 
existing extensions?
Should it be reworded or expanded?

Quite clearly, I expect most (all/a lot of?) software to fail trying to 
read a FITS
file with previously not defined BITPIX or TFORM values (as mentioned
by Preben Grosboel).

I do understand that defining new extensions to simply add new BITPIX/TFORM
values seems excessive.


Also copying with the question of the real need for f16 (and/or f128) in 
FITS,
an alternative could be to start by introducing two new FITS conventions:
* one generic convention describing how to overwrite existing keywords 
values
   (OVERWRITE keyword?)
* one specific "half-float" convention introducing BITPIX=-16 / TFORM=H
   and using the generic "overwrite" convention to hide those values to
   unmaintained software.

An image containing half floats could be described as a flat byte array in
"regular" FITS (nothing interesting to be read for "half-float" unaware
readers), and decoded by *overwrite + half-float* aware readers.

Similarly, in a BINTABLE, a half float column could be a 2B column for
regular readers and a f16 for *overwrite + half-float* aware readers.

Time will let know is those conventions are worth being put in the standard
or not (or if the introduction of f16 in the standard is worth breaking
existing software).


Maybe defining new conventions here is also excessive.
If so, should the FITS philosophy page be (slightly) re-worded/extended?

If I remember correctly, long keywords have not been accepted because:
* it would have break existing software
* the HIERARCH convention already allows to define long keywords
But, if breaking changes like new BITPIX/TFORM values are accepted,
what about other breaking changes: long keywords, UTF-8 values, ... ?


I don't have any firm convictions; I'm simply trying to better understand
the FITS philosophy and to explore the various possibilities.


François-Xavier Pineau

Le 23/07/2025 à 22:40, Mohammad Akhlaghi via fitsbits a écrit :
> As mentioned before, it is also available in C:
>
> https://www.gnu.org/software/c-intro-and-ref/manual/html_node/Floating-Representations.html
>
> Cheers,
> Mohammad
>
>
>
> On July 23, 2025 10:22:12 PM GMT+02:00, "Barrett, Paul via fitsbits" 
> <fitsbits at listmgr.nrao.edu> wrote:
>
>     These formats are available in the Julia programming language.
>     Adding them to FitsFiles.jl should take minimal effort, say a few
>     days at most.
>
>      -- Paul
>
>
>     On Wed, Jul 23, 2025, 16:13 Lucio Chiappetti via fitsbits
>     <fitsbits at listmgr.nrao.edu> wrote:
>
>         On Wed, 23 Jul 2025, Barrett, Paul via fitsbits wrote:
>
>         > Yes, definitely. I have been advocating for half-precision
>         (16-bit)
>         > floating point for several years now for radio astronomy. In
>         addition,
>         > 128-bit floats
>
>         Are these formats supported by any major programming language ?
>
>         Is it worth supporting "exotic" formats, which might be
>         suitable for some
>         niche application, when most data producers often use
>         improperly 64-bit
>         all the times, even when overshoot ?
>
>         Historically FITS went the other way round (concentrating on
>         16 and 32,
>         later 64, when mainframres with 36 or 60 bits were around).
>
>         _______________________________________________
>         fitsbits mailing list
>         fitsbits at listmgr.nrao.edu
>         https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20250728/1f075313/attachment-0001.html>