[fitsbits] CFITSIO vs document {External}

[Pierre Fernique]

Hi all,

Le 28/03/2024 à 00:43, Seaman, Robert Lewis - (rseaman) via fitsbits a 
écrit :
>
> Does Rubin have any other suggestions for updating the standard? By 
> all means make a list. The standard is not frozen, even if the 
> procedures may be a bit quaint.
>
> Rob
>
As the standard is not frozen 😉, I'd like to take this opportunity to 
recall the following discrepancies raised one year ago between the 
current description of the RICE decompression method in the standard 
(4.0), and the actual implementation in the latest version of CFITSIO 
(see below for details).

More generally, regarding the easiness / difficulty of implementing the 
new document's recommendations, it would be interesting to know which 
alternative libraries to CFITSIO have already partially or fully 
implemented the document compression/decompression recommendations. A 
new column in the table described here 
https://fits.gsfc.nasa.gov/fits_libraries.html would be very valuable 
for evaluating this question.

Best regards,
Pierre Fernique


Le 14/06/2023 à 18:42, William Pence a écrit :
> Pierre,
>
> You are correct that CFITSIO ended up using the value -/2147483646 
> instead of -//2147483647 /to represent zero-valued pixels in a 
> compressed image.  This was because the latter value was already 
> reserved in CFITSIO to represent null-valued pixels that have no 
> assigned value. As you note, this change has a small ripple effect on 
> the quantization of the other pixels in the image in that in a very 
> few cases a pixel will end up quantized into an adjacent bin because 
> of small (and insignificant) rounding errors.   Ideally, the FITS 
> Standard should be corrected to reflect this change.
>
(...)
>
> William Pence
> NASA, HEASARC (Emeritus)
>
>
>
>> On Jun 13, 2023, at 5:41 AM, Pierre Fernique via fitsbits 
>> <fitsbits at listmgr.nrao.edu> wrote:
>>
>> 
>>
>>
>> Dear Fits mailing list members,
>>
>> We (CDS and more precisely myself) ave encountered a few difficulties 
>> in implementing in java the RICE decompression codes described in the 
>> latest version of the FITS standard 
>> <https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf> in 
>> the context of updating our Hierarchical Progressive Survey (IVOA 
>> Standard HiPS 1.0 <https://www.ivoa.net/documents/HiPS/>) generation 
>> tools, and in particular Hipsgen 
>> <https://aladin.u-strasbg.fr/hips/#tools>.
>>
>> We realised that the images we were decompressing were not strictly 
>> identical to those decompressed by cfitsio-based tools. After 
>> investigation, it seems that the discrepancy was due to a variation 
>> in the way the ZERO_VALUE value was taken into account. In the Fits 
>> 4.0 document, the value used to encode the zero in the SUBTRACTIVE 
>> DITHER 2 method is 214748364*7* (page 47 "... /pixels in the 
>> floating-point image that are exactly equal to 0.0 are represented by 
>> the reserved value -2147483647 in the quantized integer array.../"). 
>> Except if I'm wrong, in the cfitsio library code, the value used is 
>> -214748364*6* (imcompress.c 
>> <https://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/imcompress.c> 
>> line 10 & 7679). One of the consequences of this divergence is the 
>> shift in the application of the random function (inhibited for the 
>> value ZERO_VALUE). And indeed, having adopted the constant used by 
>> cfitsio, we obtain the same results.
>>
>>
(...)
>>
>> At this stage, we decided to strictly follow the choices of the 
>> latest version of cfitsio (4.2.0)  (214748364*6 *and assuming the 
>> unique random function).
>>
>> If this feedback is indeed relevant, perhaps the Fits 4.0 document 
>> could be clarified on these two points to avoid any potential divergence.
>>
>> Best regards
>> Pierre Fernique
>> Centre de Données astronomiques de Strasbourg
>>
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20240328/e8c356c4/attachment.html>