[fitsbits] Justifications for 64-bit integer FITS images

[William Pence]

This list contains various examples, collected with the help of some of
my colleagues, to justify adding support in FITS for 64-bit integer images.
These examples specifically deal with cases involving 64-bit integer FITS
images with BITPIX = 64, not FITS table columns.

A. Scientific justifications
----------------------------

These cases illustrate scientific problems that require images with more
than 32-bits of precision.

1.  Catalogs of astronomical sources or observations will soon (if not
already) contain more objects than can be represented with 32-bit integers.
If one computes  histograms of the parameters in these large
databases  then 32-bit  integers will not be sufficient in general to store
the histogram values in a FITS primary array.

2.  The precise measurement of time and time intervals can result in more
clock 'ticks' than can be represented by 32-bit integers. This has been used
to justify the need for 64-bit integer 'TIME' columns in tables, but it can
just as well be used to justify the need for 64-bit integer arrays.  For
example, one might construct a 1-D FITS primary array that contains a
sequence of times for some observed repetitive phenomena, such as the
observed times of the flux maximum in a rotating object.   Or, one might
have a N-satellite x N-interval 2-D array where each entry is the time 
received from a given GPS satellite as seen by some other satellite. Why
should researchers be forced to only represent these arrays of time values
in table columns and not also as a 1-dimensional FITS primary arrays or
image extensions?

3.  When binning or summing pixels in a CCD array the sums may exceed 2 GB.
For example, if one has a 10x10 array of 1024x1024 CCDs with an average
image pixel value greater than 2000, then one cannot create the 10x10
summed image using integer*4 pixels..

4.  Use of FITS arrays for cumulative statistics (e.g., the summed counts to
this point) will be strongly limited by restrictions to I*4.

5.  Users may wish to create arrays containing the indices to various pixels
in detectors that will soon exceed 2 GB pixels in size.   Note that this may
involve non-rectangular pixelizations like HTM or HEALPix so that the total
number of pixels matters (they have a 1D index, instead of having separate
X and Y coordinate indices).

6.  The new field of gravitational wave astronomy involves very precise
measurements of time and position.  While it is difficult now to cite a
specific case where arrays of 64-bit integers will be required, it would be
prudent to begin supporting them now in FITS in case they are needed in
this, or in other emerging fields of astronomical research.

7.  Availability of 64-bit integer data may make FITS more compatible
with storage of data from numerical simulations

8.  Arrays of encryption keys and such may be stored within FITS images and
could need the extra range provided by 64-bit integers.


B. Practical or sociological justifications
-------------------------------------------

The following cases illustrate various practical reasons for supporting
64-bit integer arrays in FITS files.

1. 64-bit integers are now almost universally available in standard
programming languages.  Applications programmers will naturally compute
arrays of values using of this data type and will expect to be able to store
the values in a FITS file in their native form.  Even in cases where the
scientific observations themselves do not require more than 32-bits of
precision, the extreme largest and smallest possible values supported by
that data type (often given by a predefined constant like LONG_MAX or
LONG_MIN) are often used to flag undefined pixels and other conditions.  It
would be costly and a potential source of error or confusion if these
extreme values had to be converted into the range of a less precise data
type for storage in a FITS file.

2. FITS is the working data analysis format of choice for many current
researchers.  These researchers often write their own personal programs to
do some specific type of analysis and use FITS files to store intermediate
results.  If a researcher has calculated a 64-bit integer array of numbers,
and wants to temporarily store the values on disk, the simplest way to do
this is just to write the values to a 1-dimensional FITS primary array.
This "quick and dirty" type of programming is still widely practiced and is
often the most cost effective way to solve small problems.

3.  In order to "future proof" applications, it is prudent to use arrays of
64-bit integers even if 32-bit integers seem adequate for for current needs.
  Software applications can end up being used for decades, and it is
difficult to foresee future needs.   It costs very little to build in 64-bit
integer support when an application is being designed and written, but it
may be difficult and costly to retrofit 64-bit support into the application
later on.

4.  Nearly all other scientific data formats support the 64-bit integer data
type, so FITS suffers by comparison.  We are being encouraged to become more
interdisciplinary, so it is important to be able to import data files from
other fields (space physics, planetary astronomy, earth observations) into
FITS to be able to process such images using astronomical image processing
algorithms.  This is made more difficult if FITS does not support the same
basic data types as the other sciences.

5.  From an abstract "data model" point of view, a primary array or image
extension is identical to a table with 1 row and 1 column that contains the
image as a vector.  Conceptually, one should be able to convert any table
vector into a FITS image array, and vice-versa.  Because of this similarity,
support for 64-bit images comes at little extra cost if that data type is
already supported in tables.

6.  In some of these cases, real*8 arrays could be used instead of
integer*8, but the standard data compression algorithms do not perform as
well on real data compared to integer data.  This can be a real advantage
when sending compressed data over slow networks.

7.  "Build it, and they will come":  One should not underestimate the
ingenuity of users and programmers in finding uses for new features.  Once
64-bit integers are available in FITS, users will surely discover uses that
we have not even considered.