FITS long integer support (was Re: [fitsbits] ADASS FITS BoF on Sunday)

[Tom McGlynn]

Preben Grosbol wrote:

> On Wednesday 20 October 2004 16:29, Eric Greisen wrote:
> 
>>Since many operating systems and compilers do not support
>>64-bit integers (other than in sneaky hidden ways to read large
>>files), we should move extremely slowly to explicitly allow them in
>>FITS.
> 
> I can fully support this view. The only good argument for 64-bit
> integers is pointers as uncertainties in physical quantities hardly
> can justify such accuracy. So the issue if pointers to the HEAP
> or reference columns to rows in tables with more than 2G rows
> are important currently.  I would prefer to wait until 64-bit machines
> are the default in our community.
> 
> Preben Grosbol


While earlier discussion was not advocacy, let me discuss
where my views lie...

Eric and Preben have suggested that both the need for and the support
for 8-byte integers is sufficiently rare that it would be inappropriate
to consider revising the standard to support them.

I don't agree with either point.  Support for eight byte integers
is widespread within machines used today.  Most current C, Fortran and all Java
compilers support eight byte integers.  IDL has supported 8-byte integers
for several years.  There are doubtless many machines/compilers extent which
do not support 8-byte integers but there are many machines which still do
not support files longer than 2 GB.  Nonetheless such files are usefully
produced as FITS.  [By the by, it might be argued that Fortran has no
'standard' way to describe integers of 8 bytes.  Of course it also has no
standard way to describe integers of 2 bytes (or for that matter a completely
standard way to describe integers of 4 bytes).  However most Fortrans
that I have seen have a 'kind' corresponding to 8 byte integers.]

With regard to usage...   I personally don't seen any immediate need in the
community for images with eight-byte integer depth, however usage of
eight byte integers in tables seems very desirable.  E.g., consider an
X-ray mission detecting photons with a microsecond resolution clock.
A 4-byte integer will overflow in less than an hour.  When housekeeping data is stored
in 8-byte longs that should be the natural way to store it.  If we are counting
photons in an image, the total number of photons can easily exceed the 4-byte limit.
There are now lots of places out there where our measuring devices count beyond the billions.

Current catalogs of images are already at or passing the 2 GB limit for positive 4-byte integers.
If we wish to create FITS representations of new catalogs (or subsets of them)
we are going to find it difficult to fit the indices in 4-byte integers, while 8-bytes
will suffice for the foreseeable future.

But the most compelling need for 8 byte integers with FITS may be to support
variable length arrays.  Multi-gigabyte files are now commonplace in astronomy.
Use of variable length arrays could allow us to index information in these large
files but this cannot be done since the offsets will very quickly surpass the 4-byte
limits.  Within a few years 100 GB files are going to be normal and if we
wish the variable length records extension to be viable it needs to be able to
accommodate data on such scales.

Finally, a bit of philosophy...

As Eric noted FITS originated as an interchange format, but that is not all it
is today, nor should that be the only usage that should drive its evolution.
FITS today is used as a data format in many software packages.  FITS is also
the standard archival format for most astronomy data.  When we look at FITS
and decide whether or not to extend it, recognize that when we limit FITS
we may make other formats, e.g., HDF, more appealing to those who need the
capabilities being proscribed.

But what about those who can't read the new formats?  I don't think they
will be as numerous as some seem to be suggesting.  Many of the major libraries
already support 8-byte integers on an experimental basis.  So those who
use CFITSIO need change nothing in their code.  They already can do most
of this!  Nor will existing files, or existing data streams suddenly adopt 8-byte
integers en masse.  No existing standard FITS file will be made invalid.  What will
happen is that people will gradually recognize the they no longer need to use the
subterfuges and workarounds to stay within the legal FITS boundaries and eight-byte
integers will emerge where they are most needed.

	Regards,
	Tom McGlynn