[fitsbits] start of Public Comment Period on CONTINUE Long Kwd convention

[Mark Calabretta]

On Fri, 19 Jun 2015 12:07:43 +0200 (CEST)
Lucio Chiappetti <lucio at lambrate.inaf.it> wrote:

> This is to announce the official start of a 3-week formal Public Comment 
> Period on the incorporation of the well known CONTINUE Long String Keyword 
> convention in the FITS Standard.

Firstly, the question of supporting continued numeric types.

FITS currently allows integer types of up to 70 digits.  There are
various ways to store such a large integer:

  a) 256-bit integer data type,
  b) 128-bit integer array of length 2,
  c)  64-bit integer array of length 4,
  d)  32-bit integer array of length 8,
  e) a string.

Options (a), (b), and (c) can be eliminated on the grounds of
impossibility, implausibility, or impracticality.  Option (d) is
practical.  However, as the integer is represented in the header as a
string (by which I mean a sequence of ASCII-coded decimal digits), it
seems it would make sense to read and store the keyvalue also as a
string, i.e. option (e).  It only requires a simple copy operation.

More than 70 digits might be needed for encryption keys, hash keys, or
maybe some sort of checksum.  70 also falls short of the 97 needed
to count the number of fundamental particles in the universe,
(wikipedia.org/wiki/Elementary_particle).  However, having settled on
option (e), integers with more than 70 digits might as well be recorded
in a FITS header using a string-valued keyword.  That way they can take
advantage of the CONTINUE convention.  Thus, continuation is not
needed for integer keyvalues.

The same argument applies to floating point types, although there is
even less likelihood that more than 70 digits will be required for
those.  In practice, very high-precision floating point values are
split as the sum of integer and fractional parts recorded separately,
e.g. MJDREF = MJDREFI + MJDREFF.  This is despite the fact that 70
digits is more than enough to accomodate the precision required.  Thus,
continuation is not needed for floating point keyvalues.

For these reasons I believe that continuation is only needed for string
keyvalues.  However, while I believe that string continuation really is
needed, the proposed alteration of the standard glosses over the effect
that CONTINUE would have on FITS header parsers.

>From the cfitsio manual:

  The fits_read_key_longstr routine is unique among all the CFITSIO
  routines in that it internally allocates memory for the long string
  value; all the other CFITSIO routines that deal with arrays require
  that the calling program pre-allocate adequate space to hold the array
  of data.  Consequently, programs which use the fits_read_key_longstr
  routine must be careful to free the allocated memory for the string
  when it is no longer needed.

The current proposed change to the standard allows *any* string-valued
keyword to be continued, with six exceptions: XTENSION, TFORMn, EXTNAME,
TTYPEn, TDISPn, and TNULL.  Potentially, therefore, FITS readers must
always use fits_read_key_longstr(), or its equivalent, to read a
string-valued keyword (except for the six), and will be responsible for
freeing the memory allocated for each and every one.  The same goes for
keycomments also if they are allowed to grow in size.  There will also
be side-effects arising from the need to change the way that string
keyvalues are stored, e.g. as struct members.

Thus, while I'm sympathetic to the need for CONTINUE, I believe the
changes required to support it properly would place too great a burden
on FITS readers.  Possibly the problem could be solved with a new
long-string data type, but one thing is for sure, continuation should
not apply to any of the string-valued keywords currently recognised by
the standard.

Mark Calabretta