[fitsbits] start of Public Comment Period on CONTINUE Long Kwd convention

[Tom McGlynn (NASA/GSFC Code 660.1)]

A couple of nits and a more serious disagreement later...
     Tom

Mark Calabretta wrote:
> On Fri, 19 Jun 2015 12:07:43 +0200 (CEST)
> Lucio Chiappetti <lucio at lambrate.inaf.it> wrote:
>
>> This is to announce the official start of a 3-week formal Public Comment
>> Period on the incorporation of the well known CONTINUE Long String Keyword
>> convention in the FITS Standard.
> Firstly, the question of supporting continued numeric types.
>
> FITS currently allows integer types of up to 70 digits.  There are
> various ways to store such a large integer:
>
>    a) 256-bit integer data type,
>    b) 128-bit integer array of length 2,
>    c)  64-bit integer array of length 4,
>    d)  32-bit integer array of length 8,
>    e) a string.
>
> Options (a), (b), and (c) can be eliminated on the grounds of
> impossibility, implausibility, or impracticality.  Option (d) is
> practical.
<nit> Why are arrays of longs less plausible the arrays of ints?  
Support for
64 bit integers is pretty common these days </nit>
> However, as the integer is represented in the header as a
> string (by which I mean a sequence of ASCII-coded decimal digits), it
> seems it would make sense to read and store the keyvalue also as a
> string, i.e. option (e).  It only requires a simple copy operation.
>
> More than 70 digits might be needed for encryption keys, hash keys, or
> maybe some sort of checksum.  70 also falls short of the 97 needed
> to count the number of fundamental particles in the universe,
> (wikipedia.org/wiki/Elementary_particle).  However, having settled on
> option (e), integers with more than 70 digits might as well be recorded
> in a FITS header using a string-valued keyword.  That way they can take
> advantage of the CONTINUE convention.  Thus, continuation is not
> needed for integer keyvalues.
>
> The same argument applies to floating point types, although there is
> even less likelihood that more than 70 digits will be required for
> those.  In practice, very high-precision floating point values are
> split as the sum of integer and fractional parts recorded separately,
> e.g. MJDREF = MJDREFI + MJDREFF.  This is despite the fact that 70
> digits is more than enough to accomodate the precision required.  Thus,
> continuation is not needed for floating point keyvalues.
<nit> These is the possibility of complex values in the header.  I don't 
know if these have ever been much
used, but I think they bring the possibility of a need for more space  
for floats from infinitesimal to extremely remote </nit.
>
> For these reasons I believe that continuation is only needed for string
> keyvalues.  However, while I believe that string continuation really is
> needed, the proposed alteration of the standard glosses over the effect
> that CONTINUE would have on FITS header parsers.
>
>  From the cfitsio manual:
>
>    The fits_read_key_longstr routine is unique among all the CFITSIO
>    routines in that it internally allocates memory for the long string
>    value; all the other CFITSIO routines that deal with arrays require
>    that the calling program pre-allocate adequate space to hold the array
>    of data.  Consequently, programs which use the fits_read_key_longstr
>    routine must be careful to free the allocated memory for the string
>    when it is no longer needed.
>
> The current proposed change to the standard allows *any* string-valued
> keyword to be continued, with six exceptions: XTENSION, TFORMn, EXTNAME,
> TTYPEn, TDISPn, and TNULL.  Potentially, therefore, FITS readers must
> always use fits_read_key_longstr(), or its equivalent, to read a
> string-valued keyword (except for the six), and will be responsible for
> freeing the memory allocated for each and every one.  The same goes for
> keycomments also if they are allowed to grow in size.  There will also
> be side-effects arising from the need to change the way that string
> keyvalues are stored, e.g. as struct members.

I'm a little confused by this.  When long strings are supported I would 
assume a program that supports them (including using CFITSIO) reads the 
string value using it's standard mechanism and if it sees a trailing '&' 
some special software kicks in.  For the vast majority of string values, 
no special handling need be done.   Right now CFITSIO supports long 
strings for virtually any string keyword. Nothing is changing here.   
Clearly libraries which don't support long strings will need some 
update, but for the libraries which do the only different may be that 
the support is always enabled rather than being triggered by the 
LONGSTRN keyword or some user flag.

Not do I think that fact that CFITSIO's support for long strings is 
hampered by the primitive way C handles strings should matter too much 
in determining if this is a feature that's needed to meet the needs of 
those who read and write data using FITS.

>
> Thus, while I'm sympathetic to the need for CONTINUE, I believe the
> changes required to support it properly would place too great a burden
> on FITS readers.  Possibly the problem could be solved with a new
> long-string data type, but one thing is for sure, continuation should
> not apply to any of the string-valued keywords currently recognised by
> the standard.
>
> Mark Calabretta
>
>