[fitsbits] Start of the CONTINUE keyword Public Comment Period

[William Pence]

Hi Mark,

Mark Calabretta wrote:
> On Tue 2007/07/17 16:17:06 -0400, William Pence wrote
> in a message to: FITSBITS <fitsbits at nrao.edu>
> 
>> Mark Calabretta wrote:
>>>   1) The continuation character, '&', is redundant syntax.  As described
>>>      in the prologue of fitshdr.h (from WCSLIB, as appended), it only
>>>      indicates continuation if the following card is CONTINUE otherwise
>>>      it must be interpreted literally.
>> The redundancy is intentional in this case, and helps to avoid any
>> possible confusion over whether the FITS writer really intended this
>> convention to be used or not.
> 
> Bill,
> 
> The CONTINUE convention differs significantly from the others offered
> for the registry because it defines basic functionality that people are
> likely to want to use.  It will almost inevitably become a de facto
> standard, if it hasn't already, or at least strongly influence the way
> that continuation syntax might be standardised.  Consequently, I think
> it is worth devoting some effort to settling on a syntax that we all
> feel comfortable with.

Since this convention has been in use for ~13 years, and is in current 
use for at least 2 active missions (RXTE and Chandra), I'm assuming 
there should be no objection to documenting the current syntax of this 
convention in the Registry, once any remaining issues about the 
completeness or clarity of the documentation are resolved.

Further discussion about whether some sort of continuation functionality 
should be incorporated into the FITS Standard is certainly welcome as 
far as I am concerned, but there is another somewhat related issue that 
should probably be considered at the same time: the 8-character limit on 
keyword names.  These 2 limits (the keyword name length, and the length 
of character string keyword values) are arguably the 2 limitations of 
FITS that most affect (and irritate) data providers.   Since both of 
these issues have to do with how to encode the required information into 
the 80-character header records, it seems to me these 2 issues should be 
considered together.  In the near future (probably in September) the 
existing ESO HIERARCH keyword convention (which effectively allows 
longer keyword names) will be submitted to the Registry.  That may 
provide a catalyst for wider discussion.

In the meantime, here are a few comments on your previous comments:
> 
> My only concern with the convention as currently described is with the
> use of the '&' character, which, to reiterate, is redundant syntax.
> However, I don't advocate eliminating it.  Instead I suggest making it
> optional in precisely the way described in the prologue of fitshdr.h
> (previously appended).  In practice it can still serve the useful
> function of "guarding" trailing blanks that are to be preserved in a
> string value.  The prologue of fitshdr.h describes how this form of
> CONTINUE-based continuation works in a parser that has been implemented.

The '&' character serves 3 important functions, so I'm uneasy with 
making it optional:

1.  The redundancy of requiring both the '&' at the end of the string 
followed by a CONTINUE keyword serves to prevent misinterpretation of 
the header values.  To illustrate, consider the following keywords taken 
from an actual RXTE file:

1CTYP12 = 'CHANNEL '
1CPIX12 = '(S[msLimit1]~S[msLimit2]),((S[msLimit2]+1)~S[msLimit&'
CONTINUE  '3]+1)~S[msLimit4]),((S[msLimit4]+1)~S[msLimit5])'

If the 1CPIX12 keyword is now deleted without also deleting the 
following CONTINUE keyword (as could easily happen if the software does 
not support this convention) then you are left with this:

1CTYP12 = 'CHANNEL '
CONTINUE  '3]+1)~S[msLimit4]),((S[msLimit4]+1)~S[msLimit5])'

This will cause the 1CTYP12 keyword value to be misinterpreted if one 
does not require the '&' as part of this convention.  Requiring that the 
first string end with an '&' makes is much less likely that an orphaned 
or misplaced CONTINUE keyword will cause any damage; it will simply be 
treated in the same way as a harmless COMMENT keyword.

2. The '&' also serves as an aid to FITS reading programs by providing 
an explicit indication that the keyword may be continued.  Without this 
indicator, the reading program would always have to check the next 
keyword to see if it is a continuation, which could add more overhead. 
  This may not be a particular issue for software like your wcslib 
parser, which passes through the whole header once, to populate an 
internal structure, but it is a significant issue for other software, 
like my CFITSIO library, which reads the keywords from the header on demand.

3.  The third function of the '&', as you mentioned, is to allow 
trailing blanks to be included as a significant part of the string.

Taken together, these 3 reasons seem compelling enough to me to require 
that the '&' be used as part of this convention.

> Tying continuation to a particular data type, apart from being
> unnecessary, must be unique amongst computer-based syntaxes.  The
> argument that only string values are likely to be continued ignores
> possible future syntaxes.  For example, record-valued keywords currently
> proposed by/for WCS Paper IV might be better implemented by extending
> the keyword syntax.  They could easily be long enough to require
> continuation, and the continued portion could be a floating point value
> or something else.

As currently defined, the record-valued keywords all have string values, 
so there shouldn't be any problem with using this continuation 
convention with them, if desired.   I guess your point is that some 
other syntax might be invented in the future in which the keyword values 
are not strings.  That may be, but without concrete examples, it is hard 
to see why the current convention is not adequate.

Bill Pence
-- 
____________________________________________________________________
Dr. William Pence                       pence at milkyway.gsfc.nasa.gov
NASA/GSFC Code 662       HEASARC        +1-301-286-4599 (voice)
Greenbelt MD 20771                      +1-301-286-1684 (fax)