[fitsbits] Start of the CONTINUE keyword Public Comment Period

[William Pence]

A disk crash here over the weekend wiped out all the files related to
the CONTINUE long string keyword convention on the FITS Support Office
web site, so I've had to recreate them.  In the process, I've
revised the documentation about this convention to try to address all
the comments that have been received so far.  Please see the new PDF (or
postscript) documentation file available from the FITS Registry at
http://fits.gsfc.nasa.gov/fits_registry.html.


Here are a few more comments in response to recent postings:

Rob Seaman wrote:
> 1) Couldn't the reordering question be dealt with via appending &NNN  
> instead of &?  

Many different way of explicitly encoding the order of the continuation
keywords were suggested back in 1993-4, but the HEASARC decided that
this added complexity was not needed for it's applications. I don't
recall a single instance in the past 13 years where users encountered
problems with the keywords getting out of order.

> 
> 2) What is the reason for not strictly requiring that such a  
> continuation string end with ampersand-quote?  What advantage is  
> conveyed by permitting semantically null trailing whitespace that  
> merely confuses the handling?  

This was probably influenced by the FITS Standard, which says trailing
spaces in keyword string values are not significant.

> One could, for instance, recast a 
> CHECKSUM keyword:
> 
> CHECKSUM= 'nADTnADQnADQnADQ'    /  ASCII 1's complement checksum
> 
> as:
> 
> CHECKSUM= 'nADTnADQ&       '    /  ASCII 1's complement checksum
> CONTINUE  'nADQnADQ'
> 
> This would apply to your favorite string valued keyword as well.  The 
> example above would be a legal continuation but an illegal checksum 
> value.  (More than illegal for this example since the byte ordering 
> needs to be preserved for the zeroing feature of the checksum to work.)

I may be wrong, but I don't really see any technical problem with
writing the CHECKSUM keyword as a continuation over 2 (or more)
keywords.  The algorithm for initializing the header and calculating the
CHECKSUM keyword value(s) would need to be modified slightly, but it
would still follow the same basic procedure.  (Not to suggest that there
is any reason to do this of course).

Mark Calabretta wrote:
>   1) The continuation character, '&', is redundant syntax.  As described
>      in the prologue of fitshdr.h (from WCSLIB, as appended), it only
>      indicates continuation if the following card is CONTINUE otherwise
>      it must be interpreted literally.

The redundancy is intentional in this case, and helps to avoid any
possible confusion over whether the FITS writer really intended this
convention to be used or not.

>   2) Making the continuation character part of the keyvalue, rather than
>      a separate token, means that:
> 
>      a) Continuation is only defined for string keyvalues.

Yes, because we assumed a single keyword is adequate for recording
integer, logical, or floating point values, so continuations would not
be needed in these cases.

>      b) When the parser tokenizes a header card, instead of being a
>         separate token in itself the (redundant) continuation character
>         must be removed from one of the other tokens.  Consequently, the
>         parser is obliged to modify the keyvalue.

It seemed safer and more convenient to embedded the token in the string,
rather than perhaps as part of the comment field of the keyword.  The
token is less likely to get separated or inadvertently stripped off this
way.  Many FITS readers ignore the keyword comments, so putting vital
information there is perhaps not a good idea.

Doug Mink wrote:
> As no one has mentioned it yet, I would like to point out that there is a
> huge amount of optical data in FITS files which use the IRAF multi-keyword
> continuation method.  A 1-4 letter keyword is followed by an underscore and
> three-digit sequence number. 
>   . . .
> WAT0_001= 'system=image'
> WAT1_001= 'wtype=tnx axtype=ra lngcor = "3. 4. 4. 2. -0.05837385199330483 0.070'
> WAT2_001= 'wtype=tnx axtype=dec latcor = "3. 4. 4. 2. -0.058373851993304
> TRIM    = 'Jul 25 21:28 Trim data section is [1:2047,1:4093]'
> OVERSCAN= 'Jul 25 21:28 Overscan section is [2054:2100,100:4100] with mean=1064'
> WAT1_002= '44519483406535 -0.09397437792791556 0.1451290667953512 -4.3273459822'
> WAT1_003= '97552E-5 4.525801441223172E-4 0.009912907609377613 0.185709640211661'
> WAT1_004= '6 1.874028266281251E-5 0.01797451858969125 0.04459426127294047 0.010'
> WAT1_005= '87988458562529 -0.1525431073290788 -0.00297118661606969 "'
> WAT2_002= '044519483406535 -0.09397437792791556 0.1451290667953512 -4.808546205'
> WAT2_003= '327283E-5 4.148865918405720E-4 0.01180811673900729 0.087140524550311'
> WAT2_004= '24 -9.417303440113919E-4 0.03105863709218507 0.3049773562603359 0.01'
> WAT2_005= '469707839868787 0.0440181026974359 0.1257086385879256 "'
>   . . .

The main limitation with this of course is that the root keyword names
can only be up to 4 characters long.  The HEASARC needed a convention
that could be used with keywords that describe table columns and hence
would have a 1 to 3 digit column number following the root name.

William Thompson wrote:
> Second, I'd like to make some comments about keyword ordering.  In solar physics 
> it's common to convert FITS headers into structures upon reading. 
...
> In many cases, the arrangement of the structure is predetermined, and the FITS 
> header is mined to extract the keywords corresponding to the desired parameters. 
>   In such cases, the original order of the keywords within the header is lost.
...
> I would envision that continuation lines would best be handled by the FITS 
> reader, so that the resulting structure tag value would be the complete string. 
>   In that case, the question of the original ordering of the keyword would be 
> moot.  However, I doubt that any software would actually be changed to implement 
> this until there was a real need for it.

I agree.  As long as the structure is capable of holding the full length
of the long string keyword value, then the original order of the FITS
keywords should not need to be preserved.

Rob Seaman wrote:
> Presumably we are in a period of capturing and documenting as many local 
> conventions as possible.  The bigger question is what comes next.  Some 
> (I'll timidly suggest the checksum convention) seem mature enough to 
> become part of the standard.  Others, as with this current discussion, 
> appear to have some real issues with broad adoption.  Still others, like 
> foreign encapsulation perhaps, might simply appear as some sort of 
> appendix since a particular HDU either expresses such an object or does not.

At the moment, most of my effort is going into just documenting the
existing conventions.  At the current rate, we will probably have
captured most of the common conventions (unless there are many more
lurking around) within the next year.  After that might be a good time
to consider whether we want to promote, or further develop, some of the
conventions and make them standards.

> I guess I'm also wondering whether the point of the registry is to 
> capture legacy conventions or to encourage new conventions. 

The rules for what is appropriate for the current Registry state:
"Generally speaking, proposed new FITS conventions that are not yet in
actual use should not be submitted to the Registry until the proposal is
fairly mature and has demonstratable support from at least a significant
subset of the FITS user community."



Finally, if anyone has read this far and is still interested in further
background information about why this particular continuation convention
was chosen, here is a copy of a FITSBITS posting made back in 1994:

------------------------------------------------------------------
The OGIP FITS Working Group met recently to decide how to handle long
string keyword values in FITS files.  We urgently need such a
convention for our soon to be launched mission, so given that there is
currently no consensus within the FITS community on a standard way to
deal with this problem, we have adopted a local FITS convention for use
within our own processing environment.  Since this is not an agreed
upon standard, we will not use this convention for any keywords that we
would expect any external software systems would need to read or
interpret.

The following example illustrates the convention that we have adopted:

BIGSTRNG= 'This is a long string value that is continued &'
CONTINUE  'over three keywords in the&   '
CONTINUE  ' FITS header.'

The '&' character is used as the last non-blank character of the string
to indicate that the string is (probably) continued on the following
keyword (if the next keyword name = 'CONTINUE' as described below).  We
adopted the '&' character instead of the '\' that had been previously
proposed because of Barry Schlesinger's observation that the '\' is one
of the 10 'National Use' positions which may not be interpreted as the
same symbol by all nationalities.  Also, some Fortran compilers treat
the '\' character in a non-standard way making it difficult to write
machine-independent code that uses this character.  The '&'  seems to
be a reasonable alternative, especially since it is used as a
continuation character in Fortran 90.

Each continuation line has the keyword name = 'CONTINUE'.  We adopted
this keyword name, rather than the previous suggestion of an all blank
keyword name, in order to provide an unambiguous way of identifying any
continuation keywords even if they get separated from their parent
keyword, and to try to avoid conflicts with other FITS software that
may treat keywords with an all-blank name in a unique or special way.
Since there is no equal sign in column 9, this keyword belongs to the
same class of keywords as the COMMENT and HISTORY keywords that do not
have a defined value.   Under our convention the continuation of the
character string value is enclosed in quotes starting in column 11 (or
higher) of the 80-character record.  Any other characters (e.g. a
comment field) may optionally follow the closing quote character on any
or all of the keywords.

We also considered, but decided against, implementing a convention that
would explicitly link together all the keywords so that the correct
continuation sequence could be reconstructed even if the order of the
keywords in the header were rearranged. (Several such schemes were
suggested here on FITSBITS in the past month).  Since there is even
less consensus on the best way to implement such a scheme, or if it is
even needed, we decided to simply rely on the physical order of the
keywords in the header, which meets our own needs.  (We did confirm
with Barry Schlesinger that this reliance on the order of the keywords
in the header does conform to the FITS Standard and hence is legal to
use in a local FITS convention).  If the FITS community later decides
that a full forward and backward keyword linking capability is required
to make this convention acceptable as a Standard FITS convention, then
this can in principle be added with only minor modifications to our
convention.
--------------------------------------------------------------------

Bill Pence
-- 
____________________________________________________________________
Dr. William Pence                       pence at milkyway.gsfc.nasa.gov
NASA/GSFC Code 662       HEASARC        +1-301-286-4599 (voice)
Greenbelt MD 20771                      +1-301-286-1684 (fax)