[fitsbits] reopening of Public Comment Period on the CONTINUE convention

[Tom McGlynn (NASA/GSFC Code 660.1)]

Tried to clarify my issues below in-line.
     Tom

William Pence wrote:
> On 3/10/2016 11:20 AM, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
>> Two further nits...
>>
>> I am unclear of the legality of the use of CONTINUE keywords in a
>>      CONTINUE = '&'  / A continuation character.
>> context.
>> The new convention is described before the general constraint in 4.4.2
>> reserving the use of the keywords defined later in the
>> standard, but they are mentioned afterwards, so I think that status of
>> other uses of CONTINUE is a bit hazy.  [The use of
>> CONTINUE as a comment style keyword is explicitly allowed though.]
>
> Huh?  It is not clear to me what your question is.  If you are in
> particular referring to the statement that an orphaned CONTINUE
> keyword should be interpreted as containing commentary text, that is
> simply a restatement of the definition in section 4.1.2.3 that if any
> keyword does not have a value indicator (i.e., the equals sign and
> space character in bytes 9 and 10) then bytes 9 through 80 on that
> keyword should normally be interpreted as commentary text.  So the
> interpretation of orphaned CONTINUE keywords is not unique, and a
> similar interpretation applies to any other user-defined keyword that
> does not have a value indicator.

That's not the usage I was talking about.  As you note, orphaned 
commentary style header cards are explicitly referenced as legal.
The issue is whether it is still legal to use CONTINUE as a regular 
keyword, i.e., one with a value.  Previously it would clearly have been 
perfectly legal to use

     CONTINUE = 'Some value'

in the header since CONTINUE is not mentioned in the standard and users 
are free to use it as a keyword with a value.  Paragraph 4.4.2 says that 
keywords discussed later in the standard are optional but must be used 
only in the way described here.  If that statement applies to CONTINUE 
values, then in the future  CONTINUE can now only be used in the 
commentary style.  However CONTINUE is defined earlier in the standard 
than 4.4.2 and I can't find an equivalent statement that definitely 
applies to it.  Still they are mentioned in the new 4.4.2.4 so maybe 
the statement applies.  By contrast, since COMMENT and HISTORY keywords 
are defined in 4.4.2.4 it is clear that 4.4.2 states that  they may not 
be used with values.
>
>> If someone has used CONTINUE as a keyword, have their files become
>> illegal?
>> I don't actually care what the answer to this is, but I think it should
>> be clarified either way.
>
> No. Section 3.7 unequivocally states that any valid FITS file shall
> remain a valid FITS file at all future times.  It is because of this
> 'Once FITS, always FITS' rule that new requirements are rarely added
> to the standard and then only when the impact on existing FITS files
> is considered to be minimal.  In particular, When new reserved
> keywords are added to the Standard  (which has happened several times
> over the years), I think it is generally understood that any existing
> FITS files that may have used that keyword in a different way are
> exempt from the new requirements on how that keyword should be used.
>
I don't think it makes operational sense to say that a file is legal if 
it was written earlier than some date.  If you copy the file is the
original valid but the copy not?  If we really take this seriously, then 
we can say that use of CONTINUE in a key/value pair is now deprecated 
but still legal.  That's fine with me but I think it should be 
explicitly stated.  Actually looking at 3.7, the statement is that
any FITS <structure> shall remain valid where by context I think 
structure refers to things at the level of primary arrays and such, more 
at the HDU level than headers.  I'd assume it's there to make it clear 
than random groups are never going away!  I see no statement that a 
given FITS file cannot become invalid due to changes in the standard.

There's a bit of meta inference here: when we say that no structure can 
become invalid do we mean that no instance of the structure can become 
invalid or that the structure itself will always remain part of the 
standard.  I think the latter makes more sense.
>>
>> The other nit is the new final sentence in 4.2.1.1 is puzzling.
>> I.e., what does
>>    In general, no length limit less than 68 is implied for
>>    character-valued keywords.
>> mean?  We'd just made it clear that single record character string
>> values
>> can be up to 68 characters.  I don't know that the this following
>> sentence  does any harm, but I can't see what
>> work it is doing and it may confuse others beyond me.  In fact, there is
>> no length limit whatsover on string values now.
>
> The highlighting of that entire paragraph as being new is an editorial
> mistake.  The only thing that is actually new in that paragraph is the
> phrase "that can be represented on a single keyword record".
> Otherwise, that same wording can be found in previous versions of the
> FITS Standard document going back to the 2.0 version in 1999.  The
> sentence stating that "no length limit less than 68 is implied" was
> added back then to refute a statement in the original Wells et al.
> FITS paper which implied that character string keyword values could
> normally be assumed to be 8 characters in length, or at least that the
> value could be uniquely defined by only reading the first 8
> characters.  Recall that that original FITS paper was written in 1980
> when the amount of available RAM was usually measured in units of
> kilobytes.
>
Given that there is now no limit on the length of strings whatsoever, 
perhaps the sentence should be omitted.  Actually you could get rid of 
the entire paragraph, and start the new section with the sentence that 
notes that a single line char string can be only so long and then goes 
on to note the new capability.  I think this would flow better and make 
it seem like the long strings were an organic part of FITS rather than 
an ad hoc add on -- even if the latter is true!