[fitsbits] reopening of Public Comment Period on the CONTINUE convention

[Dick Shaw]

Hi Demitri,

I can see you feel strongly about the positioning of the ampersand character in the context of 
CONTINUE.

On Thu, 10 Mar 2016 09:00:00 -0500 Demitri Muna <demitri.muna at gmail.com> wrote:
> On Mar 9, 2016, at 4:13 AM, Peter Weilbacher <pweilbacher at aip.de> wrote:
>>> There is a standard here that is used in virtually every (every)
>>> computer language - continuation or concatenation characters are
>>> located outside of the string being quoted.
>> 
>> But we are not discussing a computer language here, but a file format.
> 
> I mentioned it as the convention will already be familiar to people. But whether it?s a file 
>format or a computer language is irrelevant; the question what is easier to parse and/or read, 
>first by a human and second by a computer. When one sees a line end with an ampersand and a 
>single quote, the current proposal says ?this might be a continuation character, or it might 
>not.? 

Good point.

>When virtually every computer language agrees on a certain syntax, it?s worth paying 
>attention to. Can anyone point to a counterexample where the continuation character is contained 
>within a quoted string anywhere?
>
>> I don't think that the FITS Standard allows to have anything between the
>> value and the '/' that starts the comment. At least that's how I
>> understand Section 4.1.1 and Appendix A. So having the continue
>> character outside the string literals would make a very fundamental
>> change to the FITS format. And quite hard on every FITS header parser.
> 
> The current proposal specifically modifies Section 4.1.1, so that is what is under discussion. 
>Given this:
> 
> STRKEY = 'This keyword value is continued&'
> CONTINUE ' over multiple keyword records.'
> 
> or this:
> 
> STRKEY = 'This keyword value is continued' &
> CONTINUE ' over multiple keyword records.'
> 
> an existing parser would treat the ampersand as part of the value and see no comment. I don?t 
>see how this is a fundamental change or how it will break existing parsers. 

Again, a good point. One of the things that has improved over the years is the availability of 
standard libraries for various languages to do "standard" text processing which means an existing 
solution could be adopted when updating FITS software, rather than something ad-hoc. These 
libraries were less common in the 1990's.

>Can you provide an 
>example where this would be a problem? You would never have this, for example:
> 
> STRKEY  = 'string' & / comment
> 
> This seems to be the edge case, where again the ampersand is not between the '/' and the 
>comment, which is still not a problem:

Unfortunately, this is certainly not an edge case. A quick browse through many FITS headers shows 
that a keyword = string value, followed by a comment, happens all. the. time. Allowing the syntax 
you suggest, in this context, would amount to a pretty big change. What you suggest is sensible 
for continuing comments, something along the lines you suggest:

> STRKEY  = 'some string that ends here but is followed by a comment' &
> CONTINUE '' / comment here 

This seems a little ad-hoc, but certainly not crazy.

>>> If the motivation is to turn into a standard something that someone
>>> else decided ad hoc to do, then I?d ask if this really is the best way
>>> to define standards? The fact that one institute did something doesn?t
>>> necessarily make it the best idea, and in this case I really don?t
>>> think it is.
[...]
>> I agree that this is not the most beautiful way to create FITS headers.
>> But since the motivation here is rather to have an incremental
>> improvement that works (we know, because it has seen usage for 20+
>> years) instead of reinventing the FITS format, I agree with the
>> proposal.
> 
> I think this is a very important point. There has been a lot of discussion about file formats 
>and pretty wide dissatisfaction with FITS. There's no question it's showing its age. I think it 
>would be one thing if there was a commitment to properly addressing the real concerns with a 
>proper FITS v2.0 versus incremental patching. The past should guide the future, not constrain it.

I have some sympathy for that point. But we do have to live with the prior history of FITS, warts 
and all. Therefore it's important to avoid introducing changes that either invalidate current FITS 
files or are highly disruptive of existing FITS software, unless there is a highly compelling 
reason or unless the proposed change can be ignored by an extant FITS reader without undue harm. 
This statement would hold more-or-less true whether we are discussing FITS, or any alternative 
standard once it has been around for awhile.

I guess the question is: can your idea be expressed in the proposed change to the Standard in a 
non-disruptive way?

-Dick