[fitsbits] further reopening of Public Comment Period on the CONTINUE convention

Fri Apr 22 10:45:40 EDT 2016

After some time, today I have time to look at the backlog of messages 
about the CONTINUE convention. Since Monday is holiday here, I would 
resume next week with the idea to update the text for the standard 
document in a way that is agreeable for everybody.

I prize particularly the opinions of Mark Calabretta and Thierry Forveille 
since they are voting members, and expressed a conditional possibility of 
voting NO, which, as you know, forces a delay of 3 months even if 
approved.

***********************************************************************
If any other IAUFWG member is conditionally considering a NO vote 
please speak up now.
***********************************************************************

As far as I can see there are the following issues (I skip the occasional 
references to a different way of handling metadata in separate extensions, 
which is a topic we may address separately later):

0)  are CONTINUEd strings useful ?
1)  are traditional strings and continued strings a different data type ? 
1b) if yes how should they be called officially ?
2)  do traditional strings and continued strings deserve a different
     handling in software ?
3)  is CONTINUE the best way to deal with continuation ?
4)  do we need to forbid usage of CONTINUE on some keywords ?
5)  do we need to put length limits on continued keywords ?
6)  *and finally* which changes to the text are needed ?

-------------------------------------------------------------------------------
0) are CONTINUEd strings useful ?

This was somehow questioned by Thierry and answered e.g. by Bill (both 19 
Apr).

I add some argument in favour of long strings for particular applications. 
One is the case when one may want to record in a keyword full pathnames of 
a file, which may be longer than 68 char. Another one is when one wants to 
record the full invocation of a command with a long argument list.

Another one is when one wants to record an array of coefficients, or even 
a "collection" of arguments and coefficients (I mean e.g. a mixture of 
integer, floats and perhaps strings). There are some of the registered 
conventions about distortions which do such things, and do it in a 
different, inelegant way. With CONTINUE they could have done it in an 
uniform way.

------------------------------------------------------------------------------
1) are traditional strings and continued strings a different data type ?

This is advocated mainly by Mark (e.g. Apr 19, 21). I note again that 
typing in FITS is somewhat loose and/or euristic and does not match 
"byte-length" types of programming languages (RICEGRN= 
18446744073709551615 is a valid FITS integer kwd, and SOMEKWD = 1.5E400 or 
OTHERKWD = 1.23456789012345678901234567890123456789 are valid FITS float 
kwds although they do not fit any current data type).

The point is that FITS readers, concerning kwd types, are fundamentally of 
two sorts, which could be called "searching" or "scanning".

- A "searching" program LOOKS for a kwd of a known name, and expects it to
   be of a known type a priori from the documentation, and prepares a
   variable of the right type to receive its value.

- A "scanning" program reads all kwds in the header in sequence, tries to
   guess their type (see e.g. my IDL example in my message of Apr 12) and
   to create a variable of the needed type. Or can even defer this at a
   later stage. Nobody forbids a program to read the entire header in
   memory as an array of CHAR(80)  "card images", or two arrays of CHAR(8)
   KWDNAMES and CHAR(68) KWDVALUES and later scan them and determine the
   needed length.

Note that if a kwd is declared long-string this means it CAN BE longer 
than 68 char, but not that it MUST BE longer. It may well be that in 
particular conditions the value fits in a single keyword record !

I suggest to replace this question with question (2) below. But I am not 
against defining or considering or labelling two "logical" types.

------------------------------------------------------------------------------
1b) if yes how should they be called officially ?

Original proposal by Mark: string and long-string.

Variant by Mark (21 Apr 22:45): non-continuable and continuable.

Current draft text: Single record string (4.2.1.1) Continued string 
(4.2.1.2)

------------------------------------------------------------------------------
2) do traditional strings and continued strings deserve a different handling
    in software ?

Discussed in various messages by Bill (about CFITSIO) and Mark (about 
WCSlib) and not only. Addressed also by van Nieuwenhoven (Apr 19) and by 
my mail of Apr 12.

I guess everybody agrees there is a LOGICAL difference, my sentence in the 
draft at the end of 4.1.1 shows what is exactly happening ... the CONTINUE 
convention breaks the 1:1 correspondence between one logical kwd (with a 
name and a value) and one 80-character header keyword, continued kwds are 
a 1-to-many correspondence !

Whether that results in a difference in programs may well depend on the 
particular programming language and implementation (see my considerations 
about 'in the eye of the beholder' (me, Apr 12) and "scanning" 
considerations in (1) above.

In principle one could always consider all kwds continuable, look if they 
are followed by a CONTINUE and stop when there are no more (or none). In 
practice that would be easy for newly-written adhoc programs, but perhaps 
inconventient for existing libraries.

------------------------------------------------------------------------------
3) is CONTINUE the best way to deal with continuation ?

This was somehow questioned by Walter Jaffe and answered by Bill (both 21 
Apr). An alternate way are indexed keyword name (SVALUE1, SVALUE2). 
However, in lack of keyword arrays, indexed keywords are used in FITS also 
to emulate array elements, or sequences (compare the evergreen CRPIXn 
CRVALn or TFORMn TTYPEn etc.).

And actually the latter are *different* keywords, while a sequence of 
SVALUE1 ... SVALUEn to be concatenated in a single "logical" kwd are not 
(see also #2 above and the sentence at the end of 4.1.1 in the draft text)

------------------------------------------------------------------------------
4) do we need to forbid usage of CONTINUE on some keywords ?

The original convention just did not recommend it use with reserved or 
mandatory or common kwds. We first strenghtened than released this 
condition, and currently we face two apparently different proposals:

"CONTINUE may be used only for keywords declared 'long-string' and must 
not be used for those declared 'string'" (Mark, since Mar 24)

"CONTINUE must not be used with of any of the mandatory or reserved 
keywords defined in this standard unless stated otherwise" (Bill, Apr 19 
and 21)

Strictly speaking, such limitation is not required for ORIGIN, AUTHOR, 
REFERENC, OBJECT, OBSERVER, TELESCOP, and INSTRUME (Bill Apr 20 23:12), 
plus WCSNAME and CNAMEia (Mark Apr 21 22:50), to which I would add 
OBSORBIT from the latest standard.

However I am ready so far to accept that all mandatory and reserved kwds 
are not continuable. If we'd want to relax such condition we could LATER 
vote to update the standard and declare e.g. ORIGIN to be a long-string.

Otherwise said, the two proposals are equivalent given that the current 
standards declares all string kwds as string. Mark's proposal is perhaps 
more general in that it requires also project conventions to explicitly 
declare long-strings.

-----------------------------------------------------------------------------
5) do we need to put length limits on continued keywords ?

This was raised by Tim Pearson (21 Apr). The FITS standard does not like 
to put limits on length (integer and float kwds are NOT bound to 16-, 32- 
or 64-bit quantities !), and rarely put limits on sequences of items (like 
e.g. NAXIS or TFIELDS to 999 but that's more or less accidental given the 
current keyword NAME length limit of 8, because NAXESnnn or TFORMnnn allow 
just 3 digits, like the 26 concurrent WCS's allowed by WCSNAMEa).

So I won't place a character limit on the maximum length of a continued 
string. I could accept a limit of e.g. 999 CONTINUE keyword record for 
each keyword, but I do not think it makes much sense.

On the other hand I gather that Tim would be happy if the maximum allowed 
length is EXPLICITLY declared as "unlimited".

------------------------------------------------------------------------------
6) *and finally* which changes to the text are needed ?

  - define names  (point 1b)

    how do we call them ?  string and long-string, non-continuable and
    continuable ? single record string and continued string ?

    do we need to introduce a definition sentence in each if 4.2.1.1 and
    4.2.1.2 ? or change the titles of the subsections ?

  - add Bill/Mark sentence ( point 4)

    Bill's sentence could be added in 4.2.1.2 after the 1..5 bullet list
    "The CONTINUE keyword must not be used with of any of the mandatory or
    reserved keywords defined in this standard unless explicitly stated
    otherwise."

    This sentence implies that private or project kwds not defined in the
    standard are continuable.

    Mark's sentence could be added in the same place
    'CONTINUE may only be used for keywords that are declared to have a
    value of type "xxxxxxxxxxx".  It must not be used for keywords of type
    "yyyyyy"'

    but requires the type names to defined. The sentence applies also to
    private or project kwds, allowing them to choose among the two 'types'.

   Since the current standard mixes the definition of "string" and
   "character" one should also add that "Keywords defined in this standard
   as 'string' or 'character' are assumed to be of type 'yyyyyy'"

  - length limit (5)

   One possibility is to explicitate it at the end of the first sentence of
   4.2.1.2

   "... for continuing arbitrarily long string values over A POTENTIALLY
   UNLIMITED SEQUENCE OF multiple consecutive keyword records using the
   following procedure:"

   Otherwise we may add after Bill's or Mark's sentence a clause.

   "The number of CONTINUE keyword records following each continuable
   keyword is arbitrary and potentially unlimited"

Have a nice weekend.

-- 
------------------------------------------------------------------------ 
Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy) For 
more info : http://www.iasf-milano.inaf.it/~lucio/personal.html 
------------------------------------------------------------------------ 
Do not like Firefox >=29 ?  Get Pale Moon !  http://www.palemoon.org