[fitsbits] BINTABLE convention for >999 columns

[Francois-Xavier PINEAU]

Dear Mark, Bill and fitsbits,

Currently, wide BINTABLEs are not supported in FITS only because of the 3
characters available to encode a column index (and currently each of the 3
characters must be a digit).

The problem is "only" due to a limitation on the way FITS keyword names
are "legally" written (the 8-character limit).
Not to a possible problem on the interpretation of those keywords
(TXXXX + column index).

Thus, in my opinion, the "problem" should be solve during the header parsing
phase only. The interpretation of parsed keywords should remain unchanged
(I get the TFORM of col nnnn by looking at keyword TFORMnnnn).

I am reluctant to embrace the idea of defining a new convention
impacting the interpretation part of a FITS reader:
if at some point FITS provide a standard mechanism relaxing the 8-character
limit (and I hope so), the wide BINTABLE convention will be obsolete,
and the additional code written to interpret 'XT' will become useless
(but probably maintained for compatibility with wide BINTABLEs written 
before
the hypothetical new FITS standard).

Without the XT name space, only the way wide BINTABLEs headers are written
will be changed (no more need for HIERARCH), not the interpretation part of
FITS readers.

My suggestions would be:
1 - to restore section 2 of the HIERARCH conventions document
2 - to add a 3rd section (or a simple sentence in section 2) stating that
     if a HIERARCH keyword contains the same name as a regular keyword,
     it MUST be declared after the regular keyword and it MUST overload it.

This is a more general convention (Mark, everybody looking at your code
knows you like genericity), solving the wide BINTABLE problem and having
no impact on the keyword interpretation part of FITS readers.

If it is adopted, I expect commonly used FITS readers to be already or 
almost
able to support wide BINTABLEs.

I agree that it is a little bit more complex for FITS writers if they
want to support wide BINTABLES: they have to detect TFIELDS > 999
to add a few HIERARCH keywords having a length = 8.


François-Xavier Pineau


P.S.: thank you Bill for the "bit of history", I am on the list for less 
than
two years, and pretty new in the FITS world.

Le 01/08/2017 à 00:50, Mark Taylor a écrit :
> On Mon, 31 Jul 2017, William Pence wrote:
>
>> None the less, CFITSIO continues to support this more general convention.  If
>> a user attempts to write a keyword longer than 8 characters the CFITSIO
>> routines will silently encode it in the FITS header using the HIERARCH
>> convention, and similarly will look for the appropriate HIERARCH keyword when
>> reading a long keyword.
>>
>> Getting back to Mark's BINTABLE convention, I agree with François-Xavier that
>> eliminating the 'XT' domain name would simplify the BINTABLE convention a
>> little bit.  Using CFITSIO, one could then simply read or write the
>> "TFORM1234" keyword, instead of "XT TFORM1234".
> But not all the keywords defined by this convention have more than
> 8 characters.
>
> Using Francois-Xavier's suggestion, we have
>
>     ...
>     TTYPE998= 'var_min_s_2'        /  label for column 998
>     TFORM998= 'D       '           /  format for column 998
>     TUNIT998= 'counts/s'           /  units for column 998
>     TTYPE999= 'XT_MORECOLS'        /  label for column 999
>     TFORM999= '813I    '           /  format for column 999
>     HIERARCH TTYPE999         = 'var_min_u_2' / label for column 999
>     HIERARCH TFORM999         = 'D' / format for column 999
>     HIERARCH TUNIT999         = 'counts/s' / units for column 999
>     HIERARCH TTYPE1000        = 'var_prob_h_2' / label for column 1000
>     HIERARCH TFORM1000        = 'D' / format for column 1000
>     ...
>
> Just using CFITSIO's transparent long-keyword-means-HIERARCH rule
> to get the extended keywords would do the right thing for TFORM1000,
> TFORM1001 etc, but if you do it for TFORM999 you will get the
> wrong thing, namely the TFORM999 'container' column rather than
> the HIERARCH TFORM999 'extended' column.  And if you look for
> TDIM1234 presumably CFITSIO will attempt to retrieve the
> non-existent and illegal TDIM1234 rather than the intended
> HIERARCH TDIM1234 keyword.
>
> On Mon, 31 Jul 2017, Francois-Xavier PINEAU wrote:
>
>> I support an "overloading" behaviour (like e.g. in CSS): same keywords
>> (TFIELDS, TXXXX999) in a more specialized context (HIERARCH) should be able to
>> overload "regular" keywords that ensure the legality of the FITS
>> file (again, waiting for an updated standard).
> Francois-Xavier suggests that the HIERARCH values take precedence
> over the non-HIERARCH ones in these cases.  That is possible but
> if I understand correctly would require a change of the CFITSIO
> behaviour that Bill describes, and in any case sounds rather opaque
> (it's not obvious whether the HIERARCH or non-HIERARCH form
> should take precedence).  The possibility of that kind of confusion
> looks to me like a good reason to stick to the namespace-required
> rule - then you know that the set of HIERARCH-derived keywords
> is the same as the set of keywords with embedded "." characters.
>
> In general I prefer the idea that software wanting to use this
> convention has to explicitly 'opt in' by recognising the convention
> in use, than that it manages to work with no client code changes
> given that the underlying library code works in a particular
> (non-standard) way.
>
> I am also a bit uncomfortable about using the HIERARCH convention
> in a way that deviates from the registered text of that convention
> (no namespace).  Since it's all outside the FITS standard anyway
> that's not a showstopper, but it makes the documentation a little
> bit more complicated (it's using something like the HIERARCH
> convention registered on the FITS web pages, but with no namespace).
>
> Mark
>
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/