[fitsbits] BINTABLE convention for >999 columns

[Mark Taylor]

On Mon, 31 Jul 2017, William Pence wrote:

> None the less, CFITSIO continues to support this more general convention.  If
> a user attempts to write a keyword longer than 8 characters the CFITSIO
> routines will silently encode it in the FITS header using the HIERARCH
> convention, and similarly will look for the appropriate HIERARCH keyword when
> reading a long keyword.
> 
> Getting back to Mark's BINTABLE convention, I agree with François-Xavier that
> eliminating the 'XT' domain name would simplify the BINTABLE convention a
> little bit.  Using CFITSIO, one could then simply read or write the
> "TFORM1234" keyword, instead of "XT TFORM1234".

But not all the keywords defined by this convention have more than
8 characters.

Using Francois-Xavier's suggestion, we have

   ...
   TTYPE998= 'var_min_s_2'        /  label for column 998
   TFORM998= 'D       '           /  format for column 998
   TUNIT998= 'counts/s'           /  units for column 998
   TTYPE999= 'XT_MORECOLS'        /  label for column 999
   TFORM999= '813I    '           /  format for column 999
   HIERARCH TTYPE999         = 'var_min_u_2' / label for column 999
   HIERARCH TFORM999         = 'D' / format for column 999
   HIERARCH TUNIT999         = 'counts/s' / units for column 999
   HIERARCH TTYPE1000        = 'var_prob_h_2' / label for column 1000
   HIERARCH TFORM1000        = 'D' / format for column 1000
   ...

Just using CFITSIO's transparent long-keyword-means-HIERARCH rule
to get the extended keywords would do the right thing for TFORM1000,
TFORM1001 etc, but if you do it for TFORM999 you will get the
wrong thing, namely the TFORM999 'container' column rather than
the HIERARCH TFORM999 'extended' column.  And if you look for
TDIM1234 presumably CFITSIO will attempt to retrieve the
non-existent and illegal TDIM1234 rather than the intended
HIERARCH TDIM1234 keyword.

On Mon, 31 Jul 2017, Francois-Xavier PINEAU wrote:

> I support an "overloading" behaviour (like e.g. in CSS): same keywords
> (TFIELDS, TXXXX999) in a more specialized context (HIERARCH) should be able to
> overload "regular" keywords that ensure the legality of the FITS
> file (again, waiting for an updated standard).

Francois-Xavier suggests that the HIERARCH values take precedence
over the non-HIERARCH ones in these cases.  That is possible but
if I understand correctly would require a change of the CFITSIO
behaviour that Bill describes, and in any case sounds rather opaque
(it's not obvious whether the HIERARCH or non-HIERARCH form
should take precedence).  The possibility of that kind of confusion
looks to me like a good reason to stick to the namespace-required
rule - then you know that the set of HIERARCH-derived keywords
is the same as the set of keywords with embedded "." characters.

In general I prefer the idea that software wanting to use this
convention has to explicitly 'opt in' by recognising the convention
in use, than that it manages to work with no client code changes
given that the underlying library code works in a particular
(non-standard) way.

I am also a bit uncomfortable about using the HIERARCH convention
in a way that deviates from the registered text of that convention
(no namespace).  Since it's all outside the FITS standard anyway
that's not a showstopper, but it makes the documentation a little
bit more complicated (it's using something like the HIERARCH
convention registered on the FITS web pages, but with no namespace).

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/