[fitsbits] BINTABLE convention for >999 columns

Fri Jul 28 10:32:07 EDT 2017

My opinion is that you should go with the HIERARCH convention, Since 
you're talking about something that won't be formally part of the FITS 
standard, the fact that it relies on something else which isn't formally 
part of the FITS standard, but which is well described and accepted, is 
not a very strong objection to me.  To my mind, the easy readability and 
potentially greater applicability outweigh the objection.

My $0.02 worth,

Bill Thompson

On 7/28/17 10:05 AM, Mark Taylor wrote:
> Coming back to this after a bit of a breather:
>
> To summarise the dicussion, enthusiasm for my proposed
> convention for wide (>999 column) BINTABLES has not been
> universal, but I am still planning to implement something
> along these lines for my purposes (STIL/STILTS/TOPCAT).
> The possibility exists of other software deciding to recognise
> such a convention at some point in the future, but I'm not
> relying on that or even necessarily recommending it.
>
> In terms of the details, there was one main difference of opinion,
> namely how to store the column metadata for the 'extended'
> columns in the FITS header.  The suggestion I put forward was
> to use a base-26 number giving headers TFORMAAA - TFORMZZZ,
> which leads to a limit of 18574 columns.  Francois-Xavier
> Pineau suggested instead using the HIERARCH convention,
> which would allow a more or less unlimited column count.
>
> For concreteness, this HIERARCH-based variant differs from
> my original proposal
> (https://listmgr.nrao.edu/pipermail/fitsbits/2017-July/002967.html)
> in the following way:
>
>      - Metadata for each extended column is encoded with keywords
>        of the form HIERARCH XT XXXXXnnnnn, where XXXXX
>        are the same keyword roots as used for normal BINTABLE extensions,
>        and nnnnn is a decimal number written as usual (no leading zeros,
>        as many digits as required).  Thus the formats for data
>        columns 999, 1000, 1001 etc are declared with the keywords
>        HIERARCH XT TFORM999, HIERARCH XT TFORM1000, HIERARCH XT TFORM1001
>        etc.  Note this uses the ESO HIERARCH convention described at
>        https://fits.gsfc.nasa.gov/registry/hierarch_keyword.html.
>        The "name space" token has been chosen as "XT" (extended table).
>
> and the example header looks identical to my original example up
> to TFORM999, but the remaining entries differ:
>
>     TTYPE998= 'var_min_s_2'        /  label for column 998
>     TFORM998= 'D       '           /  format for column 998
>     TUNIT998= 'counts/s'           /  units for column 998
>     TTYPE999= 'XT_MORECOLS'        /  label for column 999
>     TFORM999= '813I    '           /  format for column 999
>     HIERARCH XT TTYPE999         = 'var_min_u_2' / label for column 999
>     HIERARCH XT TFORM999         = 'D' / format for column 999
>     HIERARCH XT TUNIT999         = 'counts/s' / units for column 999
>     HIERARCH XT TTYPE1000        = 'var_prob_h_2' / label for column 1000
>     HIERARCH XT TFORM1000        = 'D' / format for column 1000
>      ...
>     HIERARCH XT TTYPE1203        = 'var_prob_w_2' / label for column 1203
>     HIERARCH XT TFORM1203        = 'D' / format for column 1203
>     HIERARCH XT TTYPE1204        = 'var_sigma_w_2' / label for column 1204
>     HIERARCH XT TFORM1204        = 'D' / format for column 1204
>     HIERARCH XT TUNIT1204        = 'counts/s' / units for column 1204
>     END
>
> I have implemented and tested both variants, and they both work.
> The HIERARCH solution is a bit messier to do because it relies
> on a non-standard convention.
>
> Summarising the pros and cons of these two variants:
>
>     Base-26:
>      - limited to 18,000 columns ...
>        ... but nobody has come up with a plausible case to need more
>      - looks kludgy
>      - not very human readable
>
>     HIERARCH:
>      - requires non-FITS convention (HIERARCH)
>      - effectively no column count limit
>      - 13 or so fewer characters available for column keyword values
>      - easily human readable
>
> The balance of opinion in this thread of those who have expressed
> a preference between the two seems to have been in favour of the
> HIERARCH option (Francois-Xavier Pineau, Bill Pence, Tom McGlynn)
> as opposed to the Base-26 option (me, Rob Seaman, Arnold Rots?).
> In view of that, and the nagging worry that somebody might come
> up with some reason to store 20k+ columns, I think I'm just
> about coming down on the HIERARCH side, though it does look
> less FITSy to me.
>
> This message is to give a last chance for anybody to weigh in
> on one side or the other of the Base-26/HIERARCH question,
> in particular anybody who thinks they might end up one day
> wanting to implement support for this (which may be nobody!).
> If there is no more input on that question (which is fine by me),
> I'll decide one way or the other, implement and release it in
> STIL/STILTS/TOPCAT, and report back here.
>
> Thanks for reading and for the community input on this.
>
> Mark
>
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>

-- 
William Thompson
NASA Goddard Space Flight Center
Code 671
Greenbelt, MD  20771
USA

301-286-2040
William.T.Thompson at nasa.gov