[fitsbits] BINTABLE/TABLE column count limitation

[Arnold Rots]

The problem is that you will be running afoul of the WCS-related
keywords which already use additional characters, the most extreme
being the ones for vector columns, like iCTYna.

  - Arnold

Mark Taylor wrote:
> Hi FITS,
> 
> There is an acknowledged limitation of 999 on the maximum number of
> columns in a FITS table.
> 
> The TABLE and BINTABLE extensions define the following per-column fields:
> 
>    TBCOLn  (ASCII table only)
>    TDIMn   (Binary table only)
>    TDISPn
>    TFORMn
>    TNULLn
>    TSCALn
>    TTYPEn
>    TUNITn
>    TZEROn
> 
> to describe per-column metadata for the encoded table.  Along with
> the 8-character limitation on header card keywords, this limits the
> number of columns that can be described to 999; TFORM999 is legal,
> but TFORM1000 is not.  The standard explicitly constrains the value of
> the TFIELDS keyword to <=999 in acknowledgement of this limitation.
> 
> With column counts from the large surveys in the hundreds, a couple of
> table joins can have you hitting this restriction.  I don't know what
> the data from LSST etc will look like, but extrapolating survey 
> column counts over time would suggest that single tables in the 
> thousand-column range may be upon us soon.  A user of TOPCAT
> has recently reported encountering this in science use
> (https://sympa.bris.ac.uk/sympa/arc/topcat-user/2012-06/msg00003.html),
> so already it's not merely a theoretical problem.
> 
> As a pragmatic solution, I suggest the following convention.  Columns
> before the 1000th in any table are described as per the existing
> standard, but for columns 1000-9999, the 5th alphabetic character 
> of the Txxxx keyword (if present - no change reqired for TDIMn)
> is removed to make space for an additional digit.  The existing
> constraint that the TFIELDS value shall be <=999 is also of
> course relaxed to <=9999.  Thus:
> 
>    XTENSION= 'BINTABLE'
>    ...
>    TFIELDS = 2112
>    ...
>    TFORM998= 'D'
>    TTYPE998= 'foo'
>    TFORM999= 'D'
>    TTYPE999= 'foo_err'
>    TFOR1000= 'D'
>    TTYP1000= 'bar'
>    TFOR1001= 'D'
>    TTYP1001= 'bar_err'
>    ...
> 
> Under this rule, any table with fewer than 1000 columns looks exactly
> the same as it does now.  Columns >999 in wide tables will be unreadable
> by software which is not aware of the convention, but such software
> would be incapable of dealing with 1000+-column tables in any case.
> Depending on implementation, non-aware software may be able to make
> sense of the first 999 columns of wide tables.
> 
> I am considering implementing this convention in the FITS I/O handlers
> used by STIL (the java table library used by TOPCAT and STILTS as well
> as some other client- and server-side applications).  If nothing else
> this will enable STIL users to generate syntactically legal FITS files
> (though containing illegal BINTABLE extensions) representing 1000+
> column tables which they can use within STIL (e.g. to save/load tables
> in TOPCAT), even if such files are not legible by other FITS table
> applications, while I/O of tables with <1000 columns will be unaffected.
> 
> However, if others choose to implement the same convention it could
> become a de facto standard for wide tables, and possibly a candidate
> for an update of the BINTABLE convention in a future version of the
> FITS standard.
> 
> Does anybody forsee problems with this suggestion, or want to suggest
> a better alternative?  The only possible backward compatibility issue
> or unintended consequence I can think of is if there are already
> keywords along the lines of TFORxxxx (x=[0-9]) in use in existing
> table headers, but it seems rather unlikely.  The other question is
> whether 9999 is enough.  1e5-column tables are probably a little
> way off, and extending this scheme to 5-digit column indices would
> be problematic since TDIMn and TDISPn would both degenerate to TDInnnnn,
> so I'd suggest punting that issue to future generations.
> 
> Mark
> 
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/
> 
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits
> 
--------------------------------------------------------------------------
Arnold H. Rots                                Chandra X-ray Science Center
Smithsonian Astrophysical Observatory                tel:  +1 617 496 7701
60 Garden Street, MS 67                              fax:  +1 617 495 7356
Cambridge, MA 02138                             arots at head.cfa.harvard.edu
USA                                     http://hea-www.harvard.edu/~arots/
--------------------------------------------------------------------------