[fitsbits] BINTABLE/TABLE column count limitation

[Arnold Rots]

An obvious construction following option 5 is to actually implement
joins by putting the tables from the different sources each in a
separate HDU including an index, following an HDU that provides the
join through cross-index references.

  - Arnold

Tom McGlynn wrote:
> So far possible solutions to the 999 column limit suggested are:
> 
> 1. Reduce the size of the stem of column keywords for columns > 999
>       (..., TTYPE999, TTYP1000, ...)
> 2. Allow FITS keywords with > 8 characters,  TTYPE1000
> 3 [From the TOPCAT list] Use the currently reserved space after the =
>       TTYPE999= 'xxx'
>       TTYPE1000='yyy'
> 4. Use non-decimal integers for >999, TTYPEA00
> 
> I'd also suggest at least considering
> 
> 5. Breaking up the table into multiple HDUs, presumably using the Hierarchical
> Grouping Convention (or something similar).
> 
> The last has the advantage that it fully compatible with the existing FITS 
> standard, and the disadvantage that it requires reorganization of the data.
> 
> Any others?
> 
> If nothing else I think I agree with Bob that the best solution is not obvious.
> 
> 	Tom
> 
> Rob Seaman wrote:
> > On Jun 6, 2012, at 4:45 AM, Phil Hodge wrote:
> >
> >> On 06/06/2012 06:57 AM, Mark Taylor wrote:
> >>> There is an acknowledged limitation of 999 on the maximum number of columns in a FITS table.
> >>
> >> The problem is obvious, and so is the solution:  Change FITS to allow keywords longer than eight characters.
> >
> > A limitation is not necessarily a problem.  And the colloquial use of the word "problem" is not the same as a coherent characterization of the problem space.
> >
> > A lot of code and data have been built against the 999 limitation.  What are the use cases demanding this be changed?  Do these rise to the level of taking action? Are workarounds (like splitting tables across two bintable extensions) not acceptably performant?  Are 9999 columns enough?
> >
> > In the solution space, I suspect I'm not the only one to be skeptical that keyword length can be changed at this late date.  Usually the prescription for a change to some fundamental FITS design choice is that it be relatively harmless when passed through unchanged legacy code.  This would rather cause massive failure not just to table-handling code, but at the most basic level of parsing the structure of a FITS file.  That being the case why wouldn't such a project rather choose a different standard entirely as per Perry's plea at the 2010 BoF?
> >
> > All that said, there are other options.  Instead of adding more decimal digits (either by cannibalizing alphabetic characters from the current 8, or by lengthening the keywords), surely we could consider a non-decimal base for the current three digits?  Going to hex (starting with "A00" for backwards compatibility) would permit representing an additional 1500+ columns.  Going to base-36 (0-Z) would support 34695 total columns (if I did the arithmetic right, and again starting with A00).
> >
> > Presumably there are several other options.  What are the trade-offs?  What are the implications for performance and logistics of such large tables?  Undoubtedly there are further contingent issues related to other features of the standard and conventions as Arnold says about WCS.
> >
> > Not obvious to me.
> >
> > Rob
> >
> > _______________________________________________
> > fitsbits mailing list
> > fitsbits at listmgr.cv.nrao.edu
> > http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits
> 
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits
> 
--------------------------------------------------------------------------
Arnold H. Rots                                Chandra X-ray Science Center
Smithsonian Astrophysical Observatory                tel:  +1 617 496 7701
60 Garden Street, MS 67                              fax:  +1 617 495 7356
Cambridge, MA 02138                             arots at head.cfa.harvard.edu
USA                                     http://hea-www.harvard.edu/~arots/
--------------------------------------------------------------------------