[fitsbits] BINTABLE/TABLE column count limitation

[Eric Greisen]

Rob Seaman wrote:
> On Jun 6, 2012, at 4:45 AM, Phil Hodge wrote:
> 
>> On 06/06/2012 06:57 AM, Mark Taylor wrote:
>>> There is an acknowledged limitation of 999 on the maximum number of columns in a FITS table.
>> The problem is obvious, and so is the solution:  Change FITS to allow keywords longer than eight characters.
> 
> A limitation is not necessarily a problem.  And the colloquial use of the word "problem" is not the same as a coherent characterization of the problem space.
> 
> A lot of code and data have been built against the 999 limitation.  What are the use cases demanding this be changed?  Do these rise to the level of taking action? Are workarounds (like splitting tables across two bintable extensions) not acceptably performant?  Are 9999 columns enough?
> 
> In the solution space, I suspect I'm not the only one to be skeptical that keyword length can be changed at this late date.  Usually the prescription for a change to some fundamental FITS design choice is that it be relatively harmless when passed through unchanged legacy code.  This would rather cause massive failure not just to table-handling code, but at the most basic level of parsing the structure of a FITS file.  That being the case why wouldn't such a project rather choose a different standard entirely as per Perry's plea at the 2010 BoF?
> 
> All that said, there are other options.  Instead of adding more decimal digits (either by cannibalizing alphabetic characters from the current 8, or by lengthening the keywords), surely we could consider a non-decimal base for the current three digits?  Going to hex (starting with "A00" for backwards compatibility) would permit representing an additional 1500+ columns.  Going to base-36 (0-Z) would support 34695 total columns (if I did the arithmetic right, and again starting with A00).
> 
> Presumably there are several other options.  What are the trade-offs?  What are the implications for performance and logistics of such large tables?  Undoubtedly there are further contingent issues related to other features of the standard and conventions as Arnold says about WCS.
> 
> Not obvious to me.

I second these comments - what is the actual use case?  Are we designing 
tables to have single-value columns when in fact the columns should 
contain arrays of numbers simply because "tables" of the old fashion 
sort do not contain arrays in single cells?  Note that whole images can, 
and for lexical simplicity, should be present in a single cell.  The 
BINTABLE convention arose because of the need for array values in cells 
as well as for the efficiency of binary over ASCII.

Eric Greisen