[fitsbits] BINTABLE/TABLE column count limitation

[Lucio Chiappetti]

On Wed, 6 Jun 2012, Rob Seaman wrote:

> A lot of code and data have been built against the 999 limitation.

I presume the max TFIELD=999 was done for analogy with max NAXIS = 999. 
The latter was obviously related to the NAXIS1 ... NAXIS999 keywords, so 
5-char prefix and the 8-char keyword name lead to that in the very 
beginning.

BINTABLEs came quite later, possibly one could have chosen a prefix 
shorter than 5-char TFORM ... but I guess nobody considered practical 
usage of tables with more than 999 columns.

And the "once FITS forever FITS" principle makes rather tricky to go 
beyond the 8-char kwd name limit (unless one uses very specific 
conventions for very specific purposes, e.g. ESO HIERARCH)

> Undoubtedly there are further contingent issues related to other 
> features of the standard and conventions as Arnold says about WCS.

I had forgotten to check about WCS, but Arnold's point are very relevant 
indeed.

> What are the use cases demanding this be changed?  Do these rise to the 
> level of taking action? Are workarounds (like splitting tables across 
> two bintable extensions) not acceptably performant?  Are 9999 columns 
> enough?

I suspect (to reply to Eric's question) the point is not a use of 1-deep 
columns instead of n-deep columns, but just mapping into some database 
tables.

I know that some database producers (fillers ? photometrists for once) 
like to create database with many many columns (let us store this 
parameter too, one never knows if somebody will require it).

One of my usual jobs is to import catalogues in a database and I usually 
try to select a subset of really useful columns.

I guess I will hardly consider practical to work with a db with more than 
1-2 hundred columns. But tastes may differ.

Anyhow, if the issue is to be able to export from one environment part of 
a database table in FITS and later re-import it in the SAME environment, I 
guess one can come up with an easier convention, specific of such 
environment, and consistent with the existing standard :

  - define a special keyword (or a few keywords) as signatures of such
    convention

  - dump the first n (max 999) columns in a BINTABLE extension
  - dump the next n in a second extension
  - and so on

The reverse of course would be to read each extension and juxtapose in 
memory the table pieces.

Lucio