[fitsbits] BINTABLE/TABLE column count limitation

[Rob Seaman]

FITS doesn't seek to be the answer to every question.  TOPCAT can use any internal format that makes sense.  However, the original TOPCAP issue:

	https://sympa.bris.ac.uk/sympa/arc/topcat-user/2012-06/msg00003.html

reported "the problem is that after creating the Tables I use them in other scrips/programs (e.g. within IDL) and the FITS format is the most flexible format to work with."

The problem FITS was created to solve is precisely that, of providing a flexible interchange format.  Whether or not TOPCAP provides support for 1000+ columns internally, this particular user would need to trim the number of columns to use the tables in IDL.  The question can't really be unasked at this point.

I'm skeptical about the presence of a community mandate for a FITS2 effort, and even some of the more modest changes to keyword names or lengths seem to lack consensus.  On the other hand, Lucio has already laid out most of the keywords that would be needed to split wide tables apart, and these are undoubtedly very similar to the ones needed to logically join narrow tables together :-)

In fact, if the issue is joining tables, such a scheme would very likely be significantly quicker on writing (simply concatenate the two table extensions omitting some rows of the two tables and updating a few keywords), and shouldn't be significantly slower on reading even record-by-record since the offsets into the two tables are computable.  Editing the logically-wide table would require updating two extensions, but the individual tables would be smaller similar to the tile-compression formats.  Performance should be similar to - no worse than, and perhaps even better than - a single leviathan of a table.  In fact, independent of compression, this is one way to tile tables in support of random-access efficiencies.

And such a format is already legal FITS.  The best change to a standard is no change.

Rob