[fitsbits] BINTABLE/TABLE column count limitation

[Mark Taylor]

On Thu, 7 Jun 2012, Forveille thierry wrote:

>   I concur with many that we first of all need a description of actual use
> cases, with considerably more motivation than someone coming up
> against error messages, possibly after doing something that's quite far
> from optimal (e.g. using many scalar columns for a vector).

As I tried to make clear in my initial message, this is not about
using multiple scalar columns where a single array column would be
more appropriate, it's about handling wide tables which do,
increasingly, exist in astronomy.  The PhotoObjAll table from SDSS 
is 446 columns wide.  That is less than 999, but it's in the
same ball park.  If you do a join of two tables of that kind
of size (or an internal join), you're very close to hitting that limit.
I think the user report that started me off on this was
the result of a table join between three or more wide tables, but
it doesn't really matter - from these numbers, and the fact that
tables especially from large and complicated survey instruments
seem to be getting wider, my guess is that it will hit us
eventually, *if* we want FITS to be usable as a generic container for
all the kinds of tables that people sometimes want to generate.
Do we want that?  I don't know about the FITS community, but to
serve my users I want to be able to store 1000+ column tables on 
disk in a way I can read again efficiently.  It doesn't have to be
using FITS, but it would be convenient for me (and my users) if it were.

Yes the user in question probably wasn't going to make use of every
one of those 1000+ columns.  Probably some astronomers who use
large image files don't look at every single pixel either.  As a
developer of generic data handling software I do not consider it
my responsibility to incentivise people to slim down their data by
providing tools incapable of handling larger datasets.

I aired this on fitsbits to ask whether people could see problems with
my workaround that I hadn't seen (they did) and to find out whether this
is something that other people have come across or are thinking about
(apparently not).  I have already backed out of advocating my original
suggestion as a change to the FITS standard.  I suspect that the
case for addressing this may become more pressing in the future,
but I certainly don't think it's serious enough to warrant major
violence to the existing FITS standard (e.g. relaxing the 8-character
keyword limit).  From this discussion it seems that the appetite for
wide tables in the FITS community is not strong enough to warrant a change
at the standards level, so it may be time to wind up this discussion
at least for now.  In the mean time I may decide to use one or other
of the suggestions that's come up here or an alternative of my own
as a private convention to serve the data storage requirements of
my software (and its users).

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/