[fitsbits] Fastest way to read a binary table

[Phil Hodge]

Clive Page wrote:

> One of my colleagues (in Edinburgh) has been pushing the idea that
> astronomers ought to store their tables in column-major order.  Some
> systems do that, e.g. ESO/MIDAS table system, and there is (as far as I
> recall) a user-level choice in the STSDAS table system invented at STScI.

Yes, the STSDAS table format can be either row-ordered or column-ordered,
but the column-ordered option was a mistake, in my opinion.  It's fine
if you only read/write column-by-column, but many of the generic tools
operate on tables row-by-row, e.g. selecting rows, or sorting when you
get to the point of physically rearranging the data, or writing to a FITS
file.  We found that these operations were extremely slow on large,
column-ordered tables.  Suppose you have a table with five columns (for
example) and many rows.  If the table is row ordered, reading it column-
by-column requires five passes through the file, which might not be too
much overhead.  If the table is column ordered, however, reading it row-
by-row requires almost as many passes through the file as there are rows,
and that was painful.

Phil