[fitsbits] Fastest way to read a binary table

[Clive Page]

In article <slrnaiminr.t7g.davis at aluche.mit.edu>,
John E. Davis <davis at space.mit.edu> wrote:
>
>This [memory mapping] approach would probably not work well for binary tables because
>they are stored in a fits file by rows and not by columns.  That is,
>the data for the individual columns are interleaved.  Too bad there is
>not an alternative type of extension that orders data by columns,
>since that would allow variable length data to be stored much more
>compactly than using the heap.

Thanks for your comments (although they didn't answer my original
question).  I think that memory mapping would not help much (if at all) for
binary tables for two reasons (1) as you say, data in one column are rather
sparse, as columns are stored in row-major order.
(2) The most popular hardware is 80x86, which is little-endian, so a byte
swap is needed for most data types.

One of my colleagues (in Edinburgh) has been pushing the idea that
astronomers ought to store their tables in column-major order.  Some
systems do that, e.g. ESO/MIDAS table system, and there is (as far as I
recall) a user-level choice in the STSDAS table system invented at STScI.

Can I therefore put forward the idea of doing this in FITS tables: to get
column-major order you just have a table of one row, but each column has N
elements in an array (using fixed-length vectors, not heap space).  I have
no idea how well CFITSIO would handle this for a large value of N (say
10**8 elements) but it would be interesting to try.




-- 
Clive Page   cgp at le.ac.uk