[fitsbits] Fastest way to read a binary table

[John E. Davis]

On 9 Jul 2002 09:37:20 +0100, Clive Page <cgp at nospam.le.ac.uk>
wrote:
>Is it possible to recompile CFITSIO with a bigger buffer
>
>Would that make it more efficient at handling large binary tables?

If cfitsio reads files like most programs, e.g., via the "read" system
call, then that causes data to be copied twice: once from the file
into a kernel buffer and then into the program's buffer.  If the file
is opened in read-only mode, and no byte-swapping is necessary, a
faster approach might be to utilize the "mmap" system call, which maps
the file into the virtual address space of the process.  This would
probably work best for images.  For example, a program would be able
to read a large image, e.g., a 1024Mb image on a system that has
nowhere near that amount of memory-- virtual or otherwise.

This approach would probably not work well for binary tables because
they are stored in a fits file by rows and not by columns.  That is,
the data for the individual columns are interleaved.  Too bad there is
not an alternative type of extension that orders data by columns,
since that would allow variable length data to be stored much more
compactly than using the heap.

--John