[fitsbits] Fastest way to read a binary table

[William Pence]

Reading N columns 1 at a time would require N passes through the file.  For
maximum efficiency it would be better to read all the columns of data in one
pass through the file.  You can do this by reading the first M rows from
every column, then reading the next M rows, and so on until reaching the end
of file.  The number of rows to read in each iteration, M, can be determined
by calling the 'fits_get_rowsize' routine in CFITSIO.  The value of M is not
very critical; as a rough rule of thumb you could try using M = 60000 /
NAXIS1 where 'NAXIS1' is the width of one row of your table in bytes. 
Experiment with different values of M to see if it makes much difference. 
Modern computers and disk drives use quite sophisticated memory caching
algorithms, so most of the time is often spent reading the file the first
time, and any additional passes through the file have much less overhead.

The Iterator routine in CFITSIO uses this technique of reading M rows at a
time for maximum speed, but in your case it is probably not worth the
additional effort to learn how to use the Iterator. 

-Bill Pence
-- 
____________________________________________________________________
Dr. William Pence              pence at _DeleteThis_tetra.gsfc.nasa.gov
NASA/GSFC Code 662         HEASARC         +1-301-286-4599 (voice)     
Greenbelt MD 20771                         +1-301-286-1684 (fax)

"Patrick L. Nolan" wrote:
> 
> Suppose I have a FITS file that contains a large binary table
> and I want to read it all into my program using cfitsio.
> By large I mean several hundred thousand rows.  And I have
> several hundred files.
> What's the fastest way to do it?
> 
> Currently I allocate an array for each column and then use
> fits_read_col to read each.  I'm worried that it might be
> reading the whole table into temporary storage each time I
> call fits_read_col.  Does anyone know?
> 
> I read about FITS iterators, but that looks like a steep
> learning curve.  Is it worth it?
> 
> Or is there something simple I missed?
> --
> *   Patrick L. Nolan                                          *
> *   W. W. Hansen Experimental Physics Laboratory (HEPL)       *
> *   Stanford University                                       *