New CFITSIO Test Report
William Pence
pence at tetra.gsfc.nasa.gov
Fri Dec 6 13:01:04 EST 1996
Optimized Version of CFITSIO is Now Available
---------------------------------------------
A new, much faster version of CFITSIO, the portable ANSI-C library for
reading and writing FITS format data files, is now available. This new
version (v1.11) achieves data throughputs of several megabytes per
second on current generation workstations and, for example, can read
or write a 1000 x 1000 integer*2 image or a binary table with 100,000
rows and 5 real*4 columns in under one second. The throughput is
primarily limited by the I/O speed of the underlying magnetic disk
which shows that there is no significant data throughput penalty for
using FITS as a run-time data analysis format. A more detailed report
on the current performance of CFITSIO is attached below for those that
may be interested.
Both the CFITSIO (ANSI C) and the older FITSIO (Fortran-77) software
libraries may be obtained on the WWW from:
http://heasarc.gsfc.nasa.gov/fitsio
or by anonymous ftp from: legacy.gsfc.nasa.gov in the software/fitsio
directory.
-------------------------------------------------------------------------
William Pence
High Energy Astrophysics Science Archive Research Center
NASA/GSFC
e-mail: pence at tetra.gsfc.nasa.gov
phone: (301)286-4599
-------------------------------------------------------------------------
-------------------------------------------------------------------------
Performance Testing of CFITSIO
December 1996
There has been considerable interest in the efficiency of using FITS as
a data analysis format (as opposed to just a data archive format)
therefore a series of tests have been performed on a variety of modern
workstations to compare the speed of reading and writing FITS files
with CFITSIO to that of reading and writing raw binary files using
direct calls to the low-level fwrite and fread C library routines. The
gcc compiler was used on the following test platforms:
- a 133MHz pentium with 24MB RAM running Linux
- a Silicon Graphics INDY workstation
- a 300MHz Alpha OSF/1 workstation
- a 167MHz SUN Ultra-2 workstation
A test program called speed.c (included in the CFITSIO distribution)
was written to measure the time required to first write then read
back large (~25 MB) data files. Large files were used to minimize
the affects of disk memory caching which can inflate the measured
I/O speeds.
The test results are summarized in the following table, where the
throughput is give in MB/s followed by the percentage of CPU usage
(i.e., CPU time / Elapsed time * 100) given in parentheses. There were
sometimes large fluctuations in the measured throughput, especially for
the raw binary files, so these figures represent an average of several
runs of the program.
CFITSIO Performance Benchmarks
figures in MBytes/s (%CPU usage)
133-P5 Alpha SGI INDY SUN Ultra-2
System: Linux OSF/1 IRIX Solaris
-----------------------------------------------
Raw Write 3.5(35) 4.5(20) 5.5(25) 5.5(15)
Raw Read 2.4(35) 4.5(15) 5.5(25) 5.5(10)
-----------------------------------------------
Write FITS
Image 3.8(65) 4.5(55) 5.5(30) 5.5(15)
Bintable 2.2(85) 3.5(70) 3.9(90) 4.5(85)
ASCII Tbl .15(95) 0.8(90) 0.8(90) 1.0(90)
-----------------------------------------------
Read FITS
Image 3.0(55) 5.5(60) 5.0(20) 5.5(15)
Bintable 2.3(80) 4.6(80) 3.4(70) 4.5(75)
ASCII Tbl 1.4(90) 2.0(90) 1.4(85) 1.2(75)
-----------------------------------------------
The following conclusions have been drawn from these figures:
1. When writing or reading FITS IMAGEs, the throughput of CFITSIO is
virtually the same as that of the fast raw binary file I/O. Accessing
FITS images requires relatively little CPU time (since the data are
contiguous on disk) thus the throughput is mainly limited by the I/O
speed of the magnetic disk.
2. CFITSIO's throughput for FITS binary tables is only slightly slower
(about 20% less) than for FITS images. In this case the CPU usage is
higher because the data in a FITS column are not stored consecutively
on disk, and it requires more computation to jump to each successive
row in the table. It appears that the CPU speed of the current
generation of machines closely complements the disk access speed, so
that neither factor is dominant in limiting the throughput to FITS
binary tables. One would expect that in the future, as the CPU speeds
continue to increase faster than disk access speeds, that the
throughput of FITS tables will become more heavily disk I/O limited,
just as in currently the case with FITS images.
3. The throughput for FITS ASCII tables is only about 1/3 as fast as
for binary tables because of the extra CPU time required to format or
parse the ASCII character fields. This, coupled with the fact that
ASCII tables tend to be about twice as large as binary tables for the
same information content means that the effective throughput for ASCII
tables is about 5 times slower than for binary tables. Clearly, ASCII
tables should not be used in applications which require maximum data
I/O performance.
4. There is no apparent performance degradation on the machines that
require byte swapping when accessing the FITS file (i.e., the Pentium
and Alpha systems). This is not surprising because the CPU time to
swap bytes is small compared to the disk I/O rates.
Finally, a similar Fortran test program was written to compare the
throughput of the FITSIO subroutine library (written in FORTRAN-77)
with CFITSIO. As expected CFITSIO is currently faster, largely because
it was designed using the experience gained from the older FITSIO code
and to a lesser extent because it is sometimes easier and more
efficient to perform low-level data manipulations in C rather than
Fortran-77. These preliminary tests indicate that CFITSIO is about
twice as fast as FITSIO when reading or writing FITS images and about 5
times faster when reading or writing FITS binary tables. Work is now
in progress to improve the performance of FITSIO to bring it closer to
that of CFITSIO.
--------------------------- End of Report ----------------------------
More information about the fitsbits
mailing list