[fitsbits] Large FITS files

[Steve Allen]

On Mon 2007-07-30T10:08:07 -0400, Robert J. Harris hath writ:
> I'm looking to process large ( > ~ GB) FITS datacubes.  They are large
> enough not to load into memory.  What I would like to do is input the
> cube, to perform a filtering operation on each plane of the cube, and then
> output another cube made up of the planes of the original cube after they
> have been processed.  If it weren't for the last part of it, I could use
> the extended filename syntax of cfitsio, but I do need to put the planes
> back together, and it's not obvious how I might do that.  Any help is
> appreciated.  Thanks in advance!

Our 64 Mpix images from the DEIMOS 8k*8k imager aren't that big, but
they seemed a bit more of a challenge years ago when we were first
preparing to write them.  We use the CFITSIO fits_create_memfile()
call after using the Unix shared memory routines to get a chunk of
shared memory.  Then we use the other routines to create the FITS
elements and store the offsets in data structures which are used
thereafter by code which is unaware that it is actually addressing
memory which conforms to FITS.

In early tests we experimented with using Unix memory mapped files
instead of shared memory.  It worked pretty much the same as far as
the use of fits_create_memfile() was concerned, but we found that the
operating system (with a little tuning of the kernel params) was
better at deciding how to do the swapping than our code was at trying
the same thing.

--
Steve Allen                 <sla at ucolick.org>                WGS-84 (GPS)
UCO/Lick Observatory        Natural Sciences II, Room 165    Lat  +36.99855
University of California    Voice: +1 831 459 3046           Lng -122.06015
Santa Cruz, CA 95064        http://www.ucolick.org/~sla/     Hgt +250 m