[wfc] CHECKSUM Proposal

[Don Wells]

Peter Teuben writes:
 > Dear WGAS Fits Committee members (and listeners):

I am a listener, on the WFC mailing list ex officio.

 > Bob Seaman, Bill Pence and Arnold Rots.. CHECKSUM
 > proposal... for review..
 > http://heasarc.gsfc.nasa.gov/docs/heasarc/fits/checksum.html

I want CHECKSUM to be implemented in the software which generates all
astronomical archives in order to detect errors of various types.

Our modern data recording hardware is using error correcting codes;
generally they can at least correct single bit errors in records, and
some of them may correct multiple bit errors. Modern backplane busses
are reliable, and some of them have error detection (and
correction?). Network media are reliable, and they have error
detection and re-transmission schemes. However, our modern archives
are very large, so that the very small bit error rates of the modern
hardware systems are now being multiplied by rather large numbers of
bits. How small is the final product rate?  I.e., how often do errors
happen in the long pipeline->archive->enduser chain?  It appears to me
that CHECKSUM offers us an effective tool to make that measurement.

Some archive media are known to degrade with time.  The playback
devices should give the archive managers warnings of degradation, in
the form of increasing rates of corrected bit errors. However,
CHECKSUMs in those archives will provide an additional end-to-end
check, one that will be monitored by endusers.

I expect that the error rate in our modern systems is so low that it
will be hard to measure.  I hope that it will prove to be so low that
every failure of a CHECKSUM will be worthy of a report to fitsbits
(sci.astro.fits).  We must have moved several terabytes through the
CHECKSUM algorithm by now; have any CHECKSUM failures been seen in
FITS files?

Three years ago I recommended to NRAO's Array Operations Center
personnel that they implement CHECKSUM in the VLBA Correlator Archive.
As far as I know, they have not done so. I would like to repeat that
recommendation, and also to recommend to NRAO's Green Bank people that
they implement CHECKSUM in the archive for the GBT [Green Bank
Telescope]. However the current form of the proposal contains only the
technical details, and so it won't help me in the process of
persuading the archive people to implement CHECKSUM.  In this respect
the 1995 Seaman and Pence paper was better. Does the proposal
presented to the WFC supercede the 1995 paper?  If 'yes', then I
recommend that the 1995 paper be upgraded so that I can use it. 

-Don Wells [Chair, IAU FITS Working Group]
-- 
  Donald C. Wells      Scientist - GBT Project        dwells at nrao.edu
                    http://www.cv.nrao.edu/~dwells
  National Radio Astronomy Observatory                +1-434-296-0277
  520 Edgemont Road,   Charlottesville, Virginia       22903-2475 USA
       (DCW is often in Green Bank, West Virginia, at +1-304-456-2146)