[wfc] Re: Next: Checksum proposal (revisited)

[Rob Seaman]

Arne says:

> Rob, our of the "several million" frames, how many had errors that
> were caught with the checksum?  And on what media?

I don't have all the log files at hand, but out of the last 660,000
images archived to NOAO's original "Save the Bits" data store, 20 tapes
(out of 318 pairs) failed the verification pass.  There is a high
correlation with episodes of our aging exabyte drives needing to be
sent out for periodic refurbishment.

One other advantage of including a checksum verification pass is
precisely that this can be used to detect media or equipment failure.

Note that NOAO is planning to transition to a system relying on
multiple, geographically separated, spinning disk copies.  The checksum
will provide a quick and easy method for maintaining synchronized copies.

Bill's example...

> I regard as one of the most important uses of the CHECKSUM keywords:
> The CHECKSUM keywords provide a simple mechanism for putting a 'validity
> stamp' or 'seal of approval' on the FITS data files that are retrieved
> from large public archives like the HEASARC, MAST, or NRAO.

...is also a pretty good argument for developing a FITS digital signature
standard - although the checksum will serve well during the next decade
or two while we discuss the issue :-)

This usage is a specific example of the general strategy of providing
end-to-end verification from the point of origin of a data set through
many and varied layers of data handling or processing to the final
disposition of the data set.

Note another feature of the 1's complement checksum - that it is
trivially easy to calculate differential corrections to the checksum,
if only some FITS records are changed.  For example, if a header is
updated, only the checksum for the header needs to be regenerated (and
combined with the DATASUM).  Updating the checksum for a bintable HDU
that has been edited might similarly involve only a very inexpensive
delta calculation.

More to the point, after such an edit, reviewing the delta corrected
checksum would not only verify the latest changes, but also the chain
of edits stretching all the way back to the original file.

Rob