[fitsbits] Automated FITS correction tools?

[Rob Seaman]

Hi Bill,

> First, I would hope that fitsverify (http://fits.gsfc.nasa.gov/fits_verify.html) is able to identify all the problems in the non-conformant FITS files that you have encountered.

So far as I know.  Note that problems with conformance to the standard are different than problems generated in some particular workflow.  Not only might strict conformance be a much higher bar than would otherwise be required for clean sailing through the workflow (eg, "save-the-bits" requires the minimum level of compliance with the fundamental structural keywords possible since it seeks to handle any FITS HDU object whatsoever) - but conformance gets mixed up with other requirements the workflow places on FITS keyword values, etc.  That is, workflow requirements are simultaneously more and less strict than formal conformance requirements.

> I'd like to know about any FITS format errors that fitsverify doesn't detect.

If fitsverify is called out as part of this (proposed) project, you can be confident of hearing from me :-)

> It would be interesting to see a comprehensive list of all the conformance errors or warnings that you've found.

We're responding to errors thrown by a downstream (java) component.  For instance, it cares about improperly terminated string-valued keywords, but doesn't seem to care about real valued keywords with exponents delimited in lower-case.

> It should be possible to write a tool to automatically correct some of the problems, but some types of errors would probably require expert knowledge about the contents of the file to correct.

My initial thought is to copy the fitsverify source and start adding code to fix the errors that generate downstream issues, one-by-one.  Maybe call it "fitsfixer".  It's likely the output options would want to be tweaked to suit such a role as well, eg, not quite so chatty.  For instance, an extension with a missing PCOUNT/GCOUNT would be pretty easy to fix.  Other issues require making an assumption about what the creator was attempting to say in the first place.

There are also pixel level errors, the most obvious being incorrect filling of the final record.

Rob