[evlatests] correlator lessons

[Ken Sowinski]

Today was a rough day.  Some of the lessons learned:

There is a bug in the station board configurator that leaves
files open and eventually reaches the system per process limit
killing the process.  For our current usage patterns these takes
two or three weeks to occur.  Bruce understands the cause and
will have it fixed soon.  This is the reason antennas 1 through 5
started to disappear yesterday.

The network switch was repaired at noon today and there was much
fallout; probably as much caused by our confusion as the process
itself.  Baseline boards had to be booted and came up, either
initially or after our meddling, with data flow on the ethernet
chip disabled.  Tracking this down was painful.

Various station boards were rebooted; some because of the problems
above, others on general prinicples.  A few deformatters lost their
minds after this.  The one remaining unhappy deformatter is the
one for ea5-D.  Station boards in rack 6 were rebooted because of
firmware changes and most of the antennas connected there suffered
large delay changes.  the largest were about 96 ns.