[evla-sw-discuss] power outage recovery log

[Ken Sowinski]

Recovery from power outage, October 23, 2010.

1.  I expected CM to come up initiallized with a list of antennas,
but it did not.  I restarted it, but still had to manually supply
the list of antennas.

2.  Lustre, according to KSCott, was mis-configured.  He fixed that
and rebooted cbe-master.  CBE, and d10, were then happy.

3.  Most baseline boards were not configured/programmed.  I suspect
the automatic startup gave up in disgust after a while because the
station boards were unhappy, because the antennas were still unhappy.
I reprogrammed the ones I inteded to use and found lots of fringes.
Is there an easy way, short of rebooting, to do that to all boards?

4.  Once we were getting data through the system it was apparant that
telcal was unhappy.  mchammer was not able to see lustre data.  I
suspect that the archive cannot either and that the problem is the
same as that with cbe-master.  KScott is looking at it now.  mchammer
was rebooted.

5.  ea05C and ea25A were not getting delay packets.  I had to restart
the delay distributor in mccc to get them to work.

6.  ea09B had two copies of many of its processes running and behaved
very unstably.  I rebooted it.

7.  ea13C and ea10B required deformatter realignment even though Kerry
thought they were OK

8.  The API is "dead".  At least it is not updating on the operators'
screen.

9.  I did not have permission to write a delay correction file into
the telcal output directory.