[evlatests] T1 VLA/AOC Failure Saturday

[Pat Van Buskirk]

There was a T1 line failure between the AOC and VLA Saturday at 
approximately 5:40 pm due a washout on Highway 60. It was restored 
Sunday around noon. Here's what survived and what didn't relative to EVLA.

On the Array Operator Screen, we lost the time and weather indicators; 
time recovered, but weather is still not displaying. We were also unable 
to submit an EVLA pointing script and the device browser was not 
working. The device browser may be a red herring due to simultaneous 
problems with antenna 24 which was the display at the time. Currently, 
the device browser cannot connect to dataset 0.

Dynamic scheduling would not run, giving java errors. The schedule queue 
was still active; however, the accept process catalogued an empty file, 
with a single EOF card.

The Mainsaver database is at the AOC, so we were unable to write 
maintenance forms during the outage.

The monitor and parameters database, along with the Oracle software, are 
totally isolated to the site and stayed active. Monitor data were not 
affected and appear to be intact. We were able to control the EVLA antennas.

VLA mail was down, so the operators could not receive alerts fired from 
the AOC.

The operators on duty did an excellent job of handling the outage and 
associated problems. We should all thank them for their conscientious work.

Cheers,
Pat





-- 
Discontent is the first necessity of progress.

Thomas A. Edison (1847-1931, American inventor, founder of GE)