[evlatests] All EVLA antennas

[Rob Long]

At about 1:01 PM local time, a rectifier in the -48 vdc power supply in 
the deformatter rack died.  This caused all deformatters to go down 
losing 4 IFs to each EVLA antenna.  We had exceeded our N+1 redundancy 
by one antennas worth of deformatter cards (actually one deformatter 
card), so when the rectifier died the remaining rectifiers could not 
handle the load.  We had been watching the loading on the power supply, 
but obviously missed it after ant 21 was brought online.

Dennis Mobley helped us troubleshoot the problem and installed two 
rectifiers into the power supply (replacing the broken one and restoring 
redundancy) and cycled power to all of the deformatter cards.  I 
verified that all deformatters were up and working again and notified 
operations that the antennas were back online.

I wasn't made aware of the problem until about 3:30 PM when I was 
notified by Steve Durand that all of the deformatters were dead!  I 
realize that we were in line mode, but shouldn't we have had some 
indication that all EVLA IFs were missing?  Do the T5s still get flagged 
as out of range on the EVLA antennas?  If so, will they continue to be 
flagged after Modcomps go away?  We need to take a closer look at this 
incident to ensure this doesn't happen again.

Rob