[evlatests] Last night's tests

[Ken Sowinski]

On Thu, 31 May 2007, Rick Perley wrote:

>    Tests of Modcomp-free, and Modcomp-on modes were attempted last
> night, starting just after midnight.
>
>    No useable EVLA data were obtained in any mode, at any band, either
> under Modcomp control, or not.  Most of the VLA data also appear to be
> of very questionable quality.
>
>    The evening operator reported power glitches had occurred -- I
> understand that at least some of the EVLA DTS systems were not operating.
>
>    We are attempting a quick check of system performance now.

The quick check showed that there were still problems though the
system was doing close to the right thing.  There were two primary
culprits and several secondary, or possibly induced, problems.

The correlator was in a broken state on Wednesday afternoon.  The
evidence was very difficult to see in continuum but easily seen
in line data, especially single IF modes.  amos had broken, perhaps
several days ago which prevented the vis-pipe server from booting
but had no other serious effect other than engendering confusion.
When amos ws repaired it no longer was broadcasting time notices
for the mibs.  This may, also, have been unimportant but added
to the confusion.  There were several 'serious' power glitches
Wednesday evening which may have affected antennas or control
building equipment.

Thursday morning it was discovered that many DTS modules indicated
sampler power was off.  It was not clear that the samplers really
were off; there were cases where the data seemed reasonable despite
that report.  If they were off it is not clear whether is was the
power glitches that caused it or an attempt to repair perceived
damage from the power glitches.

The correlator problem was finally noticed Thursday morning and
repaired by lunch time.  With that not confusing things it was
clear that the EVLA antennas were not behaving right while all
the VLA antennas were.  Amplitudes were lower than expected and
did not close.  After many false leads and confusion it was found
that EVLA antennas worked fine when phase switching was disabled
suggesting a timing error between EVLA and VLA (or vice verse).
Again it is not clear when the timing difference arose.  It may
have been a result of the power disturbances, or the loss of amos,
or something else.  Rob spent a lot of time looking into signals
and resetting hardware.  Eventually it worked but it is not clear
why.  My memory is that suddenly things looked more reasonable
but the event is not associated with any particular action on
Rob's part.  It might have been the fact that an EVLA script was
restarted after Rob had done all his magic.

The only lesson I see here is that the more curious things that
happen over a short time the more difficult it is to know that
the system is broken, let alone what caused all the problems.

All in all a long and not very pleasant day.