[evlatests] evla confusion

[Ken Sowinski]

> Date: Wed, 15 Feb 2006 09:16:03 -0700 (MST)
> From: Barry Clark <bclark at aoc.nrao.edu>
> Subject: Re: [evlatests] Feb 14 tests and confusion

Some comments on what Barry wrote this morning based on
a more careful look at the data from yesterday afternoon.

> The glitching has, I think, reached the state that it disqualifies
> use of the antennas for science - one never knows whether the antennas
> are performing or not.

I agree.  And if we move the antennas to their A array 
positions we will then be in the difficult position of 
trying to find moved antennas that might not be working.

> 1.  The earliest reference I can find for something that may be identified
> with this is Rick's message of December 29.

I am not convinced that Rick was reporting then what we
see now.  From tedious reading of reports from the last two
months the earliest example of behavior consistent with what
was reported yesterday was Feb 7.

We need to think about what may have been changed in the first
week of February.  A partial list:
1.  The executor used by the operators was updated on Feb 2.
2.  When it was discovered about that time that the L352s were
being fed the wrong reference there was some rearrangement of
things in the centrl LO rack.

> 2.  The effect seems to know about 10 second intervals - it seems to occur
> on the IAT 10 seconds when viewed with short integration times.  Rick's
> first suggestion, that it preferred the 10s ticks with a scan change, does
> not seem compelling to me.
> 
> This is very important.  There are only four things that know about IAT
> 10s in the normal course of affairs:  the L302, the ACU, the L304, and 
> the central LO rack.
> 
> 3.  The dropout is sometimes complete, but sometimes only a reduction by
> a large factor.  When fringes are present, the drop is accompanied by a
> jump in phase (often of order 120 degrees), but the phase remains stable
> while the glitched state persists.  The first of these tends to exonerate 
> the L302 walsh function switching, the second tends to exonerate the L302 
> DDS.  It is still possible that the L302 is doing something to cause this, 
> but the simplest failure modes in that module will not.
> 
> 4.  When a dropout occurs, it tends to occur for all three antennas, but 
> does not always.  Sometimes, but not always, when the drop affects two
> EVLA antennas, normal fringes are seen between them.  This exonerates the ACU.
> 
> 
> While not definitive, my suspicion falls on the central LO rack.

The data taken yesterday afternoon at X band has now been looked 
at more carefully.  Plots can be found at:
http://www.aoc.nrao.edu/~vdhawan/EVLA-FEB14.PS

1.  When it is happening each EVLA antennas a differenct phase and 
amplitude jump relative to VLA antennas and the three antennas.
The magnitude of the phase and amplitude changes are correlated.

2.  Most of the time it happened to all three antennas and both IF
pairs, except 13 BD seems much better beahved than any of the other
IF pairs.  The two IFs of an IF pair always seem to do the same thing.

3.  The problem seemed to be slowly going away and turning off
phase switching for antenna 16 likely did not cause any changes.
Phase switching was turned off by disabling phase switching for
all IFs at the VLA samplers and disabling phase switching in the
BD L302, but not the AC L302.  There is certainly a clue in that
there was still a residual amplitude signal in AC with glitches,
and the phase jumps continue with the same magnitude but additionally
modulated by a jump of 180 degrees in alternate 1-2/3 second 
integrations.

4.  It may only be that I have squinted at the plots too much but 
I could be convinced that Modcomp skips change the behavior.  This
cannot happen of course.  


Certainly the L302s are implicated.  Whether they are misbehaving
on their own or just being mislead is hard to distinguish.  My vote 
is that sometimes they choose the wrong heartbeat when commanding 
the DDSs.  It is not at all obvious to me how you can get them to 
do this.