[evlatests] P-Band status -- Lots of Issues

[Rick Perley]

    A 'fast-dump' continuum run was made last night at P-band, to check 
system behavior.  There are many issues to be addressed at this band...
    Two scans were done, a long 'dummy' initial scan, followed by a 
'real' scan, with the same LO and BW setup.   Each scan was 5 minutes in 
duration. 

    I'll try sort all the results by subject and my assessment of 
severity. 

1) Antennas not fringing:

    *  Antenna 4 gives no fringes in any IF, and is flagged correctly.
    *  Antennas 13 and 16 give no fringes (as expected), and are flagged 
correctly.
    *  Antenna 26, in IF#2 (B and D) gave no fringes, and were *not* 
flagged correctly.  (Hence, I could not see the '26D shredding' reported 
earlier by Ken). 

2) Antennas late on source weirdness:

    Antennas 2, 3, 9, 10, and 12 took a very long time to provide 
fringes -- up to four minutes later than the rest of the array.  All IFs 
behaved similarly.  Flagging did *not* catch this problem -- the 
antennas apparently got on source along with the others, but for some 
other reason, took a few minutes longer to get sorted out.  Some of the 
bad data were flagged, but most were not.  There is no spatial 
correlation for these antennas' locations -- other than they are all on 
the E and N arms. 

3) Weak Signals (poor SNR)

    Antenna 11 is weak by about a factor of four in amplitude, on all 
four IFs. 

4) Other flagging issues

    Immediately (< 1 second) after the start of the 2nd 'good' scan, 
antenna 8 was flagged out by the system for the rest of the 5 minute 
scan.  But the data were good, with no change in gain or phase. 

5) System Correlator Reorganization Trouble

    One of the 10 to 20 second data gaps occurred in the run:  Data from 
all antennas suddenly go incoherent, until the next 10-second tick, 
followed by a 10-second gap in the data stream, followed by good data.  
None of the incoherent data were flagged.  Ken explains this as a 
'system controller reset' (or some such).  This is easy to catch on a 
calibrator, but impossible on a generic weak source.  The frequency of 
occurence is worrysome to me -- I think perhaps once per hour.  Probably 
not very important, but bears watching. 

6)  Stability Troubles. 

    *  Antenna 14 is 'all over the map'.  More specifically, both IF 
pairs showed huge amplitude and/or phase instability for extended 
periods of time.  The two polarization in each IF pair behaved 
identically.  14A and 14C were much the worse, with long (few minutes) 
periods of complete phase incoherence (and the expected loss in 
amplitude at the same time).  The behavior in 14B and 14D was rather 
different, where large phase ramps (rather than apparent random phase) 
took place at the same times. 

    *  Antenna 21 has two different amplitude states on all four IFs -- 
but unlike other multiple level issues, the four IFs all show different 
characteristics.  The SNR (as judged by stability of the amplitudes in 
either state) is very very low.  This problem has been around for a long 
time, and Ken and I used to think it was symptomatic of absent Tcals.  
But that should give random amplitudes, with a roughly 10-second 
stability period (I think) -- what I'm seeing here is completely 
different.  Something else is amiss here.  It is important to note that 
the phases are *stable*. 

    *  Antenna 18A and 18C showed bistable phase, about 75 degrees 
apart, during much of the first scan.  The second scan was fine.  18B 
and 18D were clean. 

    * Antenna 28, (only!), showed a unique phase feature:  *Exactly* 10 
seconds before the end of the first scan, the phases on all four IFs 
stepped uniformly by 90 degrees, in the following manner -- There were 
eight steps, each exactly 1.25 seconds long,  each step was about 12 
degrees.  All IFs showed identical behavior, and the amplitudes were not 
affected.  This behavior was not shown at the end of the second scan, 
following which the correlator changed mode (to spectral line).  (The 
scan following the one with the 'steps' was an identical scan with same 
frequency, BW, and mode). 

    7) Global changes between the two identical scans.

    As noted above, the first two scans had identical setups:

    *  The EVLA and VLA suffered a global phase change upon the change 
of scan.  The change was about 100 degrees in IFs A and C, and about 45 
degrees for IFs B and D.  Sounds 'flukish' -- but I can't prove which 
array 'jumped'. 

    * A number of antenna-IFs changed their amplitude gain upon the 
change of scan:
          - 17D rose by about 20%
          - 19D rose by about a factor of 2.0
          - 23A declined by a factor of about 5 -- however, prior to the 
change, the amplitudes were bouncing back and forth between the two 
levels.  The latter (lower) level is clearly the right one, so some 
large and variable signal was present during the first scan only, which 
was 'sorted out' by the change of scan. 
          - 26C rose by about 20% -- very similar behavior to 17D. 

       Some of these troubles are related to bad Tsys measurements:
       17D shows a drop in Tsys at this point by 25% -- since the 
amplitudes were not correctly adjusted, the Tsys values recorded are not 
correct.
       19 has no Tsys values at all, so no Tsys adjustments can be 
made.  Evidently, the actually correlator coefficient changed by a 
factor of two of 19D (only).  It seems unlikely this is due to a change 
in system temperature ...
       23A has wild and crazy Tsys values during the period of amplitude 
instability -- the unstable amplitudes are clearly due to erroneous Tsys 
corrections being applied.
       26C shows no change in Tsys.  (But 26A changes by a factor of two 
in Tsys between scans, with no change in amplitude!  I can't offer a 
rational, and believable, explanation for all this). 

    That's enough for one day at this band.

    4-band report follows.