[evlatests] A most spectacular failure of 'set-and-remember'?

[Rick Perley]

    I'm calibrating 20 S-band databases, taken from November 2013 
through January 2014.    Five have been completed without any troubles 
(other than the usual satellite RFI).  But one is 'special':

    For this database, the gains of all antennas jumped up, by a factor 
of ~ 2 to 10, at times roughly in the middle of the 90-minute run.  But ...

    1) Only the A and C IFs jumped. 
    2) Each antenna jumped gain at a different time. 
    3) The new gain was in fact the correct one (judging from the 
calibrated gain values and PSum values).  Hence, *the initial power 
setup was too low by factors of a few*.  Somehow, this error was 
corrected in the middle of the run. 

    This SB was under program 13B-316.  The data were taken on 06 Dec 
2013. 

    The time patterns of the jumps are curious:

    1) Antennas 18, 19, 20, 22, 23 and 28 all jumped between 12:47 and 
12:51 (IAT).  All jumps were 'clean' -- no more than one intermediate 
value is seen.
    2) All other antennas jumped between 13:20 and 13:45.  Some of these 
jumps were not clean -- the system oscillated between the two states, or 
(in one case) hopped around three states, before stabilizing. 

    The initial scan for this run was many minutes long -- more than 
enough time for the system to find the 'right' level, and keep it.  But 
it didn't, and it didn't ...

    Any ideas?