[evlatests] Set-and-Remember Gone Wild (continued)

[Ken Sowinski]

On Tue, 8 May 2018, Rick Perley wrote:

>     As readers of 'evlatests' will know, I've been carefully calibrating 
> science data taken over the past three weeks or so.  A number of issues have 
> been found, and reports distributed.
>
>     This one deals with the system setups at X-band.  Review of the data 
> showed extremely large variations amongst antennas in required calibration 
> gain, particularly in the 10 -- 12 GHz frequency band.  These variations 
> (*over two orders of magnitude in some cases*) are far, far larger than can 
> possibly be expected due to normal variations between antennas, or due to 
> variations in bandpass shape.
>
>     The effects are also seen in the noise -- 'SPFLG' plots showed the noise 
> in the spectra, for the upper half of X-band, to be far higher than the lower 
> half, for some antennas and polarizations.
>
>     A) For the impatient, here is the bottom line:
>
>     Set-and-Remember is completely failing to correctly set the power 
> levels.  This is not rare failure, and it is not without significant 
> consequences.  In all cases, the failure is to leave the analog attenuators 
> at a very high level, so the power reaching the samplers is far too low.  We 
> are 'bit-starving' the system, resulting in significant loss of sensitivity.

Rick wrote at great length (shortened here) about whether set and
remember works correctly, particularly for the upper half of X band.
An exmaination of behavior shows that some of the X band receivers
are severly underpowered, expecially at the upper half of the band,
the attenuator setting process sets all the attenuators wide open
and we are still underpowering the sampler.  Until we can find
attenuators that can suuply gain, nothing more can be done.

In trying to understand this we looked carefully at how the servo
loops that control the T304 knobs work and interact.  Since this
is not written down anyehere I wrote a page describing how set and
remember is implemented along with possible changes to improve
efficienecy.  Given that this all works as expected and does the
best that is possible, even in adverse conditions, I suggest that
none of these proposed changes are necessary.

Here is the long version, for those who care to read more.


A summary of how levels are set for 3-bit observations must begin with
a description of how the T304 and the correlator-based servo loops are
implemented, and how the executor remembers the necessary attenuator
settings.  8-bit observations are identical but do not activate the
correlator-based servo loop.

Each attenuator in the T304 is set based on the power measured by a
detector which measures the signal after the attenuator.  Samples are
measured at 500 Hz and integrated for one second.  When enabled the
MIB looks at the most recent integration, compares it with the desired
set point and adjusts the attenuator by the difference between the
desired power and the measured power.  If at the next integration the
measured power differs from the desired power by more than 0.5 dB the
attenuator is adjusted in one dB steps until the desired power is
achieved.  When the desired power has been reached the servo loop
shuts down.  The output attenuator servo loops are not permitted to
start until the input loop has finished.  I assume that the MIB is
clever enough to ensure that a one second integration is done with an
unchanging attenuator setting.  Meanwhile, the MIB is continually
reporting the current attenuator settings on the o-stream every three
seconds.  The executor records these in an Attenuator class provided
that the value arrives at least 13 seconds after the beginning of a
scan, and no more than 1.7 seconds before the start of the following
scan.  A consequence of this is that a setup scan shorter than 18
seconds will not consistently remember attenuator settings.

Independent of the T304 servo loops another control loop using data
from the Widar Wide Band Correlator is trying to set the attenuators
and equalizers in the 3-bit paths of the T304.  Every second the
correlator measures total power and slope across the baseband for each
baseband which is in 3-bit mode.  The executor queries the correlator
for the most recent values every four seconds and derives adjustments
to the output attenuators and equalizers.  The changes are limited to
one step at a time.  As it does this it remembers what it is doing in
the Attenuator class.  This servo loop is enabled at the begining of
the setup scan and disabled just before it ends, but begins only when
all T304 attenuators have been remembered.  When data is available
from the correlator depends on whether this scan requires a new
correlator configuration.

Under some circumstances both these loops may be active at the same
time with unpredictable consequences.  Examples have been seen of the
two loops apparantly cooperating as well as examples where the
attenuator changes by many dB from second to second.  In all cases
examined it always settles down to something sensible.


Let t_scan be the start time of a setup scan and t_next the start of
the following scan.  Assume that execute() was called well before the
scheduled start time and that a correlator reconfiguration is not
required.  By default, all T304 attenuator modes are set to
SETANDREMEMBER; the RF attenuator servos to -20 dBm, the output
attenuators also to -20 dBm, which was found to drive the samplers about
right in most cases.

At time:
t_scan    Commands are sent to most antenna components,
 	  the correlator servo loop is enabled, but will not operate
 	  before t_scan+13.
t_scan+FRM_delay
           Begin the RF attenautor T304 servo loop.  Maximum possible
 	  FRM_delay is 15 seconds.
t_scan+FRM_delay+3
           Begin the output attenuators T304 servo loops, but will
 	  atart only when the RF attenuaotr setting has completed.
t_next-10 Disable correlator servo loop


This sequence can be altered with the hope of being more efficient.
1.  Delay the enabling of the correlator-based servo loop so that we are
sure that the T304 has finished its work.
2.  Change the control mode for the output attenuators to be SETATTEN
and supply a value which may have to be band dependent, so that all the
work of setting the output attenuators is done by the correlator-based
servo loop.

These two suggestions can be implemented independently or in combination.
both can be implemented in the 3bitSetup.py include file and require no
changes in the executor or scripts.