[evlatests] Set-and-Remember Gone Wild (continued)

Tue May 8 19:47:14 EDT 2018

     As readers of 'evlatests' will know, I've been carefully 
calibrating science data taken over the past three weeks or so.  A 
number of issues have been found, and reports distributed.

     This one deals with the system setups at X-band.  Review of the 
data showed extremely large variations amongst antennas in required 
calibration gain, particularly in the 10 -- 12 GHz frequency band.  
These variations (*over two orders of magnitude in some cases*) are far, 
far larger than can possibly be expected due to normal variations 
between antennas, or due to variations in bandpass shape.

     The effects are also seen in the noise -- 'SPFLG' plots showed the 
noise in the spectra, for the upper half of X-band, to be far higher 
than the lower half, for some antennas and polarizations.

     A) For the impatient, here is the bottom line:

     Set-and-Remember is completely failing to correctly set the power 
levels.  This is not rare failure, and it is not without significant 
consequences.  In all cases, the failure is to leave the analog 
attenuators at a very high level, so the power reaching the samplers is 
far too low.  We are 'bit-starving' the system, resulting in significant 
loss of sensitivity.

     B) For those interested, here are the details.  Note that all of 
this applies to the 3-bit system.  (Apologies for the length).

     1) The 2 GHz-wide analog channel power is adjusted by a variable 
attenuator, with the intention of setting it at the right level for 
quantization.  This is done by detecting the power in the full 
bandwidth.  This is done *only once* for any given setup.  The time 
chosen is the first observation at a given band and correlator setup.  
The attenuator value chosen is 'remembered' throughout the entire 
observation for that particular band and correlator setup.  If the value 
selected is incorrect, it is incorrect 'forever'.  (Keep this thought in 
mind).  Folowing this, a slope adjustment is made (to keep the spectral 
power density approximately constant over frequency), and the data sampled.

     2) The sampled data stream is divided into 'spectral windows' by 
the correlator station boards.  Because there can be significant 
variations in power between these windows due to the analog bandpass 
function, a 'digital gain' is applied to each spectral window in order 
to get its (digital) power at the optimum level for cross-correlation.  
This correction is done anew each time the system returns to a given 
band and correlator setup. Thus, unlike the 'set-and-remember', this 
correction can change over the length of a run.

     It is presumed in all this that the major gain adjustment is done 
by the analog attenuators ('set-and-remember') and the digital gain 
correction is to adjust for the relatively minor change due to bandpass 
shape and temporal changes in gain.

     Because the digital gain is dynamically changing, its effects must 
be removed prior to calibration.  The digital gain levels are recorded, 
and the data adjusted by post-processing software. ('TYAPL' in AIPS).  I 
have confirmed that this works correctly. Note that the effect of 
applying the digital gain correction is to 'return' the data to the 
state it was in when sampled (that is, at the level the analog 
attenuators set it to).

     3) Thus, in the process of calibration, the gain factors derived 
should vary (between antennas, and between spectral windows for a given 
antenna) only by the amount needed to adjust for the natural variations 
in the bandpass shape.  Perhaps a factor of two (in power), at most a 
factor of four.  (This argument presumes all antennas are equally 
sensitive.  At X-band, this is nearly always the case, to within a 
factor of better than 50%).

     But this is far from what is seen.

     In my recent data, taken last weekend, the expected, correct gain 
value should be about 8.  (Units don't matter here, but this value can 
be calculated from system parameters).  With a variation of at most a 
factor of four in power, the variation in system gains (which are in 
voltage units) should be a factor of two at most, and ideally should be 
within ~50% of the magic value of 8. So what did I see:

     a) In the 10 -- 12 GHz band, eleven antennas had gain factors more 
than a factor of two (four in power) away from the expected value.  *All 
of them are too high, meaning the actual power to the samplers was more 
than a factor of four too low*.   The worst case was ea12R -- a factor 
of about 150 (!!!!) low in power.

     This is not an isolated event.  I returned to an earlier experiment 
(3C273, taken 3 weeks ago) -- the same effect is seen. But the affected 
antennas are all different!  (And the gain effects are even larger).

     b) So what can cause this?  Options are limited:

     i) Extreme bandpass shape?  It could be that the spectral windows I 
chose to looks at are in some sort of weird 'null' in the overall 
bandpass.  But this is not the case.  I generated the 'absolute' 
bandpasses for the entire 2 GHz-wide IF.  The bandpass shape is entirely 
normal for every antenna and polarization, but the spectral power (for 
ea12R,noted above) is over 20 dB too low, for every spectral window.  
Without exception, every antenna required a large gain correction had a 
spectral power low by exactly the factor given by (value seen/8)^2.

     ii) This says the analog power to the samplers is too low. Either 
the slope filter messed up, or the set-and-remember regimen has failed.

     iii) I recovered that data used for the 'set-and-remember' 
procedure.  Plots of the digital power recorded during the S&R procedure 
are attached for two antennas, ea12 and ea13, in spectral window #24 (in 
them middle of the 10 -- 12 GHz band). The former is the one that failed 
spectacularly, the latter is one that worked.  These plots have three 
panels.  The bottom one is the one to look at -- it shows the power in 
that spectral window span as 'seen' by the sampler.  The ideal level is 
about 14 counts.  Anything within a factor of two of this is acceptable. 
(The middle panel shows the requantizer gain correction, the upper panel 
is the power following the correction by the requantizer).

     Look first at ea13:  This is what is expected.  The system 
evidently spends 30 seconds adjusting the attenuators to get close to 
the right level.  The process is often chaotic!  Note the big spike for 
ea13R, immediately after data taking commenced.  Note also that the 
initial power level is very, very low -- close to zero.

     Now look at ea12:  The LCP side 'sort-of' worked -- starting near 
zero, the power stepped upwards, but never got very close to the desired 
level.  It's a factor of two too low.  Maybe ok.  Then check the LCP 
side.  Starting near zero, the power jumped to 14 units (the right 
value!!!) within four seconds -- and then promptly returned to near 
zero, and never changed.  This is the antenna which ended up two orders 
of magnitude too low -- it never had a chance.

     The 'big spike' in power, near the beginning of the S&R regimen, is 
very common.  Most antennas show this.

   c) So what is causing this?   The facile explanation is always the 
same -- 'RFI'.  But that's not the case here.  I checked *all* the raw 
spectra from the data used for the S&R regimen.  It's all clean.   If 
'RFI' is to blame, it's outside the 2 GHz-wide bandpass that I can see.  
There are TV downlinks above 12 GHz, but you would expect them to affect 
the Ku-band data more.  The Ku-band data do not show this problem at 
anywhere near the level seen in X-band.

--------------------------------------------------------

     Bottom line:  (for real, in this case).  The system is failing us.  
It has easily measureable consequences -- significant loss in 
sensitivity.  This is not the first time I've pointed this out -- I have 
a presentation, made a couple years ago, showing bad power levels being 
set.   The problem is still with us, and deserves attention.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: EA13R
Type: application/postscript
Size: 30735 bytes
Desc: not available
URL: <http://listmgr.nrao.edu/pipermail/evlatests/attachments/20180508/55172894/attachment-0004.ai>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: EA13L
Type: application/postscript
Size: 31425 bytes
Desc: not available
URL: <http://listmgr.nrao.edu/pipermail/evlatests/attachments/20180508/55172894/attachment-0005.ai>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: EA12R
Type: application/postscript
Size: 30868 bytes
Desc: not available
URL: <http://listmgr.nrao.edu/pipermail/evlatests/attachments/20180508/55172894/attachment-0006.ai>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: EA12L
Type: application/postscript
Size: 31089 bytes
Desc: not available
URL: <http://listmgr.nrao.edu/pipermail/evlatests/attachments/20180508/55172894/attachment-0007.ai>