[evlatests] Set-and-Remember Gone Wild (continued)

[Dan Mertely]

Hi Rick.  I realize that with this email you are describing
a problem relating to *how* the system deals with changes in
total power, but I thought it would be worth mentioning that
recently I inventoried local "11 GHz" data links overshooting
the VLA.  The results are located at:

https://science.nrao.edu/facilities/vla/docs/observing/RFI/X-Band
(thanks, Emmanuel)

and show 2 link pairs going into and out of Grayhill (just north
of E64/AE8/ea12) and Davenport Pk (just NW of Datil).  They may
be the *cause* of the bad initial settings in X-band.
-Mert

On 5/8/2018 5:47 PM, Rick Perley wrote:
>      As readers of 'evlatests' will know, I've been carefully 
> calibrating science data taken over the past three weeks or so.  A 
> number of issues have been found, and reports distributed.
> 
>      This one deals with the system setups at X-band.  Review of the 
> data showed extremely large variations amongst antennas in required 
> calibration gain, particularly in the 10 -- 12 GHz frequency band. These 
> variations (*over two orders of magnitude in some cases*) are far, far 
> larger than can possibly be expected due to normal variations between 
> antennas, or due to variations in bandpass shape.
> 
>      The effects are also seen in the noise -- 'SPFLG' plots showed the 
> noise in the spectra, for the upper half of X-band, to be far higher 
> than the lower half, for some antennas and polarizations.
> 
>      A) For the impatient, here is the bottom line:
> 
>      Set-and-Remember is completely failing to correctly set the power 
> levels.  This is not rare failure, and it is not without significant 
> consequences.  In all cases, the failure is to leave the analog 
> attenuators at a very high level, so the power reaching the samplers is 
> far too low.  We are 'bit-starving' the system, resulting in significant 
> loss of sensitivity.
> 
>      B) For those interested, here are the details.  Note that all of 
> this applies to the 3-bit system.  (Apologies for the length).
> 
>      1) The 2 GHz-wide analog channel power is adjusted by a variable 
> attenuator, with the intention of setting it at the right level for 
> quantization.  This is done by detecting the power in the full 
> bandwidth.  This is done *only once* for any given setup.  The time 
> chosen is the first observation at a given band and correlator setup. 
> The attenuator value chosen is 'remembered' throughout the entire 
> observation for that particular band and correlator setup.  If the value 
> selected is incorrect, it is incorrect 'forever'.  (Keep this thought in 
> mind).  Folowing this, a slope adjustment is made (to keep the spectral 
> power density approximately constant over frequency), and the data sampled.
> 
>      2) The sampled data stream is divided into 'spectral windows' by 
> the correlator station boards.  Because there can be significant 
> variations in power between these windows due to the analog bandpass 
> function, a 'digital gain' is applied to each spectral window in order 
> to get its (digital) power at the optimum level for cross-correlation. 
> This correction is done anew each time the system returns to a given 
> band and correlator setup. Thus, unlike the 'set-and-remember', this 
> correction can change over the length of a run.
> 
>      It is presumed in all this that the major gain adjustment is done 
> by the analog attenuators ('set-and-remember') and the digital gain 
> correction is to adjust for the relatively minor change due to bandpass 
> shape and temporal changes in gain.
> 
>      Because the digital gain is dynamically changing, its effects must 
> be removed prior to calibration.  The digital gain levels are recorded, 
> and the data adjusted by post-processing software. ('TYAPL' in AIPS).  I 
> have confirmed that this works correctly. Note that the effect of 
> applying the digital gain correction is to 'return' the data to the 
> state it was in when sampled (that is, at the level the analog 
> attenuators set it to).
> 
>      3) Thus, in the process of calibration, the gain factors derived 
> should vary (between antennas, and between spectral windows for a given 
> antenna) only by the amount needed to adjust for the natural variations 
> in the bandpass shape.  Perhaps a factor of two (in power), at most a 
> factor of four.  (This argument presumes all antennas are equally 
> sensitive.  At X-band, this is nearly always the case, to within a 
> factor of better than 50%).
> 
>      But this is far from what is seen.
> 
>      In my recent data, taken last weekend, the expected, correct gain 
> value should be about 8.  (Units don't matter here, but this value can 
> be calculated from system parameters).  With a variation of at most a 
> factor of four in power, the variation in system gains (which are in 
> voltage units) should be a factor of two at most, and ideally should be 
> within ~50% of the magic value of 8. So what did I see:
> 
>      a) In the 10 -- 12 GHz band, eleven antennas had gain factors more 
> than a factor of two (four in power) away from the expected value.  *All 
> of them are too high, meaning the actual power to the samplers was more 
> than a factor of four too low*.   The worst case was ea12R -- a factor 
> of about 150 (!!!!) low in power.
> 
>      This is not an isolated event.  I returned to an earlier experiment 
> (3C273, taken 3 weeks ago) -- the same effect is seen. But the affected 
> antennas are all different!  (And the gain effects are even larger).
> 
>      b) So what can cause this?  Options are limited:
> 
>      i) Extreme bandpass shape?  It could be that the spectral windows I 
> chose to looks at are in some sort of weird 'null' in the overall 
> bandpass.  But this is not the case.  I generated the 'absolute' 
> bandpasses for the entire 2 GHz-wide IF.  The bandpass shape is entirely 
> normal for every antenna and polarization, but the spectral power (for 
> ea12R,noted above) is over 20 dB too low, for every spectral window. 
> Without exception, every antenna required a large gain correction had a 
> spectral power low by exactly the factor given by (value seen/8)^2.
> 
>      ii) This says the analog power to the samplers is too low. Either 
> the slope filter messed up, or the set-and-remember regimen has failed.
> 
>      iii) I recovered that data used for the 'set-and-remember' 
> procedure.  Plots of the digital power recorded during the S&R procedure 
> are attached for two antennas, ea12 and ea13, in spectral window #24 (in 
> them middle of the 10 -- 12 GHz band). The former is the one that failed 
> spectacularly, the latter is one that worked.  These plots have three 
> panels.  The bottom one is the one to look at -- it shows the power in 
> that spectral window span as 'seen' by the sampler.  The ideal level is 
> about 14 counts.  Anything within a factor of two of this is acceptable. 
> (The middle panel shows the requantizer gain correction, the upper panel 
> is the power following the correction by the requantizer).
> 
>      Look first at ea13:  This is what is expected.  The system 
> evidently spends 30 seconds adjusting the attenuators to get close to 
> the right level.  The process is often chaotic!  Note the big spike for 
> ea13R, immediately after data taking commenced.  Note also that the 
> initial power level is very, very low -- close to zero.
> 
>      Now look at ea12:  The LCP side 'sort-of' worked -- starting near 
> zero, the power stepped upwards, but never got very close to the desired 
> level.  It's a factor of two too low.  Maybe ok.  Then check the LCP 
> side.  Starting near zero, the power jumped to 14 units (the right 
> value!!!) within four seconds -- and then promptly returned to near 
> zero, and never changed.  This is the antenna which ended up two orders 
> of magnitude too low -- it never had a chance.
> 
>      The 'big spike' in power, near the beginning of the S&R regimen, is 
> very common.  Most antennas show this.
> 
>    c) So what is causing this?   The facile explanation is always the 
> same -- 'RFI'.  But that's not the case here.  I checked *all* the raw 
> spectra from the data used for the S&R regimen.  It's all clean.   If 
> 'RFI' is to blame, it's outside the 2 GHz-wide bandpass that I can see. 
> There are TV downlinks above 12 GHz, but you would expect them to affect 
> the Ku-band data more.  The Ku-band data do not show this problem at 
> anywhere near the level seen in X-band.
> 
> --------------------------------------------------------
> 
>      Bottom line:  (for real, in this case).  The system is failing us. 
> It has easily measureable consequences -- significant loss in 
> sensitivity.  This is not the first time I've pointed this out -- I have 
> a presentation, made a couple years ago, showing bad power levels being 
> set.   The problem is still with us, and deserves attention.
> 
> 
> _______________________________________________
> evlatests mailing list
> evlatests at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/evlatests
>