[evlatests] Set-and-Remember Gone Wild (continued)

[Rick Perley]

     ... except that I looked at both the X-power and auto-correlation 
spectra (including the autocorrelation spectrum from ea12), and saw *no* 
strong RFI with the 4 GHz total bandwidth.  I looked at the actual data 
used in the setup.

     If an RFI spike is responsible, it lies outside the 4 GHz covered 
by the digital spectra.

     Rick


On 05/09/2018 09:01 AM, Dan Mertely wrote:
> Hi Rick. I realize that with this email you are describing
> a problem relating to *how* the system deals with changes in
> total power, but I thought it would be worth mentioning that
> recently I inventoried local "11 GHz" data links overshooting
> the VLA.  The results are located at:
>
> https://science.nrao.edu/facilities/vla/docs/observing/RFI/X-Band
> (thanks, Emmanuel)
>
> and show 2 link pairs going into and out of Grayhill (just north
> of E64/AE8/ea12) and Davenport Pk (just NW of Datil).  They may
> be the *cause* of the bad initial settings in X-band.
> -Mert
>
> On 5/8/2018 5:47 PM, Rick Perley wrote:
>>      As readers of 'evlatests' will know, I've been carefully 
>> calibrating science data taken over the past three weeks or so.  A 
>> number of issues have been found, and reports distributed.
>>
>>      This one deals with the system setups at X-band.  Review of the 
>> data showed extremely large variations amongst antennas in required 
>> calibration gain, particularly in the 10 -- 12 GHz frequency band. 
>> These variations (*over two orders of magnitude in some cases*) are 
>> far, far larger than can possibly be expected due to normal 
>> variations between antennas, or due to variations in bandpass shape.
>>
>>      The effects are also seen in the noise -- 'SPFLG' plots showed 
>> the noise in the spectra, for the upper half of X-band, to be far 
>> higher than the lower half, for some antennas and polarizations.
>>
>>      A) For the impatient, here is the bottom line:
>>
>>      Set-and-Remember is completely failing to correctly set the 
>> power levels.  This is not rare failure, and it is not without 
>> significant consequences.  In all cases, the failure is to leave the 
>> analog attenuators at a very high level, so the power reaching the 
>> samplers is far too low.  We are 'bit-starving' the system, resulting 
>> in significant loss of sensitivity.
>>
>>      B) For those interested, here are the details.  Note that all of 
>> this applies to the 3-bit system.  (Apologies for the length).
>>
>>      1) The 2 GHz-wide analog channel power is adjusted by a variable 
>> attenuator, with the intention of setting it at the right level for 
>> quantization.  This is done by detecting the power in the full 
>> bandwidth.  This is done *only once* for any given setup.  The time 
>> chosen is the first observation at a given band and correlator setup. 
>> The attenuator value chosen is 'remembered' throughout the entire 
>> observation for that particular band and correlator setup.  If the 
>> value selected is incorrect, it is incorrect 'forever'.  (Keep this 
>> thought in mind).  Folowing this, a slope adjustment is made (to keep 
>> the spectral power density approximately constant over frequency), 
>> and the data sampled.
>>
>>      2) The sampled data stream is divided into 'spectral windows' by 
>> the correlator station boards.  Because there can be significant 
>> variations in power between these windows due to the analog bandpass 
>> function, a 'digital gain' is applied to each spectral window in 
>> order to get its (digital) power at the optimum level for 
>> cross-correlation. This correction is done anew each time the system 
>> returns to a given band and correlator setup. Thus, unlike the 
>> 'set-and-remember', this correction can change over the length of a run.
>>
>>      It is presumed in all this that the major gain adjustment is 
>> done by the analog attenuators ('set-and-remember') and the digital 
>> gain correction is to adjust for the relatively minor change due to 
>> bandpass shape and temporal changes in gain.
>>
>>      Because the digital gain is dynamically changing, its effects 
>> must be removed prior to calibration.  The digital gain levels are 
>> recorded, and the data adjusted by post-processing software. ('TYAPL' 
>> in AIPS).  I have confirmed that this works correctly. Note that the 
>> effect of applying the digital gain correction is to 'return' the 
>> data to the state it was in when sampled (that is, at the level the 
>> analog attenuators set it to).
>>
>>      3) Thus, in the process of calibration, the gain factors derived 
>> should vary (between antennas, and between spectral windows for a 
>> given antenna) only by the amount needed to adjust for the natural 
>> variations in the bandpass shape.  Perhaps a factor of two (in 
>> power), at most a factor of four.  (This argument presumes all 
>> antennas are equally sensitive.  At X-band, this is nearly always the 
>> case, to within a factor of better than 50%).
>>
>>      But this is far from what is seen.
>>
>>      In my recent data, taken last weekend, the expected, correct 
>> gain value should be about 8.  (Units don't matter here, but this 
>> value can be calculated from system parameters).  With a variation of 
>> at most a factor of four in power, the variation in system gains 
>> (which are in voltage units) should be a factor of two at most, and 
>> ideally should be within ~50% of the magic value of 8. So what did I 
>> see:
>>
>>      a) In the 10 -- 12 GHz band, eleven antennas had gain factors 
>> more than a factor of two (four in power) away from the expected 
>> value.  *All of them are too high, meaning the actual power to the 
>> samplers was more than a factor of four too low*. The worst case was 
>> ea12R -- a factor of about 150 (!!!!) low in power.
>>
>>      This is not an isolated event.  I returned to an earlier 
>> experiment (3C273, taken 3 weeks ago) -- the same effect is seen. But 
>> the affected antennas are all different!  (And the gain effects are 
>> even larger).
>>
>>      b) So what can cause this?  Options are limited:
>>
>>      i) Extreme bandpass shape?  It could be that the spectral 
>> windows I chose to looks at are in some sort of weird 'null' in the 
>> overall bandpass.  But this is not the case.  I generated the 
>> 'absolute' bandpasses for the entire 2 GHz-wide IF.  The bandpass 
>> shape is entirely normal for every antenna and polarization, but the 
>> spectral power (for ea12R,noted above) is over 20 dB too low, for 
>> every spectral window. Without exception, every antenna required a 
>> large gain correction had a spectral power low by exactly the factor 
>> given by (value seen/8)^2.
>>
>>      ii) This says the analog power to the samplers is too low. 
>> Either the slope filter messed up, or the set-and-remember regimen 
>> has failed.
>>
>>      iii) I recovered that data used for the 'set-and-remember' 
>> procedure.  Plots of the digital power recorded during the S&R 
>> procedure are attached for two antennas, ea12 and ea13, in spectral 
>> window #24 (in them middle of the 10 -- 12 GHz band). The former is 
>> the one that failed spectacularly, the latter is one that worked.  
>> These plots have three panels.  The bottom one is the one to look at 
>> -- it shows the power in that spectral window span as 'seen' by the 
>> sampler.  The ideal level is about 14 counts.  Anything within a 
>> factor of two of this is acceptable. (The middle panel shows the 
>> requantizer gain correction, the upper panel is the power following 
>> the correction by the requantizer).
>>
>>      Look first at ea13:  This is what is expected.  The system 
>> evidently spends 30 seconds adjusting the attenuators to get close to 
>> the right level.  The process is often chaotic!  Note the big spike 
>> for ea13R, immediately after data taking commenced.  Note also that 
>> the initial power level is very, very low -- close to zero.
>>
>>      Now look at ea12:  The LCP side 'sort-of' worked -- starting 
>> near zero, the power stepped upwards, but never got very close to the 
>> desired level.  It's a factor of two too low. Maybe ok.  Then check 
>> the LCP side.  Starting near zero, the power jumped to 14 units (the 
>> right value!!!) within four seconds -- and then promptly returned to 
>> near zero, and never changed.  This is the antenna which ended up two 
>> orders of magnitude too low -- it never had a chance.
>>
>>      The 'big spike' in power, near the beginning of the S&R regimen, 
>> is very common.  Most antennas show this.
>>
>>    c) So what is causing this?   The facile explanation is always the 
>> same -- 'RFI'.  But that's not the case here.  I checked *all* the 
>> raw spectra from the data used for the S&R regimen.  It's all 
>> clean.   If 'RFI' is to blame, it's outside the 2 GHz-wide bandpass 
>> that I can see. There are TV downlinks above 12 GHz, but you would 
>> expect them to affect the Ku-band data more.  The Ku-band data do not 
>> show this problem at anywhere near the level seen in X-band.
>>
>> --------------------------------------------------------
>>
>>      Bottom line:  (for real, in this case).  The system is failing 
>> us. It has easily measureable consequences -- significant loss in 
>> sensitivity.  This is not the first time I've pointed this out -- I 
>> have a presentation, made a couple years ago, showing bad power 
>> levels being set.   The problem is still with us, and deserves 
>> attention.
>>
>>
>> _______________________________________________
>> evlatests mailing list
>> evlatests at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/evlatests
>>
>
> _______________________________________________
> evlatests mailing list
> evlatests at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/evlatests