[Difx-users] zoom/recband and ACCOR for DiFX 2.6.1

[Adam Deller]

hi Jan,

First off, nice sleuthing to track down the proximate cause.

Second, I agree that it is pretty weird that the mere presence of a
flipped-sideband-equivalent autocorrelation makes ACCOR lose its marbles.
Makes me wonder what manner of duct tape is holding things together under
the hood...  Are the contents of the FQ tables identical in both cases?  It
would be good to then also check with e.g. UVPRT which IFs the visibilities
are ultimately being linked against.  But since the bug no longer gets
exercised once DiFX is corrected, perhaps you don't want to spend more time
chasing this.

Third, I can shed some light on equivfrequsedbybaseline.  This was
introduced in 2013 to solve a problem unrelated to zoom bands: pulse cal.
In some HSA observations, the VLA had an identical frequency setup to the
VLBA, but with no pulse cal.  So the VLA frequencies pointed to different
entries in the FQ table, even through they were the same bandwidth,
sideband, etc.  Since the VLA FQ entries were not used by any baseline, the
VLA autocorrelations were getting discarded.  Hence, equivfreqused was
introduced.  But perhaps the test for equivalence should be stricter,
enforcing that the sideband sense also be the same, rather than just the
lower band edge and bandwidth?  I guess that would solve your problem, and
I *think* it would not introduce any new ones.  Walter B may want to
comment if he thinks otherwise!

Cheers,
Adam



On Mon, 9 Dec 2019 at 23:43, Jan Wagner <jwagner105 at googlemail.com> wrote:

> Hi Adam,
>
> the raw spectra looked consistent (plotDiFX.py, m5spec) and there were no
> large DC offsets nor large Nyquist bin amplitudes.
>
> The ACCOR scaling issue turned out to be related in some way to duplicate
> autocorrelation entries in difx output. DiFX produced autocorrelations for
> the 32 MHz wide recorded bands (LSB), but in addition also produced
> autocorrelations for the 32 MHz wide zoom bands. In our test case the zooms
> are just the USB-flipped version of the recorded LSB bands. So in a sense
> the USB zoom & LSB recband autocorrelation records are redundant as they
> sample the same sky frequency range.
>
> Two workarounds were independently successful for ACCOR:
>
> 1 - original vex2difx, correlate, filter afterwards and remove redundant
> non-zoom autocorrelations, difx2fits conversion, FITLD&ACCOR: scaling is
> correct
>
> 2 - modified vex2difx so baselines reference only the zoom bands,
> recorrelate, no not filter but just run difx2fits, FITLD&ACCOR: scaling is
> correct
>
> Still not 100% sure what the actual issue is that triggers ACCOR
> mis-scaling. It was surprising that even though DiFX output from (2) still
> contains redundant autocorrelations, AIPS ACCOR worked fine and there was
> no need for .difx pre-filtering as in (1).
>
> The vex2difx modification was prompted by strange default vex2difx
> behaviour. Vex2difx chooses zooms for the first station on a baseline, but
> for the second station it prefers recorded bands over zoom bands. And it
> ignores the 'noparent=true' of those latter zooms, that ought to have
> blocked the parents of those zooms from correlation.
>
> To illustrate...
>
> v2d: addZoomFreq = freq at 86348.00/bw at 32.0/noparent at true
>
> inputfile produced by vex2difx:
> Ys zoom 86348U 32.0 x Ef rec 86380L 32.0
> Ef zoom 86348U 32.0 x Ys rec 86380L 32.0
> On zoom 86348U 32.0 x Ys rec 86380L 32.0
>
> I.e. vex2difx ends up correlating zoom x recorded parent, instead of zoom
> x zoom. I've added a small change to trunk vex2difx (commit r9362). Now
> addZoomFreq 'noparent=true' gets honoured when looking for a matching band
> at the second station, and the vex2difx result is closer to expected, e.g.,
>
> inputfile produced by updated vex2difx (r9362):
> Ys zoom 86348U 32.0 x Ef zoom 86348U 32.0
> Ef zoom 86348U 32.0 x Ys zoom 86348U 32.0
> On zoom 86348U 32.0 x Ys zoom 86348U 32.0
>
> Like mentioned earlier when I recorrelate with this proper zoom x zoom
> -only inputfile, the final resultant FITS yielded normal corrections in
> ACCOR. Perhaps difx2fits treats USB zoom x LSB recband somehow differently
> and adjusts the auto spectra, and treats correctly rec x rec or zoom x zoom
> cases and their auto spectra...
>
> Anyway, our issue is sort of fixed in DiFX trunk now. Or at least worked
> around...
>
>
> It is curious though that even with zoom x zoom -only baselines the DiFX
> output still has redundant autocorrelations from rec bands, e.g.,
>
> Ef-Ef auto 86348U 32.0 from zoom
> Ef-Ef auto 86380L 32.0 from rec band
> Ys-Ys auto 86348U 32.0 from zoom
> Ys-Ys auto 86380L 32.0 from rec band
> ...
>
> A question is whether those redundant autocorrelations (zoom + recband)
> should truly be present in the DiFX output?
>
> Looking at DiFX configuration.cpp, all frequencies referenced from the
> BASELINE table are set to 'frequsedbybaseline[<fqId>] = true'. This marks
> them so that autocorrelation spectra will be output. In our case those
> would be zoom band autocorrelations.
>
> However, configuration.cpp adds the concept of "equivalent frequency". All
> frequencies in the FREQ table that are unreferenced by any BASELINE are
> compared against the frequencies used on the baselines. When an
> unreferenced frequency matches one actually used on a baseline (same sky
> freq range, same bandwidth) then that unreferenced frequency gets set
> 'equivfrequsedbybaseline[<fqId>] = true'.
>
> In our test case the zoom x zoom baselines (with autocorrelations via
> 'frequsedbybaseline[<fqId>] = true') have matching recorded bands or
> recorded frequencies that are not producing visibility data, but, they will
> nevertheless be enabled for output of (redundant) autocorrelations via the
> "equivalent frequency" check.
>
> Does this 'equivfrequsedbybaseline' still have an active legit use?
>
> Or is it maybe a leftover from DiFX 2.5?
>
> Are the auto spectra of frequencies not utilized by any BASELINE still
> useful somewhere for calibration?
>
> regards,
> Jan
>
> Am 26.11.2019 um 23:35 schrieb Adam Deller:
>
> Hi Jan,
>
> What if you plot the raw (uncorrected) autocorrelation and cross
> correlation amplitudes using POSSM? Based on what you've written, I expect
> it will show that the cross correlation amplitudes are identical in the two
> correlations and the autocorrelation amplitudes are different.  This of
> course should not be, but that would be consistent with what ACCOR is
> doing.  But it would be good to isolate that first, because then that tells
> us we need to go find how the autocorrelation amplitudes could possibly be
> different with a simple flip.
>
> The one thing I could think of is that there is a large DC spike and it
> has been zeroed out in the zoom band case (because the DC channel became
> the upper edge channel and gets forced to zero) and that has affected the
> band-average autocorrelation amplitude.  POSSUM spectra of the raw
> amplitudes should confirm one way or the other.
>
> Cheers,
> Adam
>
> On Tue, 26 Nov 2019 at 02:13, Jan Wagner via Difx-users <
> difx-users at listmgr.nrao.edu> wrote:
>
>> Noted one strange issue in DiFX 2.6.1. Has anyone noticed this and
>> can confirm it?
>>
>> The scaling of DiFX data converted with difx2fits and loaded into
>> AIPS appears to be different when using zoom band correlation,
>> versus correlating natively recorded bands:
>>
>>
>> Case: Stations recoded 32 MHz LSB at 86348.00 MHz. Two correlation
>> runs with DiFX 2.6.1.
>>
>> Correlation (1): 32 MHz LSB recorded bands flipped to USB via
>> addZoomFreq freq at 86348.00/bw at 32.0/noparent at true
>>
>> Correlation (2): 32 MHz LSB recorded bands correlated as-is
>>
>>
>> Processing: difx2fits, AIPS FITLD digicor=1, ACCOR solint=2/60,
>> CLCAL #2 = CL#1 + SN#Accor, POSSM cross power spectra docalib=1
>> gainuse=CL#2.
>>
>>
>> Result: compared "flux density" on three baselines and 4 IFs
>>
>> Data from zoom correlation (1): amplitudes ~0.5 Jy, accor gains
>> ~0.65 or thereabouts
>>
>> Data from rec band correlation (2): amplitudes ~1.0 Jy, accor gains
>> ~1.00
>>
>>
>> It is strange that 32 MHz zoom (identical to 32 MHz recoderd except
>> for sideband flip to USB) would produce a factor 2 difference in the
>> scaling of cross power spectra. Has anyone noticed this? Is there a
>> workaround? The experiments in question were already correlated some
>> months ago and not all raw data are available anymore. In case
>> someone has already ran into this issue, is simply a "manual"
>> rescaling by factor 2 in AIPS a permissible solution?
>>
>> regards,
>> Jan
>>
>>
>> _______________________________________________
>> Difx-users mailing list
>> Difx-users at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>>
>
>
> --
> !=============================================================!
> A/Prof. Adam Deller
> ARC Future Fellow
> Centre for Astrophysics & Supercomputing
> Swinburne University of Technology
> John St, Hawthorn VIC 3122 Australia
> phone: +61 3 9214 5307
> fax: +61 3 9214 8797
>
> office days (usually): Mon-Thu
> !=============================================================!
>
>
>

-- 
!=============================================================!
A/Prof. Adam Deller
ARC Future Fellow
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797

office days (usually): Mon-Thu
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20191210/28616a15/attachment.html>