[Difx-users] zoom/recband and ACCOR for DiFX 2.6.1
Walter Brisken
wbrisken at nrao.edu
Tue Dec 10 08:43:10 EST 2019
Probably best to find or generate a test dataset for this. I had offered
to arrange for collection of a test data set for output bands anyway. We
could add a couple scans that could verify this as well.
-W
-------------------------
Walter Brisken
NRAO
Deputy Assistant Director for VLBA Development
(575)-835-7133 (office)
(505)-234-5912 (cell)
On Tue, 10 Dec 2019, Adam Deller via Difx-users wrote:
> hi Jan,
> First off, nice sleuthing to track down the proximate cause.
>
> Second, I agree that it is pretty weird that the mere presence of a flipped-sideband-equivalent autocorrelation
> makes ACCOR lose its marbles. Makes me wonder what manner of duct tape is holding things together under the
> hood... Are the contents of the FQ tables identical in both cases? It would be good to then also check with e.g.
> UVPRT which IFs the visibilities are ultimately being linked against. But since the bug no longer gets exercised
> once DiFX is corrected, perhaps you don't want to spend more time chasing this.
>
> Third, I can shed some light on equivfrequsedbybaseline. This was introduced in 2013 to solve a problem unrelated
> to zoom bands: pulse cal. In some HSA observations, the VLA had an identical frequency setup to the VLBA, but with
> no pulse cal. So the VLA frequencies pointed to different entries in the FQ table, even through they were the same
> bandwidth, sideband, etc. Since the VLA FQ entries were not used by any baseline, the VLA autocorrelations were
> getting discarded. Hence, equivfreqused was introduced. But perhaps the test for equivalence should be stricter,
> enforcing that the sideband sense also be the same, rather than just the lower band edge and bandwidth? I guess
> that would solve your problem, and I *think* it would not introduce any new ones. Walter B may want to comment if
> he thinks otherwise!
>
> Cheers,
> Adam
>
>
>
> On Mon, 9 Dec 2019 at 23:43, Jan Wagner <jwagner105 at googlemail.com> wrote:
> Hi Adam,
>
> the raw spectra looked consistent (plotDiFX.py, m5spec) and there were no large DC offsets nor large Nyquist
> bin amplitudes.
>
> The ACCOR scaling issue turned out to be related in some way to duplicate autocorrelation entries in difx
> output. DiFX produced autocorrelations for the 32 MHz wide recorded bands (LSB), but in addition also
> produced autocorrelations for the 32 MHz wide zoom bands. In our test case the zooms are just the USB-flipped
> version of the recorded LSB bands. So in a sense the USB zoom & LSB recband autocorrelation records are
> redundant as they sample the same sky frequency range.
>
> Two workarounds were independently successful for ACCOR:
>
> 1 - original vex2difx, correlate, filter afterwards and remove redundant non-zoom autocorrelations, difx2fits
> conversion, FITLD&ACCOR: scaling is correct
>
> 2 - modified vex2difx so baselines reference only the zoom bands, recorrelate, no not filter but just run
> difx2fits, FITLD&ACCOR: scaling is correct
>
> Still not 100% sure what the actual issue is that triggers ACCOR mis-scaling. It was surprising that even
> though DiFX output from (2) still contains redundant autocorrelations, AIPS ACCOR worked fine and there was
> no need for .difx pre-filtering as in (1).
>
> The vex2difx modification was prompted by strange default vex2difx behaviour. Vex2difx chooses zooms for the
> first station on a baseline, but for the second station it prefers recorded bands over zoom bands. And it
> ignores the 'noparent=true' of those latter zooms, that ought to have blocked the parents of those zooms from
> correlation.
>
> To illustrate...
>
> v2d: addZoomFreq = freq at 86348.00/bw at 32.0/noparent at true
>
> inputfile produced by vex2difx:
> Ys zoom 86348U 32.0 x Ef rec 86380L 32.0
> Ef zoom 86348U 32.0 x Ys rec 86380L 32.0
> On zoom 86348U 32.0 x Ys rec 86380L 32.0
>
> I.e. vex2difx ends up correlating zoom x recorded parent, instead of zoom x zoom. I've added a small change
> to trunk vex2difx (commit r9362). Now addZoomFreq 'noparent=true' gets honoured when looking for a matching
> band at the second station, and the vex2difx result is closer to expected, e.g.,
>
> inputfile produced by updated vex2difx (r9362):
> Ys zoom 86348U 32.0 x Ef zoom 86348U 32.0
> Ef zoom 86348U 32.0 x Ys zoom 86348U 32.0
> On zoom 86348U 32.0 x Ys zoom 86348U 32.0
>
> Like mentioned earlier when I recorrelate with this proper zoom x zoom -only inputfile, the final resultant
> FITS yielded normal corrections in ACCOR. Perhaps difx2fits treats USB zoom x LSB recband somehow differently
> and adjusts the auto spectra, and treats correctly rec x rec or zoom x zoom cases and their auto spectra...
>
> Anyway, our issue is sort of fixed in DiFX trunk now. Or at least worked around...
>
>
> It is curious though that even with zoom x zoom -only baselines the DiFX output still has redundant
> autocorrelations from rec bands, e.g.,
>
> Ef-Ef auto 86348U 32.0 from zoom
> Ef-Ef auto 86380L 32.0 from rec band
> Ys-Ys auto 86348U 32.0 from zoom
> Ys-Ys auto 86380L 32.0 from rec band
> ...
>
> A question is whether those redundant autocorrelations (zoom + recband) should truly be present in the DiFX
> output?
>
> Looking at DiFX configuration.cpp, all frequencies referenced from the BASELINE table are set to
> 'frequsedbybaseline[<fqId>] = true'. This marks them so that autocorrelation spectra will be output. In our
> case those would be zoom band autocorrelations.
>
> However, configuration.cpp adds the concept of "equivalent frequency". All frequencies in the FREQ table that
> are unreferenced by any BASELINE are compared against the frequencies used on the baselines. When an
> unreferenced frequency matches one actually used on a baseline (same sky freq range, same bandwidth) then
> that unreferenced frequency gets set 'equivfrequsedbybaseline[<fqId>] = true'.
>
> In our test case the zoom x zoom baselines (with autocorrelations via 'frequsedbybaseline[<fqId>] = true')
> have matching recorded bands or recorded frequencies that are not producing visibility data, but, they will
> nevertheless be enabled for output of (redundant) autocorrelations via the "equivalent frequency" check.
>
> Does this 'equivfrequsedbybaseline' still have an active legit use?
>
> Or is it maybe a leftover from DiFX 2.5?
>
> Are the auto spectra of frequencies not utilized by any BASELINE still useful somewhere for calibration?
>
> regards,
> Jan
>
> Am 26.11.2019 um 23:35 schrieb Adam Deller:
> Hi Jan,
> What if you plot the raw (uncorrected) autocorrelation and cross correlation amplitudes using POSSM?
> Based on what you've written, I expect it will show that the cross correlation amplitudes are identical
> in the two correlations and the autocorrelation amplitudes are different. This of course should not
> be, but that would be consistent with what ACCOR is doing. But it would be good to isolate that first,
> because then that tells us we need to go find how the autocorrelation amplitudes could possibly be
> different with a simple flip.
>
> The one thing I could think of is that there is a large DC spike and it has been zeroed out in the zoom
> band case (because the DC channel became the upper edge channel and gets forced to zero) and that has
> affected the band-average autocorrelation amplitude. POSSUM spectra of the raw amplitudes should
> confirm one way or the other.
>
> Cheers,
> Adam
>
> On Tue, 26 Nov 2019 at 02:13, Jan Wagner via Difx-users <difx-users at listmgr.nrao.edu> wrote:
> Noted one strange issue in DiFX 2.6.1. Has anyone noticed this and
> can confirm it?
>
> The scaling of DiFX data converted with difx2fits and loaded into
> AIPS appears to be different when using zoom band correlation,
> versus correlating natively recorded bands:
>
>
> Case: Stations recoded 32 MHz LSB at 86348.00 MHz. Two correlation
> runs with DiFX 2.6.1.
>
> Correlation (1): 32 MHz LSB recorded bands flipped to USB via
> addZoomFreq freq at 86348.00/bw at 32.0/noparent at true
>
> Correlation (2): 32 MHz LSB recorded bands correlated as-is
>
>
> Processing: difx2fits, AIPS FITLD digicor=1, ACCOR solint=2/60,
> CLCAL #2 = CL#1 + SN#Accor, POSSM cross power spectra docalib=1
> gainuse=CL#2.
>
>
> Result: compared "flux density" on three baselines and 4 IFs
>
> Data from zoom correlation (1): amplitudes ~0.5 Jy, accor gains
> ~0.65 or thereabouts
>
> Data from rec band correlation (2): amplitudes ~1.0 Jy, accor gains
> ~1.00
>
>
> It is strange that 32 MHz zoom (identical to 32 MHz recoderd except
> for sideband flip to USB) would produce a factor 2 difference in the
> scaling of cross power spectra. Has anyone noticed this? Is there a
> workaround? The experiments in question were already correlated some
> months ago and not all raw data are available anymore. In case
> someone has already ran into this issue, is simply a "manual"
> rescaling by factor 2 in AIPS a permissible solution?
>
> regards,
> Jan
>
>
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>
>
>
> --
> !=============================================================!
> A/Prof. Adam Deller
> ARC Future Fellow
> Centre for Astrophysics & Supercomputing
> Swinburne University of Technology
> John St, Hawthorn VIC 3122 Australia
> phone: +61 3 9214 5307
> fax: +61 3 9214 8797
>
> office days (usually): Mon-Thu
> !=============================================================!
>
>
>
>
> --
> !=============================================================!
> A/Prof. Adam Deller
> ARC Future Fellow
> Centre for Astrophysics & Supercomputing
> Swinburne University of Technology
> John St, Hawthorn VIC 3122 Australia
> phone: +61 3 9214 5307
> fax: +61 3 9214 8797
>
> office days (usually): Mon-Thu
> !=============================================================!
>
>
More information about the Difx-users
mailing list