[Difx-users] zoom/recband and ACCOR for DiFX 2.6.1

[Adam Deller]

That would be great!

On Wed, 11 Dec 2019 at 00:43, Walter Brisken via Difx-users <
difx-users at listmgr.nrao.edu> wrote:

>
> Probably best to find or generate a test dataset for this.  I had offered
> to arrange for collection of a test data set for output bands anyway.  We
> could add a couple scans that could verify this as well.
>
> -W
>
>
>
> -------------------------
> Walter Brisken
> NRAO
> Deputy Assistant Director for VLBA Development
> (575)-835-7133 (office)
> (505)-234-5912 (cell)
>
> On Tue, 10 Dec 2019, Adam Deller via Difx-users wrote:
>
> > hi Jan,
> > First off, nice sleuthing to track down the proximate cause.
> >
> > Second, I agree that it is pretty weird that the mere presence of a
> flipped-sideband-equivalent autocorrelation
> > makes ACCOR lose its marbles.  Makes me wonder what manner of duct tape
> is holding things together under the
> > hood...  Are the contents of the FQ tables identical in both cases?  It
> would be good to then also check with e.g.
> > UVPRT which IFs the visibilities are ultimately being linked against.
> But since the bug no longer gets exercised
> > once DiFX is corrected, perhaps you don't want to spend more time
> chasing this.
> >
> > Third, I can shed some light on equivfrequsedbybaseline.  This was
> introduced in 2013 to solve a problem unrelated
> > to zoom bands: pulse cal.  In some HSA observations, the VLA had an
> identical frequency setup to the VLBA, but with
> > no pulse cal.  So the VLA frequencies pointed to different entries in
> the FQ table, even through they were the same
> > bandwidth, sideband, etc.  Since the VLA FQ entries were not used by any
> baseline, the VLA autocorrelations were
> > getting discarded.  Hence, equivfreqused was introduced.  But perhaps
> the test for equivalence should be stricter,
> > enforcing that the sideband sense also be the same, rather than just the
> lower band edge and bandwidth?  I guess
> > that would solve your problem, and I *think* it would not introduce any
> new ones.  Walter B may want to comment if
> > he thinks otherwise!
> >
> > Cheers,
> > Adam
> >
> >
> >
> > On Mon, 9 Dec 2019 at 23:43, Jan Wagner <jwagner105 at googlemail.com>
> wrote:
> >       Hi Adam,
> >
> > the raw spectra looked consistent (plotDiFX.py, m5spec) and there were
> no large DC offsets nor large Nyquist
> > bin amplitudes.
> >
> > The ACCOR scaling issue turned out to be related in some way to
> duplicate autocorrelation entries in difx
> > output. DiFX produced autocorrelations for the 32 MHz wide recorded
> bands (LSB), but in addition also
> > produced autocorrelations for the 32 MHz wide zoom bands. In our test
> case the zooms are just the USB-flipped
> > version of the recorded LSB bands. So in a sense the USB zoom & LSB
> recband autocorrelation records are
> > redundant as they sample the same sky frequency range.
> >
> > Two workarounds were independently successful for ACCOR:
> >
> > 1 - original vex2difx, correlate, filter afterwards and remove redundant
> non-zoom autocorrelations, difx2fits
> > conversion, FITLD&ACCOR: scaling is correct
> >
> > 2 - modified vex2difx so baselines reference only the zoom bands,
> recorrelate, no not filter but just run
> > difx2fits, FITLD&ACCOR: scaling is correct
> >
> > Still not 100% sure what the actual issue is that triggers ACCOR
> mis-scaling. It was surprising that even
> > though DiFX output from (2) still contains redundant autocorrelations,
> AIPS ACCOR worked fine and there was
> > no need for .difx pre-filtering as in (1).
> >
> > The vex2difx modification was prompted by strange default vex2difx
> behaviour. Vex2difx chooses zooms for the
> > first station on a baseline, but for the second station it prefers
> recorded bands over zoom bands. And it
> > ignores the 'noparent=true' of those latter zooms, that ought to have
> blocked the parents of those zooms from
> > correlation.
> >
> > To illustrate...
> >
> > v2d: addZoomFreq = freq at 86348.00/bw at 32.0/noparent at true
> >
> > inputfile produced by vex2difx:
> > Ys zoom 86348U 32.0 x Ef rec 86380L 32.0
> > Ef zoom 86348U 32.0 x Ys rec 86380L 32.0
> > On zoom 86348U 32.0 x Ys rec 86380L 32.0
> >
> > I.e. vex2difx ends up correlating zoom x recorded parent, instead of
> zoom x zoom. I've added a small change
> > to trunk vex2difx (commit r9362). Now addZoomFreq 'noparent=true' gets
> honoured when looking for a matching
> > band at the second station, and the vex2difx result is closer to
> expected, e.g.,
> >
> > inputfile produced by updated vex2difx (r9362):
> > Ys zoom 86348U 32.0 x Ef zoom 86348U 32.0
> > Ef zoom 86348U 32.0 x Ys zoom 86348U 32.0
> > On zoom 86348U 32.0 x Ys zoom 86348U 32.0
> >
> > Like mentioned earlier when I recorrelate with this proper zoom x zoom
> -only inputfile, the final resultant
> > FITS yielded normal corrections in ACCOR. Perhaps difx2fits treats USB
> zoom x LSB recband somehow differently
> > and adjusts the auto spectra, and treats correctly rec x rec or zoom x
> zoom cases and their auto spectra...
> >
> > Anyway, our issue is sort of fixed in DiFX trunk now. Or at least worked
> around...
> >
> >
> > It is curious though that even with zoom x zoom -only baselines the DiFX
> output still has redundant
> > autocorrelations from rec bands, e.g.,
> >
> > Ef-Ef auto 86348U 32.0 from zoom
> > Ef-Ef auto 86380L 32.0 from rec band
> > Ys-Ys auto 86348U 32.0 from zoom
> > Ys-Ys auto 86380L 32.0 from rec band
> > ...
> >
> > A question is whether those redundant autocorrelations (zoom + recband)
> should truly be present in the DiFX
> > output?
> >
> > Looking at DiFX configuration.cpp, all frequencies referenced from the
> BASELINE table are set to
> > 'frequsedbybaseline[<fqId>] = true'. This marks them so that
> autocorrelation spectra will be output. In our
> > case those would be zoom band autocorrelations.
> >
> > However, configuration.cpp adds the concept of "equivalent frequency".
> All frequencies in the FREQ table that
> > are unreferenced by any BASELINE are compared against the frequencies
> used on the baselines. When an
> > unreferenced frequency matches one actually used on a baseline (same sky
> freq range, same bandwidth) then
> > that unreferenced frequency gets set 'equivfrequsedbybaseline[<fqId>] =
> true'.
> >
> > In our test case the zoom x zoom baselines (with autocorrelations via
> 'frequsedbybaseline[<fqId>] = true')
> > have matching recorded bands or recorded frequencies that are not
> producing visibility data, but, they will
> > nevertheless be enabled for output of (redundant) autocorrelations via
> the "equivalent frequency" check.
> >
> > Does this 'equivfrequsedbybaseline' still have an active legit use?
> >
> > Or is it maybe a leftover from DiFX 2.5?
> >
> > Are the auto spectra of frequencies not utilized by any BASELINE still
> useful somewhere for calibration?
> >
> > regards,
> > Jan
> >
> > Am 26.11.2019 um 23:35 schrieb Adam Deller:
> >       Hi Jan,
> > What if you plot the raw (uncorrected) autocorrelation and cross
> correlation amplitudes using POSSM?
> > Based on what you've written, I expect it will show that the cross
> correlation amplitudes are identical
> > in the two correlations and the autocorrelation amplitudes are
> different.  This of course should not
> > be, but that would be consistent with what ACCOR is doing.  But it would
> be good to isolate that first,
> > because then that tells us we need to go find how the autocorrelation
> amplitudes could possibly be
> > different with a simple flip.
> >
> > The one thing I could think of is that there is a large DC spike and it
> has been zeroed out in the zoom
> > band case (because the DC channel became the upper edge channel and gets
> forced to zero) and that has
> > affected the band-average autocorrelation amplitude.  POSSUM spectra of
> the raw amplitudes should
> > confirm one way or the other.
> >
> > Cheers,
> > Adam
> >
> > On Tue, 26 Nov 2019 at 02:13, Jan Wagner via Difx-users <
> difx-users at listmgr.nrao.edu> wrote:
> >       Noted one strange issue in DiFX 2.6.1. Has anyone noticed this and
> >       can confirm it?
> >
> >       The scaling of DiFX data converted with difx2fits and loaded into
> >       AIPS appears to be different when using zoom band correlation,
> >       versus correlating natively recorded bands:
> >
> >
> >       Case: Stations recoded 32 MHz LSB at 86348.00 MHz. Two correlation
> >       runs with DiFX 2.6.1.
> >
> >       Correlation (1): 32 MHz LSB recorded bands flipped to USB via
> >       addZoomFreq freq at 86348.00/bw at 32.0/noparent at true
> >
> >       Correlation (2): 32 MHz LSB recorded bands correlated as-is
> >
> >
> >       Processing: difx2fits, AIPS FITLD digicor=1, ACCOR solint=2/60,
> >       CLCAL #2 = CL#1 + SN#Accor, POSSM cross power spectra docalib=1
> >       gainuse=CL#2.
> >
> >
> >       Result: compared "flux density" on three baselines and 4 IFs
> >
> >       Data from zoom correlation (1): amplitudes ~0.5 Jy, accor gains
> >       ~0.65 or thereabouts
> >
> >       Data from rec band correlation (2): amplitudes ~1.0 Jy, accor gains
> >       ~1.00
> >
> >
> >       It is strange that 32 MHz zoom (identical to 32 MHz recoderd except
> >       for sideband flip to USB) would produce a factor 2 difference in
> the
> >       scaling of cross power spectra. Has anyone noticed this? Is there a
> >       workaround? The experiments in question were already correlated
> some
> >       months ago and not all raw data are available anymore. In case
> >       someone has already ran into this issue, is simply a "manual"
> >       rescaling by factor 2 in AIPS a permissible solution?
> >
> >       regards,
> >       Jan
> >
> >
> >       _______________________________________________
> >       Difx-users mailing list
> >       Difx-users at listmgr.nrao.edu
> >       https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >
> >
> >
> > --
> > !=============================================================!
> > A/Prof. Adam Deller
> > ARC Future Fellow
> > Centre for Astrophysics & Supercomputing
> > Swinburne University of Technology
> > John St, Hawthorn VIC 3122 Australia
> > phone: +61 3 9214 5307
> > fax: +61 3 9214 8797
> >
> > office days (usually): Mon-Thu
> > !=============================================================!
> >
> >
> >
> >
> > --
> > !=============================================================!
> > A/Prof. Adam Deller
> > ARC Future Fellow
> > Centre for Astrophysics & Supercomputing
> > Swinburne University of Technology
> > John St, Hawthorn VIC 3122 Australia
> > phone: +61 3 9214 5307
> > fax: +61 3 9214 8797
> >
> > office days (usually): Mon-Thu
> > !=============================================================!
> >
> >_______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>


-- 
!=============================================================!
A/Prof. Adam Deller
ARC Future Fellow
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797

office days (usually): Mon-Thu
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20191211/6176e121/attachment-0001.html>