[Difx-users] zoom/recband and ACCOR for DiFX 2.6.1

[Jan Wagner]

Hi Adam,

the raw spectra looked consistent (plotDiFX.py, m5spec) and there
were no large DC offsets nor large Nyquist bin amplitudes.

The ACCOR scaling issue turned out to be related in some way to
duplicate autocorrelation entries in difx output. DiFX produced
autocorrelations for the 32 MHz wide recorded bands (LSB), but in
addition also produced autocorrelations for the 32 MHz wide zoom
bands. In our test case the zooms are just the USB-flipped version
of the recorded LSB bands. So in a sense the USB zoom & LSB recband
autocorrelation records are redundant as they sample the same sky
frequency range.

Two workarounds were independently successful for ACCOR:

1 - original vex2difx, correlate, filter afterwards and remove
redundant non-zoom autocorrelations, difx2fits conversion,
FITLD&ACCOR: scaling is correct

2 - modified vex2difx so baselines reference only the zoom bands,
recorrelate, no not filter but just run difx2fits, FITLD&ACCOR:
scaling is correct

Still not 100% sure what the actual issue is that triggers ACCOR
mis-scaling. It was surprising that even though DiFX output from (2)
still contains redundant autocorrelations, AIPS ACCOR worked fine
and there was no need for .difx pre-filtering as in (1).

The vex2difx modification was prompted by strange default vex2difx
behaviour. Vex2difx chooses zooms for the first station on a
baseline, but for the second station it prefers recorded bands over
zoom bands. And it ignores the 'noparent=true' of those latter
zooms, that ought to have blocked the parents of those zooms from
correlation.

To illustrate...

v2d: addZoomFreq = freq at 86348.00/bw at 32.0/noparent at true

inputfile produced by vex2difx:
Ys zoom 86348U 32.0 x Ef rec 86380L 32.0
Ef zoom 86348U 32.0 x Ys rec 86380L 32.0
On zoom 86348U 32.0 x Ys rec 86380L 32.0

I.e. vex2difx ends up correlating zoom x recorded parent, instead of
zoom x zoom. I've added a small change to trunk vex2difx (commit
r9362). Now addZoomFreq 'noparent=true' gets honoured when looking
for a matching band at the second station, and the vex2difx result
is closer to expected, e.g.,

inputfile produced by updated vex2difx (r9362):
Ys zoom 86348U 32.0 x Ef zoom 86348U 32.0
Ef zoom 86348U 32.0 x Ys zoom 86348U 32.0
On zoom 86348U 32.0 x Ys zoom 86348U 32.0

Like mentioned earlier when I recorrelate with this proper zoom x
zoom -only inputfile, the final resultant FITS yielded normal
corrections in ACCOR. Perhaps difx2fits treats USB zoom x LSB
recband somehow differently and adjusts the auto spectra, and treats
correctly rec x rec or zoom x zoom cases and their auto spectra...

Anyway, our issue is sort of fixed in DiFX trunk now. Or at least
worked around...


It is curious though that even with zoom x zoom -only baselines the
DiFX output still has redundant autocorrelations from rec bands, e.g.,

Ef-Ef auto 86348U 32.0 from zoom
Ef-Ef auto 86380L 32.0 from rec band
Ys-Ys auto 86348U 32.0 from zoom
Ys-Ys auto 86380L 32.0 from rec band
...

A question is whether those redundant autocorrelations (zoom +
recband) should truly be present in the DiFX output?

Looking at DiFX configuration.cpp, all frequencies referenced from
the BASELINE table are set to 'frequsedbybaseline[<fqId>] = true'.
This marks them so that autocorrelation spectra will be output. In
our case those would be zoom band autocorrelations.

However, configuration.cpp adds the concept of "equivalent
frequency". All frequencies in the FREQ table that are unreferenced
by any BASELINE are compared against the frequencies used on the
baselines. When an unreferenced frequency matches one actually used
on a baseline (same sky freq range, same bandwidth) then that
unreferenced frequency gets set 'equivfrequsedbybaseline[<fqId>] =
true'.

In our test case the zoom x zoom baselines (with autocorrelations
via 'frequsedbybaseline[<fqId>] = true') have matching recorded
bands or recorded frequencies that are not producing visibility
data, but, they will nevertheless be enabled for output of
(redundant) autocorrelations via the "equivalent frequency" check.

Does this 'equivfrequsedbybaseline' still have an active legit use?

Or is it maybe a leftover from DiFX 2.5?

Are the auto spectra of frequencies not utilized by any BASELINE
still useful somewhere for calibration?

regards,
Jan

Am 26.11.2019 um 23:35 schrieb Adam Deller:
> Hi Jan,
>
> What if you plot the raw (uncorrected) autocorrelation and cross
> correlation amplitudes using POSSM? Based on what you've written,
> I expect it will show that the cross correlation amplitudes are
> identical in the two correlations and the autocorrelation
> amplitudes are different.  This of course should not be, but that
> would be consistent with what ACCOR is doing.  But it would be
> good to isolate that first, because then that tells us we need to
> go find how the autocorrelation amplitudes could possibly be
> different with a simple flip.
>
> The one thing I could think of is that there is a large DC spike
> and it has been zeroed out in the zoom band case (because the DC
> channel became the upper edge channel and gets forced to zero) and
> that has affected the band-average autocorrelation amplitude. 
> POSSUM spectra of the raw amplitudes should confirm one way or the
> other.
>
> Cheers,
> Adam
>
> On Tue, 26 Nov 2019 at 02:13, Jan Wagner via Difx-users
> <difx-users at listmgr.nrao.edu <mailto:difx-users at listmgr.nrao.edu>>
> wrote:
>
>     Noted one strange issue in DiFX 2.6.1. Has anyone noticed this and
>     can confirm it?
>
>     The scaling of DiFX data converted with difx2fits and loaded into
>     AIPS appears to be different when using zoom band correlation,
>     versus correlating natively recorded bands:
>
>
>     Case: Stations recoded 32 MHz LSB at 86348.00 MHz. Two correlation
>     runs with DiFX 2.6.1.
>
>     Correlation (1): 32 MHz LSB recorded bands flipped to USB via
>     addZoomFreq freq at 86348.00/bw at 32.0/noparent at true
>
>     Correlation (2): 32 MHz LSB recorded bands correlated as-is
>
>
>     Processing: difx2fits, AIPS FITLD digicor=1, ACCOR solint=2/60,
>     CLCAL #2 = CL#1 + SN#Accor, POSSM cross power spectra docalib=1
>     gainuse=CL#2.
>
>
>     Result: compared "flux density" on three baselines and 4 IFs
>
>     Data from zoom correlation (1): amplitudes ~0.5 Jy, accor gains
>     ~0.65 or thereabouts
>
>     Data from rec band correlation (2): amplitudes ~1.0 Jy, accor
>     gains
>     ~1.00
>
>
>     It is strange that 32 MHz zoom (identical to 32 MHz recoderd
>     except
>     for sideband flip to USB) would produce a factor 2 difference
>     in the
>     scaling of cross power spectra. Has anyone noticed this? Is
>     there a
>     workaround? The experiments in question were already
>     correlated some
>     months ago and not all raw data are available anymore. In case
>     someone has already ran into this issue, is simply a "manual"
>     rescaling by factor 2 in AIPS a permissible solution?
>
>     regards,
>     Jan
>
>
>     _______________________________________________
>     Difx-users mailing list
>     Difx-users at listmgr.nrao.edu <mailto:Difx-users at listmgr.nrao.edu>
>     https://listmgr.nrao.edu/mailman/listinfo/difx-users
>
>
>
> -- 
> !=============================================================!
> A/Prof. Adam Deller         
> ARC Future Fellow
> Centre for Astrophysics & Supercomputing 
> Swinburne University of Technology    
> John St, Hawthorn VIC 3122 Australia
> phone: +61 3 9214 5307
> fax: +61 3 9214 8797
>
> office days (usually): Mon-Thu
> !=============================================================!


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20191209/587d2bac/attachment.html>