<div dir="ltr"><div dir="ltr">hi Jan,<div><br></div><div>First off, nice sleuthing to track down the proximate cause. </div><div><br></div><div>Second, I agree that it is pretty weird that the mere presence of a flipped-sideband-equivalent autocorrelation makes ACCOR lose its marbles. Makes me wonder what manner of duct tape is holding things together under the hood... Are the contents of the FQ tables identical in both cases? It would be good to then also check with e.g. UVPRT which IFs the visibilities are ultimately being linked against. But since the bug no longer gets exercised once DiFX is corrected, perhaps you don't want to spend more time chasing this.</div><div><br></div><div>Third, I can shed some light on equivfrequsedbybaseline. This was introduced in 2013 to solve a problem unrelated to zoom bands: pulse cal. In some HSA observations, the VLA had an identical frequency setup to the VLBA, but with no pulse cal. So the VLA frequencies pointed to different entries in the FQ table, even through they were the same bandwidth, sideband, etc. Since the VLA FQ entries were not used by any baseline, the VLA autocorrelations were getting discarded. Hence, equivfreqused was introduced. But perhaps the test for equivalence should be stricter, enforcing that the sideband sense also be the same, rather than just the lower band edge and bandwidth? I guess that would solve your problem, and I *think* it would not introduce any new ones. Walter B may want to comment if he thinks otherwise!</div><div><br></div><div>Cheers,<br>Adam</div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 9 Dec 2019 at 23:43, Jan Wagner <<a href="mailto:jwagner105@googlemail.com" target="_blank">jwagner105@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div>Hi Adam,</div>
<div><br>
</div>
<div>the raw spectra looked consistent
(plotDiFX.py, m5spec) and there were no large DC offsets nor large
Nyquist bin amplitudes.<br>
</div>
<div><br>
</div>
<div>The ACCOR scaling issue turned out to
be related in some way to duplicate autocorrelation entries in
difx output. DiFX produced autocorrelations for the 32 MHz wide
recorded bands (LSB), but in addition also produced
autocorrelations for the 32 MHz wide zoom bands. In our test case
the zooms are just the USB-flipped version of the recorded LSB
bands. So in a sense the USB zoom & LSB recband
autocorrelation records are redundant as they sample the same sky
frequency range.<br>
<br>
Two workarounds were independently successful for ACCOR:<br>
<br>
1 - original vex2difx, correlate, filter afterwards and remove
redundant non-zoom autocorrelations, difx2fits conversion,
FITLD&ACCOR: scaling is correct<br>
<br>
2 - modified vex2difx so baselines reference only the zoom bands,
recorrelate, no not filter but just run difx2fits,
FITLD&ACCOR: scaling is correct<br>
<br>
Still not 100% sure what the actual issue is that triggers ACCOR
mis-scaling. It was surprising that even though DiFX output from
(2) still contains redundant autocorrelations, AIPS ACCOR worked
fine and there was no need for .difx pre-filtering as in (1).<br>
</div>
<div><br>
The vex2difx modification was prompted by strange default vex2difx
behaviour. Vex2difx chooses zooms for the first station on a
baseline, but for the second station it prefers recorded bands
over zoom bands. And it ignores the 'noparent=true' of those
latter zooms, that ought to have blocked the parents of those
zooms from correlation.<br>
<br>
To illustrate...<br>
<br>
v2d: addZoomFreq = <a href="mailto:freq@86348.00/bw@32.0/noparent@true" target="_blank">freq@86348.00/bw@32.0/noparent@true</a><br>
<br>
inputfile produced by vex2difx:<br>
Ys zoom 86348U 32.0 x Ef rec 86380L 32.0<br>
Ef zoom 86348U 32.0 x Ys rec 86380L 32.0<br>
On zoom 86348U 32.0 x Ys rec 86380L 32.0<br>
</div>
<div><br>
</div>
<div>I.e. vex2difx ends up correlating zoom
x recorded parent, instead of zoom x zoom. I've added a small
change to trunk vex2difx (commit r9362). Now addZoomFreq
'noparent=true' gets honoured when looking for a matching band at
the second station, and the vex2difx result is closer to expected,
e.g.,<br>
<br>
inputfile produced by updated vex2difx (r9362):<br>
Ys zoom 86348U 32.0 x Ef zoom 86348U 32.0<br>
Ef zoom 86348U 32.0 x Ys zoom 86348U 32.0<br>
On zoom 86348U 32.0 x Ys zoom 86348U 32.0<br>
<br>
Like mentioned earlier when I recorrelate with this proper zoom x
zoom -only inputfile, the final resultant FITS yielded normal
corrections in ACCOR. Perhaps difx2fits treats USB zoom x LSB
recband somehow differently and adjusts the auto spectra, and
treats correctly rec x rec or zoom x zoom cases and their auto
spectra...<br>
</div>
<div><br>
</div>
<div>Anyway, our issue is sort of fixed in
DiFX trunk now. Or at least worked around...<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>It is curious though that even with
zoom x zoom -only baselines the DiFX output still has redundant
autocorrelations from rec bands, e.g.,</div>
<div><br>
</div>
<div>Ef-Ef auto 86348U 32.0 from zoom<br>
Ef-Ef auto 86380L 32.0 from rec band<br>
Ys-Ys auto 86348U 32.0 from zoom<br>
Ys-Ys auto 86380L 32.0 from rec band<br>
...
<br>
<br>
<div>A question is whether those redundant
autocorrelations (zoom + recband) should truly be present in the
DiFX output?<br>
<br>
Looking at DiFX configuration.cpp, all frequencies referenced
from the BASELINE table are set to
'frequsedbybaseline[<fqId>] = true'. This marks them so
that autocorrelation spectra will be output. In our case those
would be zoom band autocorrelations.<br>
</div>
<br>
However, configuration.cpp adds the concept of "equivalent
frequency". All frequencies in the FREQ table that are
unreferenced by any BASELINE are compared against the frequencies
used on the baselines. When an unreferenced frequency matches one
actually used on a baseline (same sky freq range, same bandwidth)
then that unreferenced frequency gets set
'equivfrequsedbybaseline[<fqId>] = true'. <br>
<br>
In our test case the zoom x zoom baselines (with autocorrelations
via 'frequsedbybaseline[<fqId>] = true') have matching
recorded bands or recorded frequencies that are not producing
visibility data, but, they will nevertheless be enabled for output
of (redundant) autocorrelations via the "equivalent frequency"
check.<br>
<br>
Does this 'equivfrequsedbybaseline' still have an active legit
use?<br>
<br>
Or is it maybe a leftover from DiFX 2.5?<br>
<br>
Are the auto spectra of frequencies not utilized by any BASELINE
still useful somewhere for calibration?<br>
<br>
regards,<br>
Jan</div>
<div><br>
</div>
<div>Am 26.11.2019 um 23:35 schrieb Adam
Deller:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Jan,
<div><br>
</div>
<div>What if you plot the raw (uncorrected) autocorrelation and
cross correlation amplitudes using POSSM? Based on what you've
written, I expect it will show that the cross correlation
amplitudes are identical in the two correlations and the
autocorrelation amplitudes are different. This of course
should not be, but that would be consistent with what ACCOR is
doing. But it would be good to isolate that first,
because then that tells us we need to go find how the
autocorrelation amplitudes could possibly be different with a
simple flip.</div>
<div><br>
</div>
<div>The one thing I could think of is that there is a large DC
spike and it has been zeroed out in the zoom band case
(because the DC channel became the upper edge channel and gets
forced to zero) and that has affected the band-average
autocorrelation amplitude. POSSUM spectra of the raw
amplitudes should confirm one way or the other.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Adam</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 26 Nov 2019 at 02:13,
Jan Wagner via Difx-users <<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nrao.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Noted
one strange issue in DiFX 2.6.1. Has anyone noticed this and<br>
can confirm it?<br>
<br>
The scaling of DiFX data converted with difx2fits and loaded
into<br>
AIPS appears to be different when using zoom band correlation,<br>
versus correlating natively recorded bands:<br>
<br>
<br>
Case: Stations recoded 32 MHz LSB at 86348.00 MHz. Two
correlation<br>
runs with DiFX 2.6.1.<br>
<br>
Correlation (1): 32 MHz LSB recorded bands flipped to USB via<br>
addZoomFreq <a href="mailto:freq@86348.00/bw@32.0/noparent@true" target="_blank">freq@86348.00/bw@32.0/noparent@true</a><br>
<br>
Correlation (2): 32 MHz LSB recorded bands correlated as-is<br>
<br>
<br>
Processing: difx2fits, AIPS FITLD digicor=1, ACCOR
solint=2/60,<br>
CLCAL #2 = CL#1 + SN#Accor, POSSM cross power spectra
docalib=1<br>
gainuse=CL#2.<br>
<br>
<br>
Result: compared "flux density" on three baselines and 4 IFs<br>
<br>
Data from zoom correlation (1): amplitudes ~0.5 Jy, accor
gains<br>
~0.65 or thereabouts<br>
<br>
Data from rec band correlation (2): amplitudes ~1.0 Jy, accor
gains<br>
~1.00<br>
<br>
<br>
It is strange that 32 MHz zoom (identical to 32 MHz recoderd
except<br>
for sideband flip to USB) would produce a factor 2 difference
in the<br>
scaling of cross power spectra. Has anyone noticed this? Is
there a<br>
workaround? The experiments in question were already
correlated some<br>
months ago and not all raw data are available anymore. In case<br>
someone has already ran into this issue, is simply a "manual"<br>
rescaling by factor 2 in AIPS a permissible solution?<br>
<br>
regards,<br>
Jan<br>
<br>
<br>
_______________________________________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">!=============================================================!<br>
<div dir="ltr" style="font-size:12.8px">A/Prof.
Adam Deller </div>
<div dir="ltr" style="font-size:12.8px">ARC
Future Fellow</div>
</div>
<div style="font-size:12.8px">Centre
for Astrophysics &
Supercomputing </div>
<div dir="ltr" style="font-size:12.8px">Swinburne
University of Technology <br>
John St, Hawthorn VIC 3122 Australia</div>
<div style="font-size:12.8px">phone:
+61 3 9214 5307</div>
<div style="font-size:12.8px">fax: +61
3 9214 8797</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">office
days (usually): Mon-Thu<br>
!=============================================================!</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px">!=============================================================!<br><div dir="ltr" style="font-size:12.8px">A/Prof. Adam Deller </div><div dir="ltr" style="font-size:12.8px">ARC Future Fellow</div></div><div style="font-size:12.8px">Centre for Astrophysics & Supercomputing </div><div dir="ltr" style="font-size:12.8px">Swinburne University of Technology <br>John St, Hawthorn VIC 3122 Australia</div><div style="font-size:12.8px">phone: +61 3 9214 5307</div><div style="font-size:12.8px">fax: +61 3 9214 8797</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">office days (usually): Mon-Thu<br>!=============================================================!</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>