<div dir="ltr">That would be great!</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 11 Dec 2019 at 00:43, Walter Brisken via Difx-users <<a href="mailto:difx-users@listmgr.nrao.edu">difx-users@listmgr.nrao.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Probably best to find or generate a test dataset for this. I had offered <br>
to arrange for collection of a test data set for output bands anyway. We <br>
could add a couple scans that could verify this as well.<br>
<br>
-W<br>
<br>
<br>
<br>
-------------------------<br>
Walter Brisken<br>
NRAO<br>
Deputy Assistant Director for VLBA Development<br>
(575)-835-7133 (office)<br>
(505)-234-5912 (cell)<br>
<br>
On Tue, 10 Dec 2019, Adam Deller via Difx-users wrote:<br>
<br>
> hi Jan,<br>
> First off, nice sleuthing to track down the proximate cause. <br>
> <br>
> Second, I agree that it is pretty weird that the mere presence of a flipped-sideband-equivalent autocorrelation<br>
> makes ACCOR lose its marbles. Makes me wonder what manner of duct tape is holding things together under the<br>
> hood... Are the contents of the FQ tables identical in both cases? It would be good to then also check with e.g.<br>
> UVPRT which IFs the visibilities are ultimately being linked against. But since the bug no longer gets exercised<br>
> once DiFX is corrected, perhaps you don't want to spend more time chasing this.<br>
> <br>
> Third, I can shed some light on equivfrequsedbybaseline. This was introduced in 2013 to solve a problem unrelated<br>
> to zoom bands: pulse cal. In some HSA observations, the VLA had an identical frequency setup to the VLBA, but with<br>
> no pulse cal. So the VLA frequencies pointed to different entries in the FQ table, even through they were the same<br>
> bandwidth, sideband, etc. Since the VLA FQ entries were not used by any baseline, the VLA autocorrelations were<br>
> getting discarded. Hence, equivfreqused was introduced. But perhaps the test for equivalence should be stricter,<br>
> enforcing that the sideband sense also be the same, rather than just the lower band edge and bandwidth? I guess<br>
> that would solve your problem, and I *think* it would not introduce any new ones. Walter B may want to comment if<br>
> he thinks otherwise!<br>
> <br>
> Cheers,<br>
> Adam<br>
> <br>
> <br>
> <br>
> On Mon, 9 Dec 2019 at 23:43, Jan Wagner <<a href="mailto:jwagner105@googlemail.com" target="_blank">jwagner105@googlemail.com</a>> wrote:<br>
> Hi Adam,<br>
> <br>
> the raw spectra looked consistent (plotDiFX.py, m5spec) and there were no large DC offsets nor large Nyquist<br>
> bin amplitudes.<br>
> <br>
> The ACCOR scaling issue turned out to be related in some way to duplicate autocorrelation entries in difx<br>
> output. DiFX produced autocorrelations for the 32 MHz wide recorded bands (LSB), but in addition also<br>
> produced autocorrelations for the 32 MHz wide zoom bands. In our test case the zooms are just the USB-flipped<br>
> version of the recorded LSB bands. So in a sense the USB zoom & LSB recband autocorrelation records are<br>
> redundant as they sample the same sky frequency range.<br>
> <br>
> Two workarounds were independently successful for ACCOR:<br>
> <br>
> 1 - original vex2difx, correlate, filter afterwards and remove redundant non-zoom autocorrelations, difx2fits<br>
> conversion, FITLD&ACCOR: scaling is correct<br>
> <br>
> 2 - modified vex2difx so baselines reference only the zoom bands, recorrelate, no not filter but just run<br>
> difx2fits, FITLD&ACCOR: scaling is correct<br>
> <br>
> Still not 100% sure what the actual issue is that triggers ACCOR mis-scaling. It was surprising that even<br>
> though DiFX output from (2) still contains redundant autocorrelations, AIPS ACCOR worked fine and there was<br>
> no need for .difx pre-filtering as in (1).<br>
> <br>
> The vex2difx modification was prompted by strange default vex2difx behaviour. Vex2difx chooses zooms for the<br>
> first station on a baseline, but for the second station it prefers recorded bands over zoom bands. And it<br>
> ignores the 'noparent=true' of those latter zooms, that ought to have blocked the parents of those zooms from<br>
> correlation.<br>
> <br>
> To illustrate...<br>
> <br>
> v2d: addZoomFreq = freq@86348.00/bw@32.0/noparent@true<br>
> <br>
> inputfile produced by vex2difx:<br>
> Ys zoom 86348U 32.0 x Ef rec 86380L 32.0<br>
> Ef zoom 86348U 32.0 x Ys rec 86380L 32.0<br>
> On zoom 86348U 32.0 x Ys rec 86380L 32.0<br>
> <br>
> I.e. vex2difx ends up correlating zoom x recorded parent, instead of zoom x zoom. I've added a small change<br>
> to trunk vex2difx (commit r9362). Now addZoomFreq 'noparent=true' gets honoured when looking for a matching<br>
> band at the second station, and the vex2difx result is closer to expected, e.g.,<br>
> <br>
> inputfile produced by updated vex2difx (r9362):<br>
> Ys zoom 86348U 32.0 x Ef zoom 86348U 32.0<br>
> Ef zoom 86348U 32.0 x Ys zoom 86348U 32.0<br>
> On zoom 86348U 32.0 x Ys zoom 86348U 32.0<br>
> <br>
> Like mentioned earlier when I recorrelate with this proper zoom x zoom -only inputfile, the final resultant<br>
> FITS yielded normal corrections in ACCOR. Perhaps difx2fits treats USB zoom x LSB recband somehow differently<br>
> and adjusts the auto spectra, and treats correctly rec x rec or zoom x zoom cases and their auto spectra...<br>
> <br>
> Anyway, our issue is sort of fixed in DiFX trunk now. Or at least worked around...<br>
> <br>
> <br>
> It is curious though that even with zoom x zoom -only baselines the DiFX output still has redundant<br>
> autocorrelations from rec bands, e.g.,<br>
> <br>
> Ef-Ef auto 86348U 32.0 from zoom<br>
> Ef-Ef auto 86380L 32.0 from rec band<br>
> Ys-Ys auto 86348U 32.0 from zoom<br>
> Ys-Ys auto 86380L 32.0 from rec band<br>
> ...<br>
> <br>
> A question is whether those redundant autocorrelations (zoom + recband) should truly be present in the DiFX<br>
> output?<br>
> <br>
> Looking at DiFX configuration.cpp, all frequencies referenced from the BASELINE table are set to<br>
> 'frequsedbybaseline[<fqId>] = true'. This marks them so that autocorrelation spectra will be output. In our<br>
> case those would be zoom band autocorrelations.<br>
> <br>
> However, configuration.cpp adds the concept of "equivalent frequency". All frequencies in the FREQ table that<br>
> are unreferenced by any BASELINE are compared against the frequencies used on the baselines. When an<br>
> unreferenced frequency matches one actually used on a baseline (same sky freq range, same bandwidth) then<br>
> that unreferenced frequency gets set 'equivfrequsedbybaseline[<fqId>] = true'.<br>
> <br>
> In our test case the zoom x zoom baselines (with autocorrelations via 'frequsedbybaseline[<fqId>] = true')<br>
> have matching recorded bands or recorded frequencies that are not producing visibility data, but, they will<br>
> nevertheless be enabled for output of (redundant) autocorrelations via the "equivalent frequency" check.<br>
> <br>
> Does this 'equivfrequsedbybaseline' still have an active legit use?<br>
> <br>
> Or is it maybe a leftover from DiFX 2.5?<br>
> <br>
> Are the auto spectra of frequencies not utilized by any BASELINE still useful somewhere for calibration?<br>
> <br>
> regards,<br>
> Jan<br>
> <br>
> Am 26.11.2019 um 23:35 schrieb Adam Deller:<br>
> Hi Jan,<br>
> What if you plot the raw (uncorrected) autocorrelation and cross correlation amplitudes using POSSM?<br>
> Based on what you've written, I expect it will show that the cross correlation amplitudes are identical<br>
> in the two correlations and the autocorrelation amplitudes are different. This of course should not<br>
> be, but that would be consistent with what ACCOR is doing. But it would be good to isolate that first,<br>
> because then that tells us we need to go find how the autocorrelation amplitudes could possibly be<br>
> different with a simple flip.<br>
> <br>
> The one thing I could think of is that there is a large DC spike and it has been zeroed out in the zoom<br>
> band case (because the DC channel became the upper edge channel and gets forced to zero) and that has<br>
> affected the band-average autocorrelation amplitude. POSSUM spectra of the raw amplitudes should<br>
> confirm one way or the other.<br>
> <br>
> Cheers,<br>
> Adam<br>
> <br>
> On Tue, 26 Nov 2019 at 02:13, Jan Wagner via Difx-users <<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nrao.edu</a>> wrote:<br>
> Noted one strange issue in DiFX 2.6.1. Has anyone noticed this and<br>
> can confirm it?<br>
><br>
> The scaling of DiFX data converted with difx2fits and loaded into<br>
> AIPS appears to be different when using zoom band correlation,<br>
> versus correlating natively recorded bands:<br>
> <br>
><br>
> Case: Stations recoded 32 MHz LSB at 86348.00 MHz. Two correlation<br>
> runs with DiFX 2.6.1.<br>
><br>
> Correlation (1): 32 MHz LSB recorded bands flipped to USB via<br>
> addZoomFreq freq@86348.00/bw@32.0/noparent@true<br>
><br>
> Correlation (2): 32 MHz LSB recorded bands correlated as-is<br>
> <br>
><br>
> Processing: difx2fits, AIPS FITLD digicor=1, ACCOR solint=2/60,<br>
> CLCAL #2 = CL#1 + SN#Accor, POSSM cross power spectra docalib=1<br>
> gainuse=CL#2.<br>
> <br>
><br>
> Result: compared "flux density" on three baselines and 4 IFs<br>
><br>
> Data from zoom correlation (1): amplitudes ~0.5 Jy, accor gains<br>
> ~0.65 or thereabouts<br>
><br>
> Data from rec band correlation (2): amplitudes ~1.0 Jy, accor gains<br>
> ~1.00<br>
> <br>
><br>
> It is strange that 32 MHz zoom (identical to 32 MHz recoderd except<br>
> for sideband flip to USB) would produce a factor 2 difference in the<br>
> scaling of cross power spectra. Has anyone noticed this? Is there a<br>
> workaround? The experiments in question were already correlated some<br>
> months ago and not all raw data are available anymore. In case<br>
> someone has already ran into this issue, is simply a "manual"<br>
> rescaling by factor 2 in AIPS a permissible solution?<br>
><br>
> regards,<br>
> Jan<br>
> <br>
><br>
> _______________________________________________<br>
> Difx-users mailing list<br>
> <a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
> <a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
> <br>
> <br>
> <br>
> --<br>
> !=============================================================!<br>
> A/Prof. Adam Deller <br>
> ARC Future Fellow<br>
> Centre for Astrophysics & Supercomputing <br>
> Swinburne University of Technology <br>
> John St, Hawthorn VIC 3122 Australia<br>
> phone: +61 3 9214 5307<br>
> fax: +61 3 9214 8797<br>
> <br>
> office days (usually): Mon-Thu<br>
> !=============================================================!<br>
> <br>
> <br>
> <br>
> <br>
> --<br>
> !=============================================================!<br>
> A/Prof. Adam Deller <br>
> ARC Future Fellow<br>
> Centre for Astrophysics & Supercomputing <br>
> Swinburne University of Technology <br>
> John St, Hawthorn VIC 3122 Australia<br>
> phone: +61 3 9214 5307<br>
> fax: +61 3 9214 8797<br>
> <br>
> office days (usually): Mon-Thu<br>
> !=============================================================!<br>
> <br>
>_______________________________________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px">!=============================================================!<br><div dir="ltr" style="font-size:12.8px">A/Prof. Adam Deller </div><div dir="ltr" style="font-size:12.8px">ARC Future Fellow</div></div><div style="font-size:12.8px">Centre for Astrophysics & Supercomputing </div><div dir="ltr" style="font-size:12.8px">Swinburne University of Technology <br>John St, Hawthorn VIC 3122 Australia</div><div style="font-size:12.8px">phone: +61 3 9214 5307</div><div style="font-size:12.8px">fax: +61 3 9214 8797</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">office days (usually): Mon-Thu<br>!=============================================================!</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>