<div dir="ltr">Hi Joe,<div><br></div><div>When you say "I still only seem to get results that look right if I maintain sampling rate = bandwidth in the vex file", I assume that you're running with your vexload patch then, and with the vexload patch taken away when using the other approach. What "should" work (I think) is no patch, bandwidth set to the correct value, sample_rate set to the incorrect (2x bandwidth) value, and sampling = complex in the v2d file. </div><div><br></div><div>Can you send the test_1.input file that is generated in each case? (Noting what if any hand-edits you'd made). The input file is designed to be human-readable (albeit a bit clunky) so diff'ing them should immediately make clear the difference that is resulting between the two cases.</div><div><br></div><div>Cheers,</div><div>Adam</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 17 Jun 2021 at 05:40, Joe Skeens <<a href="mailto:jskeens1@utexas.edu">jskeens1@utexas.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Adam and Chris,<div><br></div><div>Thanks a lot for the pointers. I added the sampling=COMPLEX line to the .v2d file, and I think I have the correct form in the $TRACKS section, but I still only seem to get results that look right if I maintain sampling rate = bandwidth in the vex file. I've attached amplitude vs. frequency plots for a 3 MHz tone injected in phase for bandwidth = sample rate (BW_1.png) and bandwidth = sample rate/2 (BW_2.png) in the vex file. For clarity, the left plot is the amplitude of the highest amplitude frequency bin as a function of time, and the bottom plot is an average of all integration periods. 10 seconds of data were used with a low level of uncorrelated noise. If you're curious, I've also attached the .v2d and .vex files I used to generate these plots in DiFX. Only a few select lines have been changed from the synthetic tests (Chris is even still listed as PI, haha). Let me know if these files don't come through and you'd like to see them--I'm not sure what filters this email might pass through.</div><div><br></div><div>Best,</div><div>Joe</div><div><br></div><div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 14, 2021 at 7:49 PM Phillips, Chris (S&A, Marsfield) <Chris.Phillips@csiro.au> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
Hi Joe, Adam
<div><br>
</div>
<div>I cannot remember if the handing of the sample rate as described is just something I overlooked or a more fundamental issue. It may well be just something I overlooked.</div>
<div><br>
</div>
<div>Internally mpifxcorr certainly uses the “correct” sampling rate for complex data. It would have been easier to just lie about the sampling rate entirely (less code to tweak) but I felt it would be something that would cause subtle bugs in the
future if not handled correctly.</div>
<div><br>
</div>
<div>For the vex file, I just use the sampling rate of double the bandwidth. Vex2difx definitely DOES handle data labelled as complex on the vex file, but this is probably just using my own non-standard and un-documented extension to vex 1.5. </div>
<div><br>
</div>
<div>If you have a .vex file with the station schedules as real sampled Mark5B (thats a typical. Initial setup), then there is a perl script “addVDIF.pl” which comes with DIFX which will changes the setup to VDIF, and has a “complex” option. Note addVDIF.pl
depends on Astro::Vex, which comes with DIFX but can be a pain to install because of possible conflicts between vex and perl Lex parser. The —perl option option on the install-diff script should install all of this.</div>
<div><br>
</div>
<div>addVDIF has the following options:</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">
<span style="font-variant-ligatures:no-common-ligatures"> -bits <N> Set bit depth to N (default 2)</span></div>
<div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">
<span style="font-variant-ligatures:no-common-ligatures"> -framesize <N> Set frame size to N (default 8032)</span></div>
<div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">
<span style="font-variant-ligatures:no-common-ligatures"> -complex Complex samples</span></div>
</div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div>So you can set appropriate #bits and label as complex. </div>
<div><br>
</div>
<div>
<div>The main this this does is set the track format in $TRACK section:</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">
<span style="font-variant-ligatures:no-common-ligatures"> track_frame_format =
</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(255,255,255);background-color:rgb(0,0,0)">VDIF</span><span style="font-variant-ligatures:no-common-ligatures">C/8032/2;</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
</div>
<div>VDIFC is complex VDIF data in “Single Sideband” mode. If you have “Double Sideband” mode, you need VDIFD (from memory). addVDIF.pl does not support that, but it would be a trivial change.</div>
<div><br>
</div>
<div><br>
</div>
<div>So essentially, if you just leave sample rate a 2xbandwidth and use the above track_frame_format in the vex file, it should “just work”.</div>
<div><br>
</div>
<div>Note you need trunk DIFX for all combinations of complex, USB/LSB, single/double sideband to work. Lots are broken in current release.</div>
<div><br>
</div>
<div>We probably should re-assess the notes on the wiki and update. For example notes on VDIF looks out of date and are buried near the bottom of the Documentation page after a lot of much less important stuff.</div>
<div><br>
</div>
<div>Cheers</div>
<div>Chris</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<blockquote type="cite">
<div>On Jun 15, 2021, at 8:40 AM, Adam Deller via Difx-users <<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nrao.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">Hi Joe,
<div><br>
</div>
<div>Right, I see. So if you leave the code and vex file untouched and just change the ANTENNA entries in the v2d file (add sampling=COMPLEX inside the ANTENNA blocks), I think you should be ok. What is the data format? Does that match the example,
or do you need to override that too?</div>
<div><br>
</div>
<div>Cheers,<br>
Adam</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 15 Jun 2021 at 00:21, Joe Skeens <<a href="mailto:jskeens1@utexas.edu" target="_blank">jskeens1@utexas.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi Adam,
<div><br>
</div>
<div>Thanks for the help and the quick response! We've been using an adapted version of the files given in the pre-packaged tests that come with DiFX to establish a proof-of-concept run with simulated complex data. That is to say, I'm changing the
premade .vex file by hand to match the properties of the simulated signals I'm feeding into the correlator. Prior to removing the if statement I mentioned, I was giving a bandwidth of sample rate divide by 2 in the .vex file, running vex2difx, then manually
changing the bandwidth to the sample rate in the .input file before running mpifxcorr. I'm not sure if that leads to a problem somewhere in the guts of DiFX, but we've been able to see injected tones in the output FITS file using this method.</div>
<div><br>
</div>
<div>Best,</div>
<div>Joe</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, Jun 12, 2021 at 4:56 AM Adam Deller <<a href="mailto:adeller@astro.swin.edu.au" target="_blank">adeller@astro.swin.edu.au</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi Joe,
<div><br>
</div>
<div>I don't believe the current vex standard is actually capable of representing that a signal is complex sampled (a problem which is one of the justifications for the long-awaited vex2 standard). To sidestep this, the v2d file can state that the
data sampling is complex - but the vex loader doesn't have access to that information. So the vex file basically can't represent the full info, and it has to be wrong or at least incomplete in one way or another. So if you just change the sample rate in
the vex file (but still tell vex2difx via the v2d file that the sampling is complex) you should be ok.</div>
<div><br>
</div>
<div>Out of interest, how did you get this vex file (with the sampling rate set to the bandwidth) in the first place? <br>
</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Adam</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, 12 Jun 2021 at 03:22, Joe Skeens via Difx-users <<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nrao.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hello all,<br>
<br>
I've been working with a couple of collaborators on running DiFX with<br>
complex signals, and we've found something that looks like it may be an<br>
oversight. In the vex2difx file 'vexload.cpp,' there's a simple if<br>
statement check which seems to ensure the sample rate is above the Nyquist frequency:<br>
<br>
if(bandwidth - stream.sampRate/2 > 1e-6)<br>
<br>
This causes vex2difx to exit on an error condition for bandwidth greater<br>
than sample rate divided by 2, but for a complex signal, the signal is<br>
only undersampled for bandwidth greater than sample rate. Commenting out<br>
this check and rebuilding DiFX seems to correct the problem. Is there<br>
something I'm missing with respect to how the bandwidth is processed in<br>
the channel definitions under $FREQ in the .vex file, or has this check<br>
been designed with only real signals in mind? Thanks in advance for any<br>
insights.<br>
<div><br>
</div>
<div>Best,</div>
<div>Joe Skeens</div>
</div>
_______________________________________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">!=============================================================!<br>
<div dir="ltr" style="font-size:12.8px">A/Prof. Adam Deller </div>
<div dir="ltr" style="font-size:12.8px">ARC Future Fellow</div>
</div>
<div style="font-size:12.8px">Centre for Astrophysics & Supercomputing </div>
<div dir="ltr" style="font-size:12.8px">Swinburne University of Technology <br>
John St, Hawthorn VIC 3122 Australia</div>
<div style="font-size:12.8px">phone: +61 3 9214 5307</div>
<div style="font-size:12.8px">fax: +61 3 9214 8797</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">office days (usually): Mon-Thu<br>
!=============================================================!</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">!=============================================================!<br>
<div dir="ltr" style="font-size:12.8px">A/Prof. Adam Deller </div>
<div dir="ltr" style="font-size:12.8px">ARC Future Fellow</div>
</div>
<div style="font-size:12.8px">Centre for Astrophysics & Supercomputing </div>
<div dir="ltr" style="font-size:12.8px">Swinburne University of Technology <br>
John St, Hawthorn VIC 3122 Australia</div>
<div style="font-size:12.8px">phone: +61 3 9214 5307</div>
<div style="font-size:12.8px">fax: +61 3 9214 8797</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">office days (usually): Mon-Thu<br>
!=============================================================!</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px">!=============================================================!<br><div dir="ltr" style="font-size:12.8px">A/Prof. Adam Deller </div><div dir="ltr" style="font-size:12.8px">ARC Future Fellow</div></div><div style="font-size:12.8px">Centre for Astrophysics & Supercomputing </div><div dir="ltr" style="font-size:12.8px">Swinburne University of Technology <br>John St, Hawthorn VIC 3122 Australia</div><div style="font-size:12.8px">phone: +61 3 9214 5307</div><div style="font-size:12.8px">fax: +61 3 9214 8797</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">office days (usually): Mon-Thu<br>!=============================================================!</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>