[Difx-users] vex2difx bandwidth-sample rate check

[Adam Deller]

Hi Joe,

When you say "I still only seem to get results that look right if I
maintain sampling rate = bandwidth in the vex file", I assume that you're
running with your vexload patch then, and with the vexload patch taken away
when using the other approach. What "should" work (I think) is no patch,
bandwidth set to the correct value, sample_rate set to the incorrect (2x
bandwidth) value, and sampling = complex in the v2d file.

Can you send the test_1.input file that is generated in each case?  (Noting
what if any hand-edits you'd made). The input file is designed to be
human-readable (albeit a bit clunky) so diff'ing them should immediately
make clear the difference that is resulting between the two cases.

Cheers,
Adam

On Thu, 17 Jun 2021 at 05:40, Joe Skeens <jskeens1 at utexas.edu> wrote:

> Hi Adam and Chris,
>
> Thanks a lot for the pointers. I added the sampling=COMPLEX line to the
> .v2d file, and I think I have the correct form in the $TRACKS section, but
> I still only seem to get results that look right if I maintain sampling
> rate = bandwidth in the vex file. I've attached amplitude vs. frequency
> plots for a 3 MHz tone injected in phase for bandwidth = sample rate
> (BW_1.png) and bandwidth = sample rate/2 (BW_2.png) in the vex file.  For
> clarity, the left plot is the amplitude of the highest amplitude frequency
> bin as a function of time, and the bottom plot is an average of all
> integration periods. 10 seconds of data were used with a low level of
> uncorrelated noise. If you're curious, I've also attached the .v2d and .vex
> files I used to generate these plots in DiFX. Only a few select lines have
> been changed from the synthetic tests (Chris is even still listed as PI,
> haha). Let me know if these files don't come through and you'd like to see
> them--I'm not sure what filters this email might pass through.
>
> Best,
> Joe
>
>
>
> On Mon, Jun 14, 2021 at 7:49 PM Phillips, Chris (S&A, Marsfield)
> <Chris.Phillips at csiro.au> wrote:
>
>> Hi Joe, Adam
>>
>> I cannot remember if the handing of the sample rate as described is just
>> something I overlooked or a more fundamental issue. It may well be just
>> something I overlooked.
>>
>> Internally mpifxcorr certainly uses the “correct” sampling rate for
>> complex data. It would have been easier to just lie about the sampling rate
>> entirely (less code to tweak)  but I felt it would be something that would
>> cause subtle bugs in the future if not handled correctly.
>>
>> For the vex file, I just use the sampling rate of double the bandwidth.
>> Vex2difx definitely DOES handle data labelled as complex on the vex file,
>> but this is probably just using my own non-standard and un-documented
>> extension to vex 1.5.
>>
>> If you have a .vex file with the station schedules as real sampled Mark5B
>> (thats a typical. Initial setup), then there is a perl script “addVDIF.pl”
>> which comes with DIFX which will changes the setup to VDIF, and has a
>> “complex” option. Note addVDIF.pl depends on Astro::Vex, which comes with
>> DIFX but can be a pain to install because of possible conflicts between vex
>> and perl Lex parser.  The —perl option option on the install-diff script
>> should install all of this.
>>
>> addVDIF has the following options:
>>
>>
>>  -bits <N>       Set bit depth to N (default 2)
>>  -framesize <N>  Set frame size to N (default 8032)
>>  -complex        Complex samples
>>
>> So you can set appropriate #bits and label as complex.
>>
>> The main this this does is set  the track format in $TRACK section:
>>
>>   track_frame_format = VDIFC/8032/2;
>>
>> VDIFC is complex VDIF data in “Single Sideband” mode. If you have “Double
>> Sideband” mode, you need VDIFD (from memory). addVDIF.pl does not support
>> that, but it would be a trivial change.
>>
>>
>> So essentially, if you just leave sample rate a 2xbandwidth and use the
>> above track_frame_format in the vex file, it should “just work”.
>>
>> Note you need trunk DIFX for all combinations of complex, USB/LSB,
>> single/double sideband to work. Lots are broken in current release.
>>
>> We probably should re-assess the notes on the wiki and update. For
>> example notes on VDIF looks out of date and are buried near the bottom of
>> the Documentation page after a lot of much less important stuff.
>>
>> Cheers
>> Chris
>>
>>
>>
>> On Jun 15, 2021, at 8:40 AM, Adam Deller via Difx-users <
>> difx-users at listmgr.nrao.edu> wrote:
>>
>> Hi Joe,
>>
>> Right, I see.  So if you leave the code and vex file untouched and just
>> change the ANTENNA entries in the v2d file (add sampling=COMPLEX inside the
>> ANTENNA blocks), I think you should be ok.  What is the data format?  Does
>> that match the example, or do you need to override that too?
>>
>> Cheers,
>> Adam
>>
>> On Tue, 15 Jun 2021 at 00:21, Joe Skeens <jskeens1 at utexas.edu> wrote:
>>
>>> Hi Adam,
>>>
>>> Thanks for the help and the quick response! We've been using an adapted
>>> version of the files given in the pre-packaged tests that come with DiFX to
>>> establish a proof-of-concept run with simulated complex data. That is to
>>> say, I'm changing the premade .vex file by hand to match the properties of
>>> the simulated signals I'm feeding into the correlator. Prior to removing
>>> the if statement I mentioned, I was giving a bandwidth of sample rate
>>> divide by 2 in the .vex file, running vex2difx, then manually changing the
>>> bandwidth to the sample rate in the .input file before running mpifxcorr.
>>> I'm not sure if that leads to a problem somewhere in the guts of DiFX, but
>>> we've been able to see injected tones in the output FITS file using this
>>> method.
>>>
>>> Best,
>>> Joe
>>>
>>> On Sat, Jun 12, 2021 at 4:56 AM Adam Deller <adeller at astro.swin.edu.au>
>>> wrote:
>>>
>>>> Hi Joe,
>>>>
>>>> I don't believe the current vex standard is actually capable of
>>>> representing that a signal is complex sampled (a problem which is one of
>>>> the justifications for the long-awaited vex2 standard).  To sidestep this,
>>>> the v2d file can state that the data sampling is complex - but the vex
>>>> loader doesn't have access to that information.  So the vex file basically
>>>> can't represent the full info, and it has to be wrong or at least
>>>> incomplete in one way or another.  So if you just change the sample rate in
>>>> the vex file (but still tell vex2difx via the v2d file that the sampling is
>>>> complex) you should be ok.
>>>>
>>>> Out of interest, how did you get this vex file (with the sampling rate
>>>> set to the bandwidth) in the first place?
>>>>
>>>> Cheers,
>>>> Adam
>>>>
>>>> On Sat, 12 Jun 2021 at 03:22, Joe Skeens via Difx-users <
>>>> difx-users at listmgr.nrao.edu> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> I've been working with a couple of collaborators on running DiFX with
>>>>> complex signals, and we've found something that looks like it may be an
>>>>> oversight. In the vex2difx file 'vexload.cpp,' there's a simple if
>>>>> statement check which seems to ensure the sample rate is above the
>>>>> Nyquist frequency:
>>>>>
>>>>> if(bandwidth - stream.sampRate/2 > 1e-6)
>>>>>
>>>>> This causes vex2difx to exit on an error condition for bandwidth
>>>>> greater
>>>>> than sample rate divided by 2, but for a complex signal, the signal is
>>>>> only undersampled for bandwidth greater than sample rate. Commenting
>>>>> out
>>>>> this check and rebuilding DiFX seems to correct the problem. Is there
>>>>> something I'm missing with respect to how the bandwidth is processed in
>>>>> the channel definitions under $FREQ in the .vex file, or has this check
>>>>> been designed with only real signals in mind? Thanks in advance for any
>>>>> insights.
>>>>>
>>>>> Best,
>>>>> Joe Skeens
>>>>> _______________________________________________
>>>>> Difx-users mailing list
>>>>> Difx-users at listmgr.nrao.edu
>>>>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>>>>>
>>>>
>>>>
>>>> --
>>>> !=============================================================!
>>>> A/Prof. Adam Deller
>>>> ARC Future Fellow
>>>> Centre for Astrophysics & Supercomputing
>>>> Swinburne University of Technology
>>>> John St, Hawthorn VIC 3122 Australia
>>>> phone: +61 3 9214 5307
>>>> fax: +61 3 9214 8797
>>>>
>>>> office days (usually): Mon-Thu
>>>> !=============================================================!
>>>>
>>>
>>
>> --
>> !=============================================================!
>> A/Prof. Adam Deller
>> ARC Future Fellow
>> Centre for Astrophysics & Supercomputing
>> Swinburne University of Technology
>> John St, Hawthorn VIC 3122 Australia
>> phone: +61 3 9214 5307
>> fax: +61 3 9214 8797
>>
>> office days (usually): Mon-Thu
>> !=============================================================!
>> _______________________________________________
>> Difx-users mailing list
>> Difx-users at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>>
>>
>>

-- 
!=============================================================!
A/Prof. Adam Deller
ARC Future Fellow
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797

office days (usually): Mon-Thu
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20210617/39b4e890/attachment-0001.html>