[Difx-users] Getting difx working on VDIF files - newbie {External} {External}

[Hugh Garsden]

Hi Adam,

I'm wanting to get difx running on VDIF files to correlate e-merlin data (I'm at Jodrell Bank). Ultimately, we want to correlate streaming data, not files. It turns out it's not that easy to get a vex file I can look at, we don't create them here so we have to dig out appropriate 3rd party ones that I'm allowed to look at. It is happening though.

The VDIF files are not from an observation, they contain continuous wave data generated for testing.  They are valid though. But the error that I'm getting is from vex2difx and the contents of the files haven't been inspected at that point, so it is something about the vex file.

I've got difx running on the example data set so I can test it and know it is working on LBA files. Then I swap in the VDIF and focus on any problems there.

Thanks,

    Hugh




________________________________
From: Adam Deller <adeller at astro.swin.edu.au>
Sent: 24 June 2024 05:32
To: Hugh Garsden <hugh.garsden at manchester.ac.uk>
Cc: difx-users at listmgr.nrao.edu <difx-users at listmgr.nrao.edu>
Subject: Re: [Difx-users] Getting difx working on VDIF files - newbie {External}

Hi Hugh,

Can I ask the context in which you want to make this substitution? I.e., where did you get the VDIF files, what observation are they from, etc?  Generally I usually find it easier to provide sched with a key file that describes the observation correctly, and have sched produce a vex file that describes the observation correctly, and then just use that - rather than hacking vex files, which is error prone.

That said, the information on fanout on this page may be relevant: https://www.atnf.csiro.au/people/Tasso.Tzioumis/vlbi/dokuwiki/doku.php/difx/fanoutdef [atnf.csiro.au]<https://urldefense.com/v3/__https://www.atnf.csiro.au/people/Tasso.Tzioumis/vlbi/dokuwiki/doku.php/difx/fanoutdef__;!!PDiH4ENfjr2_Jw!CA9d72uf5DwNrs3sWh1NSvo6atm8ii-MoT1I24s3FFRS4tOHC1I8p1MvVp5AcjOpMFDT8NobOEFSW0e3vhMaFBFTEVx-b9Kazw$>.

Cheers,
Adam

On Sat, 22 Jun 2024 at 02:47, Hugh Garsden via Difx-users <difx-users at listmgr.nrao.edu<mailto:difx-users at listmgr.nrao.edu>> wrote:
Hello all,

I'm new to difx and I want to run it on VDIF files. I've seen some recent mails about VDIF files so they are obviously being used, but I was looking at the manual [github.com]<https://urldefense.com/v3/__https://github.com/difx/difx/wiki/vex2difx*vdif-issues__;Iw!!PDiH4ENfjr2_Jw!CA9d72uf5DwNrs3sWh1NSvo6atm8ii-MoT1I24s3FFRS4tOHC1I8p1MvVp5AcjOpMFDT8NobOEFSW0e3vhMaFBFTEVy0RbDJrw$> for VDIF information it seems to be incomplete: "TO BE ADDED". I wondered if I could get more info or example files or something.

Here is a much longer story of what I'm up to:

I have difx running on the example files in the vf252 dataset here [atnf.csiro.au]<https://urldefense.com/v3/__https://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/datasets__;!!PDiH4ENfjr2_Jw!CA9d72uf5DwNrs3sWh1NSvo6atm8ii-MoT1I24s3FFRS4tOHC1I8p1MvVp5AcjOpMFDT8NobOEFSW0e3vhMaFBFTEVyf3Ry4MA$>. I stripped down the vex/v2d files to correlate only one file each of the PARKES and ATCA data. That works. So for PARKES I have this filelist:

    /home/jbcaadmin/work/v252f/v252f_Pk_161_012830.lba

and for ATCA this filelist:

    /home/jbcaadmin/work/v252f/v252f_At_161_012830.lba

and the vex/v2d files contain the minimum neccessary to make everything work.

Now I want to replace the above files with two VDIF files, and make the necessary changes to the vex/v2d files to run difx. There will be changes to timing, sampling rate etc., to accomodate the new files, but the VDIF files don't contain all the information necessary for  vex/v2d files, so I'm going to "pretend" that it some of it is the same as for the LBA files above. (The VDIF files are just test files and not actually from an observation, but they are valid.)

I made all the changes that I think are necessary, but vex2difx gives me this error:

Warning: job y_1 not written since it correlates no data
This is often due to media not being specified, all frequency Ids being
unselected, or too many antennas being explicitly unselected by time.
It is also possible that the vex file is faulty and missing, e.g., an $IF
section, leading to missing polarisation information.
nBaseline=0  minSubarraySize=2

I think this is something to do with the $TRACKS block in the vex file. I have this in the $TRACKS:

$TRACKS;
*
def LBA_.4Ch2bit1to1;
     track_frame_format = VDIF/8032/4;
enddef;

but from the manual I think more is needed, because it says:

As a concrete example, a complete $TRACKS section may resemble:

TOBEADDED

so I'm wondering what I'm missing. I tried adding some fanout_def statements, based on there being a single 4 bit data stream in the VDIF. Then vex2difx was happy, and vex2difx and calcif2 ran successfully, and the correlator difx started ok, but seemed to be quite confused about the input data streams. Also the manual says that fanout_def is not used for VDIF.

I've attached vex/v2d files of the set that works (LBA files) and the set that doesn't work (VDIF files).


Thanks,


               Hugh

_______________________________________________
Difx-users mailing list
Difx-users at listmgr.nrao.edu<mailto:Difx-users at listmgr.nrao.edu>
https://listmgr.nrao.edu/mailman/listinfo/difx-users [listmgr.nrao.edu]<https://urldefense.com/v3/__https://listmgr.nrao.edu/mailman/listinfo/difx-users__;!!PDiH4ENfjr2_Jw!CA9d72uf5DwNrs3sWh1NSvo6atm8ii-MoT1I24s3FFRS4tOHC1I8p1MvVp5AcjOpMFDT8NobOEFSW0e3vhMaFBFTEVzuT9M5Mg$>


--
!=============================================================!
Prof. Adam Deller
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20240625/bc119af8/attachment-0001.html>