<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Adam,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I'm wanting to get difx running on VDIF files to correlate e-merlin data (I'm at Jodrell Bank). Ultimately, we want to correlate streaming data, not files. It turns out it's not that easy to get a vex file I can look at, we don't create them here so we have
to dig out appropriate 3<sup>rd</sup> party ones that I'm allowed to look at. It is happening though.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The VDIF files are not from an observation, they contain continuous wave data generated for testing. They are valid though. But the error that I'm getting is from vex2difx and the contents of the files haven't been inspected at that point, so it is something
about the vex file.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I've got difx running on the example data set so I can test it and know it is working on LBA files. Then I swap in the VDIF and focus on any problems there.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hugh</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: "Courier New", monospace; font-size: 8pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt; margin-left: 0px; font-family: "Courier New", monospace; font-size: 8pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Adam Deller <adeller@astro.swin.edu.au><br>
<b>Sent:</b> 24 June 2024 05:32<br>
<b>To:</b> Hugh Garsden <hugh.garsden@manchester.ac.uk><br>
<b>Cc:</b> difx-users@listmgr.nrao.edu <difx-users@listmgr.nrao.edu><br>
<b>Subject:</b> Re: [Difx-users] Getting difx working on VDIF files - newbie {External}</span>
<div> </div>
</div>
<div style="direction: ltr;">Hi Hugh,</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">Can I ask the context in which you want to make this substitution? I.e., where did you get the VDIF files, what observation are they from, etc? Generally I usually find it easier to provide sched with a key file that describes
the observation correctly, and have sched produce a vex file that describes the observation correctly, and then just use that - rather than hacking vex files, which is error prone.</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">That said, the information on fanout on this page may be relevant:
<a href="https://urldefense.com/v3/__https://www.atnf.csiro.au/people/Tasso.Tzioumis/vlbi/dokuwiki/doku.php/difx/fanoutdef__;!!PDiH4ENfjr2_Jw!CA9d72uf5DwNrs3sWh1NSvo6atm8ii-MoT1I24s3FFRS4tOHC1I8p1MvVp5AcjOpMFDT8NobOEFSW0e3vhMaFBFTEVx-b9Kazw$" id="OWA670e6fbb-ac32-360b-1653-bff00700ff6c" class="OWAAutoLink" data-auth="NotApplicable">
https://www.atnf.csiro.au/people/Tasso.Tzioumis/vlbi/dokuwiki/doku.php/difx/fanoutdef [atnf.csiro.au]</a>.</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">Cheers,</div>
<div style="direction: ltr;">Adam</div>
<br>
<div style="direction: ltr;">On Sat, 22 Jun 2024 at 02:47, Hugh Garsden via Difx-users <<a href="mailto:difx-users@listmgr.nrao.edu" id="OWAcce2bf7d-0013-1afc-cd95-41d31073eb5c" class="OWAAutoLink">difx-users@listmgr.nrao.edu</a>> wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 1px solid rgb(204, 204, 204);">
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello all,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I'm new to difx and I want to run it on VDIF files. I've seen some recent mails about VDIF files so they are obviously being used, but I was looking at the
<a href="https://urldefense.com/v3/__https://github.com/difx/difx/wiki/vex2difx*vdif-issues__;Iw!!PDiH4ENfjr2_Jw!CA9d72uf5DwNrs3sWh1NSvo6atm8ii-MoT1I24s3FFRS4tOHC1I8p1MvVp5AcjOpMFDT8NobOEFSW0e3vhMaFBFTEVy0RbDJrw$" id="x_m_7919744823270802123LPlnk127304" class="OWAAutoLink" title="https://github.com/difx/difx/wiki/vex2difx#vdif-issues" data-auth="NotApplicable"><font color="red"><b>MailScanner has detected a possible fraud attempt from "urldefense.com" claiming to be</b></font>
manual [github.com]</a> for VDIF information it seems to be incomplete: "TO BE ADDED". I wondered if I could get more info or example files or something.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Here is a much longer story of what I'm up to:</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have difx running on the example files in the vf252 dataset <a href="https://urldefense.com/v3/__https://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/datasets__;!!PDiH4ENfjr2_Jw!CA9d72uf5DwNrs3sWh1NSvo6atm8ii-MoT1I24s3FFRS4tOHC1I8p1MvVp5AcjOpMFDT8NobOEFSW0e3vhMaFBFTEVyf3Ry4MA$" id="x_m_7919744823270802123LPlnk167774" class="OWAAutoLink" title="https://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/datasets" data-auth="NotApplicable">
here [atnf.csiro.au]</a>. I stripped down the vex/v2d files to correlate only one file each of the PARKES and ATCA data. That works. So for PARKES I have this filelist:</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
/home/jbcaadmin/work/v252f/v252f_Pk_161_012830.lba</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
and for ATCA this filelist:</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
/home/jbcaadmin/work/v252f/v252f_At_161_012830.lba</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
and the vex/v2d files contain the minimum neccessary to make everything work.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Now I want to replace the above files with two VDIF files, and make the necessary changes to the vex/v2d files to run difx. There will be changes to timing, sampling rate etc., to accomodate the new files, but the VDIF files don't contain all the information
necessary for vex/v2d files, so I'm going to "pretend" that it some of it is the same as for the LBA files above. (The VDIF files are just test files and not actually from an observation, but they are valid.)</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I made all the changes that I think are necessary, but vex2difx gives me this error:</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Warning: job y_1 not written since it correlates no data</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This is often due to media not being specified, all frequency Ids being</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
unselected, or too many antennas being explicitly unselected by time.</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
It is also possible that the vex file is faulty and missing, e.g., an $IF</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
section, leading to missing polarisation information.</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
nBaseline=0 minSubarraySize=2</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I think this is something to do with the $TRACKS block in the vex file. I have this in the $TRACKS:</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
$TRACKS;</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
*</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
def LBA_.4Ch2bit1to1;</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
track_frame_format = VDIF/8032/4;</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
enddef;</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
but from the manual I think more is needed, because it says:</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As a concrete example, a complete $TRACKS section may resemble:</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; margin-left: 40px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
TOBEADDED</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
so I'm wondering what I'm missing. I tried adding some fanout_def statements, based on there being a single 4 bit data stream in the VDIF. Then vex2difx was happy, and vex2difx and calcif2 ran successfully, and the correlator difx started ok, but seemed to
be quite confused about the input data streams. Also the manual says that fanout_def is not used for VDIF. </div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I've attached vex/v2d files of the set that works (LBA files) and the set that doesn't work (VDIF files).</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hugh</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
_______________________________________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" id="OWAc8672d6d-364b-3422-bb74-f94c89e6fd6c" class="OWAAutoLink">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://urldefense.com/v3/__https://listmgr.nrao.edu/mailman/listinfo/difx-users__;!!PDiH4ENfjr2_Jw!CA9d72uf5DwNrs3sWh1NSvo6atm8ii-MoT1I24s3FFRS4tOHC1I8p1MvVp5AcjOpMFDT8NobOEFSW0e3vhMaFBFTEVzuT9M5Mg$" id="OWAb01d5a89-001f-221a-6799-3985bbfa6346" class="OWAAutoLink" data-auth="NotApplicable">https://listmgr.nrao.edu/mailman/listinfo/difx-users
[listmgr.nrao.edu]</a><br>
</blockquote>
<br>
<br>
--<br>
<div style="direction: ltr; font-size: 12.8px;">!=============================================================!</div>
<div style="direction: ltr; font-size: 12.8px;">Prof. Adam Deller </div>
<div style="direction: ltr; font-size: 12.8px;">Centre for Astrophysics & Supercomputing </div>
<div style="direction: ltr; font-size: 12.8px;">Swinburne University of Technology <br>
John St, Hawthorn VIC 3122 Australia</div>
<div style="direction: ltr; font-size: 12.8px;">phone: +61 3 9214 5307</div>
<div style="direction: ltr; font-size: 12.8px;">fax: +61 3 9214 8797</div>
<div style="direction: ltr; font-size: 12.8px;">!=============================================================!</div>
</body>
</html>