[Difx-users] Question : What library/code is DiFX using to read VDIF files

[Paul Demorest]

hi all,

Not sure I have a whole lot to add here, but a couple quick comments 
about the VDIF capabilites in dspsr..

The main thing is that it only handles a pretty specific flavor of VDIF, 
namely single-thread with either one or two polarization channels.  If 
this is a one-off data analysis project I would probably recommend just 
munging your data into this format using vmux before feeding it into 
dspsr.  I've done this occasionally for various projects, it's a little 
kludgey but works fine.  If this is something that will need to be done 
more often you could think about generalizing the dspsr VDIF reader.

Also, it looks like the vdifio-based code that dspsr uses may already 
have some modifications to read legacy headers.  Apparently Andrew 
Jameson put this in about a year ago, I don't have any experience using 
it though.

Cheers,
Paul

On 2020-03-19 19:25, Adam Deller via Difx-users wrote:
> Yep, that'll be your problem, I suspect.  It shouldn't be too much work 
> to
> fix vdifio to understand legacy 16 byte headers, but it would be good 
> to
> only have to do it once.  I'd suggest that it might be best to fix 
> vdifio
> in DiFX, and then make sure that dspsr pulls this update (might need 
> some
> checking/coordination with Paul D).
> 
> If you want to do a real quick and dirty hack check, just change the
> definition of VDIF_HEADER_BYTES from 32 to 16 locally and recompile.
> 
> If you get it going, would you be interested in making a better and 
> more
> permanent fix to VDIFIO?
> 
> Cheers,
> Adam
> 
> On Fri, 20 Mar 2020 at 12:16, Stuart Weston <stuart.weston at aut.ac.nz> 
> wrote:
> 
>> >> legacy (16 byte) headers?
>> 
>> 
>> 
>> I think our chain (DBBC/Fila10G/Jive5ab) is producing 16byte headers.
>> 
>> 
>> 
>> Well when I do a dumb read (python script) to check the data I have to 
>> use
>> 16bytes
>> 
>> 
>> 
>> Stuart Weston
>> 
>> 
>> 
>> Mobile: 021 713062
>> 
>> Skype: stuart.d.weston
>> Email:  stuart.weston at aut.ac.nz
>> 
>> http://www.atnf.csiro.au/people/Stuart.Weston/index.html
>> 
>> Software Engineer
>> Institute for Radio Astronomy & Space Research (IRASR)
>> School of Engineering, Computer and Mathematical Sciences
>> Faculty of Design and Creative Technologies
>> Auckland University of Technology, New Zealand.
>> 
>> http://www.irasr.aut.ac.nz/
>> 
>> 
>> 
>> *From:* Adam Deller <adeller at astro.swin.edu.au>
>> *Sent:* Friday, 20 March 2020 2:11 PM
>> *To:* Stuart Weston <stuart.weston at aut.ac.nz>
>> *Cc:* Phillips, Chris (CASS, Marsfield) <Chris.Phillips at csiro.au>;
>> difx-users at listmgr.nrao.edu
>> *Subject:* Re: [Difx-users] Question : What library/code is DiFX using 
>> to
>> read VDIF files
>> 
>> 
>> 
>> Hi Stuart,
>> 
>> 
>> 
>> Is your VDIF data regular (32 byte) or legacy (16 byte) headers?  I'm
>> pretty sure that the latter is known to be not supported by VDIFIO (a
>> relatively simple fix, that should certainly be made at some stage...)
>> 
>> 
>> 
>> Cheers,
>> 
>> Adam
>> 
>> 
>> 
>> On Fri, 20 Mar 2020 at 11:54, Stuart Weston <stuart.weston at aut.ac.nz>
>> wrote:
>> 
>> Thanks Adam,
>> 
>> 
>> 
>> I am working with some code that Willem hacked together for DSPSR 
>> using
>> Mark5Access.
>> 
>> 
>> 
>> Trying to get it to use our single dish data recorded in vdif format 
>> from
>> the DBBC/Fila10G/Jive5ab.
>> 
>> 
>> 
>> Paul’s code doesn’t read our vdif ☹ data
>> 
>> 
>> 
>> Stuart Weston
>> 
>> 
>> 
>> Mobile: 021 713062
>> 
>> Skype: stuart.d.weston
>> Email:  stuart.weston at aut.ac.nz
>> 
>> http://www.atnf.csiro.au/people/Stuart.Weston/index.html
>> 
>> Software Engineer
>> Institute for Radio Astronomy & Space Research (IRASR)
>> School of Engineering, Computer and Mathematical Sciences
>> Faculty of Design and Creative Technologies
>> Auckland University of Technology, New Zealand.
>> 
>> http://www.irasr.aut.ac.nz/
>> 
>> 
>> 
>> *From:* Adam Deller <adeller at astro.swin.edu.au>
>> *Sent:* Friday, 20 March 2020 1:50 PM
>> *To:* Stuart Weston <stuart.weston at aut.ac.nz>
>> *Cc:* Phillips, Chris (CASS, Marsfield) <Chris.Phillips at csiro.au>;
>> difx-users at listmgr.nrao.edu
>> *Subject:* Re: [Difx-users] Question : What library/code is DiFX using 
>> to
>> read VDIF files
>> 
>> 
>> 
>> Hi Stuart,
>> 
>> 
>> 
>> mpifxcorr uses mark5access to unpack VDIF samples.  The vdifio library
>> that is also included with DiFX is quite limited - apart from what 
>> Chris
>> already noted, I believe it doesn't support legacy VDIF.  Without any
>> disrespect to anyone who worked on it (including me :) ) it was really 
>> a
>> fairly quick hack to support some simple utilities.
>> 
>> 
>> 
>> Now, dspsr I think based some of its VDIF support code on vdifio.  
>> Paul
>> Demorest first made a copy of vdifio onto git - whether it ultimately
>> became an external mirror of the vdifio hosted on SVN in DiFX or if it 
>> has
>> branched off (and had functionality added) I'm actually not sure.
>> 
>> 
>> 
>> I know dspsr is capable of processing the VDIF-formatted data produced 
>> by
>> the VLA tied array system, so I'm pretty sure that unpacking is 
>> already
>> available in dspsr somewhere!  I'd suggest checking with Paul 
>> Demorest,
>> since he is definitely the key person for VDIF in dspsr.
>> 
>> 
>> 
>> Cheers,
>> 
>> Adam
>> 
>> 
>> 
>> On Fri, 20 Mar 2020 at 10:32, Stuart Weston via Difx-users <
>> difx-users at listmgr.nrao.edu> wrote:
>> 
>> >> If your question is a licensing one
>> 
>> 
>> 
>> Nope its not that.
>> 
>> 
>> 
>> So I take it DiFX uses Mark5access, yes ?
>> 
>> 
>> 
>> Stuart Weston
>> 
>> 
>> 
>> Mobile: 021 713062
>> 
>> Skype: stuart.d.weston
>> Email:  stuart.weston at aut.ac.nz
>> 
>> http://www.atnf.csiro.au/people/Stuart.Weston/index.html
>> 
>> Software Engineer
>> Institute for Radio Astronomy & Space Research (IRASR)
>> School of Engineering, Computer and Mathematical Sciences
>> Faculty of Design and Creative Technologies
>> Auckland University of Technology, New Zealand.
>> 
>> http://www.irasr.aut.ac.nz/
>> 
>> 
>> 
>> *From:* Phillips, Chris (CASS, Marsfield) <Chris.Phillips at csiro.au>
>> *Sent:* Friday, 20 March 2020 11:43 AM
>> *To:* Stuart Weston <stuart.weston at aut.ac.nz>
>> *Cc:* difx-users at listmgr.nrao.edu
>> *Subject:* Re: [Difx-users] Question : What library/code is DiFX using 
>> to
>> read VDIF files
>> 
>> 
>> 
>> Hi Stuart
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Just wondering if we can use the DiFX VDIF library/code instead ?
>> 
>> 
>> 
>> 
>> 
>> It is not clear what your question actually is.
>> 
>> 
>> 
>> The initial VDIFIO library I wrote had *no* sampling unpacking 
>> routines.
>> Others may had added some, I am not sure. Without that the answer 
>> would be
>> essentially be no. VDIFIO is in no way a plug in replacement for
>> mark5access.
>> 
>> 
>> 
>> If your question is a licensing one, then yes as all of DFIX is GPLv3 
>> now.
>> 
>> 
>> 
>> Cheers
>> 
>> Chris
>> 
>> _______________________________________________
>> Difx-users mailing list
>> Difx-users at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>> 
>> 
>> 
>> 
>> --
>> 
>> !=============================================================!
>> 
>> A/Prof. Adam Deller
>> 
>> ARC Future Fellow
>> 
>> Centre for Astrophysics & Supercomputing
>> 
>> Swinburne University of Technology
>> John St, Hawthorn VIC 3122 Australia
>> 
>> phone: +61 3 9214 5307
>> 
>> fax: +61 3 9214 8797
>> 
>> 
>> 
>> office days (usually): Mon-Thu
>> !=============================================================!
>> 
>> 
>> 
>> 
>> --
>> 
>> !=============================================================!
>> 
>> A/Prof. Adam Deller
>> 
>> ARC Future Fellow
>> 
>> Centre for Astrophysics & Supercomputing
>> 
>> Swinburne University of Technology
>> John St, Hawthorn VIC 3122 Australia
>> 
>> phone: +61 3 9214 5307
>> 
>> fax: +61 3 9214 8797
>> 
>> 
>> 
>> office days (usually): Mon-Thu
>> !=============================================================!
>> 
> 
> 
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users