[Difx-users] Question : What library/code is DiFX using to read VDIF files

Fri Mar 20 03:42:12 EDT 2020

Hi all,

I can safely say that FiLa10G does NOT produce legacy VDIF headers, it
produces nice 32-byte VDIF headers. Always has.
jive5ab produces legacy VDIF headers when using the cornerturning
functionality. However, not two days ago I merged a patch in the master
branch that would fix this.

So, if that would be only thing keeping you from reading the VDIF, it can
be A Thing Of The Past.

Cheers,
h

On Fri, 20 Mar 2020 at 03:57, Paul Demorest via Difx-users <
difx-users at listmgr.nrao.edu> wrote:

> hi all,
>
> Not sure I have a whole lot to add here, but a couple quick comments
> about the VDIF capabilites in dspsr..
>
> The main thing is that it only handles a pretty specific flavor of VDIF,
> namely single-thread with either one or two polarization channels.  If
> this is a one-off data analysis project I would probably recommend just
> munging your data into this format using vmux before feeding it into
> dspsr.  I've done this occasionally for various projects, it's a little
> kludgey but works fine.  If this is something that will need to be done
> more often you could think about generalizing the dspsr VDIF reader.
>
> Also, it looks like the vdifio-based code that dspsr uses may already
> have some modifications to read legacy headers.  Apparently Andrew
> Jameson put this in about a year ago, I don't have any experience using
> it though.
>
> Cheers,
> Paul
>
> On 2020-03-19 19:25, Adam Deller via Difx-users wrote:
> > Yep, that'll be your problem, I suspect.  It shouldn't be too much work
> > to
> > fix vdifio to understand legacy 16 byte headers, but it would be good
> > to
> > only have to do it once.  I'd suggest that it might be best to fix
> > vdifio
> > in DiFX, and then make sure that dspsr pulls this update (might need
> > some
> > checking/coordination with Paul D).
> >
> > If you want to do a real quick and dirty hack check, just change the
> > definition of VDIF_HEADER_BYTES from 32 to 16 locally and recompile.
> >
> > If you get it going, would you be interested in making a better and
> > more
> > permanent fix to VDIFIO?
> >
> > Cheers,
> > Adam
> >
> > On Fri, 20 Mar 2020 at 12:16, Stuart Weston <stuart.weston at aut.ac.nz>
> > wrote:
> >
> >> >> legacy (16 byte) headers?
> >>
> >>
> >>
> >> I think our chain (DBBC/Fila10G/Jive5ab) is producing 16byte headers.
> >>
> >>
> >>
> >> Well when I do a dumb read (python script) to check the data I have to
> >> use
> >> 16bytes
> >>
> >>
> >>
> >> Stuart Weston
> >>
> >>
> >>
> >> Mobile: 021 713062
> >>
> >> Skype: stuart.d.weston
> >> Email:  stuart.weston at aut.ac.nz
> >>
> >> http://www.atnf.csiro.au/people/Stuart.Weston/index.html
> >>
> >> Software Engineer
> >> Institute for Radio Astronomy & Space Research (IRASR)
> >> School of Engineering, Computer and Mathematical Sciences
> >> Faculty of Design and Creative Technologies
> >> Auckland University of Technology, New Zealand.
> >>
> >> http://www.irasr.aut.ac.nz/
> >>
> >>
> >>
> >> *From:* Adam Deller <adeller at astro.swin.edu.au>
> >> *Sent:* Friday, 20 March 2020 2:11 PM
> >> *To:* Stuart Weston <stuart.weston at aut.ac.nz>
> >> *Cc:* Phillips, Chris (CASS, Marsfield) <Chris.Phillips at csiro.au>;
> >> difx-users at listmgr.nrao.edu
> >> *Subject:* Re: [Difx-users] Question : What library/code is DiFX using
> >> to
> >> read VDIF files
> >>
> >>
> >>
> >> Hi Stuart,
> >>
> >>
> >>
> >> Is your VDIF data regular (32 byte) or legacy (16 byte) headers?  I'm
> >> pretty sure that the latter is known to be not supported by VDIFIO (a
> >> relatively simple fix, that should certainly be made at some stage...)
> >>
> >>
> >>
> >> Cheers,
> >>
> >> Adam
> >>
> >>
> >>
> >> On Fri, 20 Mar 2020 at 11:54, Stuart Weston <stuart.weston at aut.ac.nz>
> >> wrote:
> >>
> >> Thanks Adam,
> >>
> >>
> >>
> >> I am working with some code that Willem hacked together for DSPSR
> >> using
> >> Mark5Access.
> >>
> >>
> >>
> >> Trying to get it to use our single dish data recorded in vdif format
> >> from
> >> the DBBC/Fila10G/Jive5ab.
> >>
> >>
> >>
> >> Paul’s code doesn’t read our vdif ☹ data
> >>
> >>
> >>
> >> Stuart Weston
> >>
> >>
> >>
> >> Mobile: 021 713062
> >>
> >> Skype: stuart.d.weston
> >> Email:  stuart.weston at aut.ac.nz
> >>
> >> http://www.atnf.csiro.au/people/Stuart.Weston/index.html
> >>
> >> Software Engineer
> >> Institute for Radio Astronomy & Space Research (IRASR)
> >> School of Engineering, Computer and Mathematical Sciences
> >> Faculty of Design and Creative Technologies
> >> Auckland University of Technology, New Zealand.
> >>
> >> http://www.irasr.aut.ac.nz/
> >>
> >>
> >>
> >> *From:* Adam Deller <adeller at astro.swin.edu.au>
> >> *Sent:* Friday, 20 March 2020 1:50 PM
> >> *To:* Stuart Weston <stuart.weston at aut.ac.nz>
> >> *Cc:* Phillips, Chris (CASS, Marsfield) <Chris.Phillips at csiro.au>;
> >> difx-users at listmgr.nrao.edu
> >> *Subject:* Re: [Difx-users] Question : What library/code is DiFX using
> >> to
> >> read VDIF files
> >>
> >>
> >>
> >> Hi Stuart,
> >>
> >>
> >>
> >> mpifxcorr uses mark5access to unpack VDIF samples.  The vdifio library
> >> that is also included with DiFX is quite limited - apart from what
> >> Chris
> >> already noted, I believe it doesn't support legacy VDIF.  Without any
> >> disrespect to anyone who worked on it (including me :) ) it was really
> >> a
> >> fairly quick hack to support some simple utilities.
> >>
> >>
> >>
> >> Now, dspsr I think based some of its VDIF support code on vdifio.
> >> Paul
> >> Demorest first made a copy of vdifio onto git - whether it ultimately
> >> became an external mirror of the vdifio hosted on SVN in DiFX or if it
> >> has
> >> branched off (and had functionality added) I'm actually not sure.
> >>
> >>
> >>
> >> I know dspsr is capable of processing the VDIF-formatted data produced
> >> by
> >> the VLA tied array system, so I'm pretty sure that unpacking is
> >> already
> >> available in dspsr somewhere!  I'd suggest checking with Paul
> >> Demorest,
> >> since he is definitely the key person for VDIF in dspsr.
> >>
> >>
> >>
> >> Cheers,
> >>
> >> Adam
> >>
> >>
> >>
> >> On Fri, 20 Mar 2020 at 10:32, Stuart Weston via Difx-users <
> >> difx-users at listmgr.nrao.edu> wrote:
> >>
> >> >> If your question is a licensing one
> >>
> >>
> >>
> >> Nope its not that.
> >>
> >>
> >>
> >> So I take it DiFX uses Mark5access, yes ?
> >>
> >>
> >>
> >> Stuart Weston
> >>
> >>
> >>
> >> Mobile: 021 713062
> >>
> >> Skype: stuart.d.weston
> >> Email:  stuart.weston at aut.ac.nz
> >>
> >> http://www.atnf.csiro.au/people/Stuart.Weston/index.html
> >>
> >> Software Engineer
> >> Institute for Radio Astronomy & Space Research (IRASR)
> >> School of Engineering, Computer and Mathematical Sciences
> >> Faculty of Design and Creative Technologies
> >> Auckland University of Technology, New Zealand.
> >>
> >> http://www.irasr.aut.ac.nz/
> >>
> >>
> >>
> >> *From:* Phillips, Chris (CASS, Marsfield) <Chris.Phillips at csiro.au>
> >> *Sent:* Friday, 20 March 2020 11:43 AM
> >> *To:* Stuart Weston <stuart.weston at aut.ac.nz>
> >> *Cc:* difx-users at listmgr.nrao.edu
> >> *Subject:* Re: [Difx-users] Question : What library/code is DiFX using
> >> to
> >> read VDIF files
> >>
> >>
> >>
> >> Hi Stuart
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Just wondering if we can use the DiFX VDIF library/code instead ?
> >>
> >>
> >>
> >>
> >>
> >> It is not clear what your question actually is.
> >>
> >>
> >>
> >> The initial VDIFIO library I wrote had *no* sampling unpacking
> >> routines.
> >> Others may had added some, I am not sure. Without that the answer
> >> would be
> >> essentially be no. VDIFIO is in no way a plug in replacement for
> >> mark5access.
> >>
> >>
> >>
> >> If your question is a licensing one, then yes as all of DFIX is GPLv3
> >> now.
> >>
> >>
> >>
> >> Cheers
> >>
> >> Chris
> >>
> >> _______________________________________________
> >> Difx-users mailing list
> >> Difx-users at listmgr.nrao.edu
> >> https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >>
> >>
> >>
> >>
> >> --
> >>
> >> !=============================================================!
> >>
> >> A/Prof. Adam Deller
> >>
> >> ARC Future Fellow
> >>
> >> Centre for Astrophysics & Supercomputing
> >>
> >> Swinburne University of Technology
> >> John St, Hawthorn VIC 3122 Australia
> >>
> >> phone: +61 3 9214 5307
> >>
> >> fax: +61 3 9214 8797
> >>
> >>
> >>
> >> office days (usually): Mon-Thu
> >> !=============================================================!
> >>
> >>
> >>
> >>
> >> --
> >>
> >> !=============================================================!
> >>
> >> A/Prof. Adam Deller
> >>
> >> ARC Future Fellow
> >>
> >> Centre for Astrophysics & Supercomputing
> >>
> >> Swinburne University of Technology
> >> John St, Hawthorn VIC 3122 Australia
> >>
> >> phone: +61 3 9214 5307
> >>
> >> fax: +61 3 9214 8797
> >>
> >>
> >>
> >> office days (usually): Mon-Thu
> >> !=============================================================!
> >>
> >
> >
> > _______________________________________________
> > Difx-users mailing list
> > Difx-users at listmgr.nrao.edu
> > https://listmgr.nrao.edu/mailman/listinfo/difx-users
>
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20200320/508e7a09/attachment-0001.html>