[Difx-users] Question : What library/code is DiFX using to read VDIF files
Franz Kirsten
franz.kirsten at chalmers.se
Fri Mar 20 03:54:50 EDT 2020
Hi all,
I've been using dspsr/digifil for about two years now to process both
legacy and non-legacy VDIF. (AJ fixed the legacy VDIF issue for me in
dspsr). Like Paul said, it only understands single-thread (up to-) two
channel VDIF. For things to work you need to create a header file along
the lines of the attached example.
At first the single-thread limitation might seem to be a hassle. Turns
out though that in this way you get to nicely parallelise your
processing by working on individual subbands. I use jive5ab to extract
individual IFs and write them out to seperate files. Let me know in case
you're interested in this route.
Cheers,
Franz
On 20/3/20 3:56 am, Paul Demorest via Difx-users wrote:
> hi all,
>
> Not sure I have a whole lot to add here, but a couple quick comments
> about the VDIF capabilites in dspsr..
>
> The main thing is that it only handles a pretty specific flavor of VDIF,
> namely single-thread with either one or two polarization channels. If
> this is a one-off data analysis project I would probably recommend just
> munging your data into this format using vmux before feeding it into
> dspsr. I've done this occasionally for various projects, it's a little
> kludgey but works fine. If this is something that will need to be done
> more often you could think about generalizing the dspsr VDIF reader.
>
> Also, it looks like the vdifio-based code that dspsr uses may already
> have some modifications to read legacy headers. Apparently Andrew
> Jameson put this in about a year ago, I don't have any experience using
> it though.
>
> Cheers,
> Paul
>
> On 2020-03-19 19:25, Adam Deller via Difx-users wrote:
>> Yep, that'll be your problem, I suspect. It shouldn't be too much
>> work to
>> fix vdifio to understand legacy 16 byte headers, but it would be good to
>> only have to do it once. I'd suggest that it might be best to fix vdifio
>> in DiFX, and then make sure that dspsr pulls this update (might need some
>> checking/coordination with Paul D).
>>
>> If you want to do a real quick and dirty hack check, just change the
>> definition of VDIF_HEADER_BYTES from 32 to 16 locally and recompile.
>>
>> If you get it going, would you be interested in making a better and more
>> permanent fix to VDIFIO?
>>
>> Cheers,
>> Adam
>>
>> On Fri, 20 Mar 2020 at 12:16, Stuart Weston <stuart.weston at aut.ac.nz>
>> wrote:
>>
>>> >> legacy (16 byte) headers?
>>>
>>>
>>>
>>> I think our chain (DBBC/Fila10G/Jive5ab) is producing 16byte headers.
>>>
>>>
>>>
>>> Well when I do a dumb read (python script) to check the data I have
>>> to use
>>> 16bytes
>>>
>>>
>>>
>>> Stuart Weston
>>>
>>>
>>>
>>> Mobile: 021 713062
>>>
>>> Skype: stuart.d.weston
>>> Email: stuart.weston at aut.ac.nz
>>>
>>> http://www.atnf.csiro.au/people/Stuart.Weston/index.html
>>>
>>> Software Engineer
>>> Institute for Radio Astronomy & Space Research (IRASR)
>>> School of Engineering, Computer and Mathematical Sciences
>>> Faculty of Design and Creative Technologies
>>> Auckland University of Technology, New Zealand.
>>>
>>> http://www.irasr.aut.ac.nz/
>>>
>>>
>>>
>>> *From:* Adam Deller <adeller at astro.swin.edu.au>
>>> *Sent:* Friday, 20 March 2020 2:11 PM
>>> *To:* Stuart Weston <stuart.weston at aut.ac.nz>
>>> *Cc:* Phillips, Chris (CASS, Marsfield) <Chris.Phillips at csiro.au>;
>>> difx-users at listmgr.nrao.edu
>>> *Subject:* Re: [Difx-users] Question : What library/code is DiFX
>>> using to
>>> read VDIF files
>>>
>>>
>>>
>>> Hi Stuart,
>>>
>>>
>>>
>>> Is your VDIF data regular (32 byte) or legacy (16 byte) headers? I'm
>>> pretty sure that the latter is known to be not supported by VDIFIO (a
>>> relatively simple fix, that should certainly be made at some stage...)
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Adam
>>>
>>>
>>>
>>> On Fri, 20 Mar 2020 at 11:54, Stuart Weston <stuart.weston at aut.ac.nz>
>>> wrote:
>>>
>>> Thanks Adam,
>>>
>>>
>>>
>>> I am working with some code that Willem hacked together for DSPSR using
>>> Mark5Access.
>>>
>>>
>>>
>>> Trying to get it to use our single dish data recorded in vdif format
>>> from
>>> the DBBC/Fila10G/Jive5ab.
>>>
>>>
>>>
>>> Paul’s code doesn’t read our vdif ☹ data
>>>
>>>
>>>
>>> Stuart Weston
>>>
>>>
>>>
>>> Mobile: 021 713062
>>>
>>> Skype: stuart.d.weston
>>> Email: stuart.weston at aut.ac.nz
>>>
>>> http://www.atnf.csiro.au/people/Stuart.Weston/index.html
>>>
>>> Software Engineer
>>> Institute for Radio Astronomy & Space Research (IRASR)
>>> School of Engineering, Computer and Mathematical Sciences
>>> Faculty of Design and Creative Technologies
>>> Auckland University of Technology, New Zealand.
>>>
>>> http://www.irasr.aut.ac.nz/
>>>
>>>
>>>
>>> *From:* Adam Deller <adeller at astro.swin.edu.au>
>>> *Sent:* Friday, 20 March 2020 1:50 PM
>>> *To:* Stuart Weston <stuart.weston at aut.ac.nz>
>>> *Cc:* Phillips, Chris (CASS, Marsfield) <Chris.Phillips at csiro.au>;
>>> difx-users at listmgr.nrao.edu
>>> *Subject:* Re: [Difx-users] Question : What library/code is DiFX
>>> using to
>>> read VDIF files
>>>
>>>
>>>
>>> Hi Stuart,
>>>
>>>
>>>
>>> mpifxcorr uses mark5access to unpack VDIF samples. The vdifio library
>>> that is also included with DiFX is quite limited - apart from what Chris
>>> already noted, I believe it doesn't support legacy VDIF. Without any
>>> disrespect to anyone who worked on it (including me :) ) it was really a
>>> fairly quick hack to support some simple utilities.
>>>
>>>
>>>
>>> Now, dspsr I think based some of its VDIF support code on vdifio. Paul
>>> Demorest first made a copy of vdifio onto git - whether it ultimately
>>> became an external mirror of the vdifio hosted on SVN in DiFX or if
>>> it has
>>> branched off (and had functionality added) I'm actually not sure.
>>>
>>>
>>>
>>> I know dspsr is capable of processing the VDIF-formatted data
>>> produced by
>>> the VLA tied array system, so I'm pretty sure that unpacking is already
>>> available in dspsr somewhere! I'd suggest checking with Paul Demorest,
>>> since he is definitely the key person for VDIF in dspsr.
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Adam
>>>
>>>
>>>
>>> On Fri, 20 Mar 2020 at 10:32, Stuart Weston via Difx-users <
>>> difx-users at listmgr.nrao.edu> wrote:
>>>
>>> >> If your question is a licensing one
>>>
>>>
>>>
>>> Nope its not that.
>>>
>>>
>>>
>>> So I take it DiFX uses Mark5access, yes ?
>>>
>>>
>>>
>>> Stuart Weston
>>>
>>>
>>>
>>> Mobile: 021 713062
>>>
>>> Skype: stuart.d.weston
>>> Email: stuart.weston at aut.ac.nz
>>>
>>> http://www.atnf.csiro.au/people/Stuart.Weston/index.html
>>>
>>> Software Engineer
>>> Institute for Radio Astronomy & Space Research (IRASR)
>>> School of Engineering, Computer and Mathematical Sciences
>>> Faculty of Design and Creative Technologies
>>> Auckland University of Technology, New Zealand.
>>>
>>> http://www.irasr.aut.ac.nz/
>>>
>>>
>>>
>>> *From:* Phillips, Chris (CASS, Marsfield) <Chris.Phillips at csiro.au>
>>> *Sent:* Friday, 20 March 2020 11:43 AM
>>> *To:* Stuart Weston <stuart.weston at aut.ac.nz>
>>> *Cc:* difx-users at listmgr.nrao.edu
>>> *Subject:* Re: [Difx-users] Question : What library/code is DiFX
>>> using to
>>> read VDIF files
>>>
>>>
>>>
>>> Hi Stuart
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Just wondering if we can use the DiFX VDIF library/code instead ?
>>>
>>>
>>>
>>>
>>>
>>> It is not clear what your question actually is.
>>>
>>>
>>>
>>> The initial VDIFIO library I wrote had *no* sampling unpacking routines.
>>> Others may had added some, I am not sure. Without that the answer
>>> would be
>>> essentially be no. VDIFIO is in no way a plug in replacement for
>>> mark5access.
>>>
>>>
>>>
>>> If your question is a licensing one, then yes as all of DFIX is GPLv3
>>> now.
>>>
>>>
>>>
>>> Cheers
>>>
>>> Chris
>>>
>>> _______________________________________________
>>> Difx-users mailing list
>>> Difx-users at listmgr.nrao.edu
>>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>>>
>>>
>>>
>>>
>>> --
>>>
>>> !=============================================================!
>>>
>>> A/Prof. Adam Deller
>>>
>>> ARC Future Fellow
>>>
>>> Centre for Astrophysics & Supercomputing
>>>
>>> Swinburne University of Technology
>>> John St, Hawthorn VIC 3122 Australia
>>>
>>> phone: +61 3 9214 5307
>>>
>>> fax: +61 3 9214 8797
>>>
>>>
>>>
>>> office days (usually): Mon-Thu
>>> !=============================================================!
>>>
>>>
>>>
>>>
>>> --
>>>
>>> !=============================================================!
>>>
>>> A/Prof. Adam Deller
>>>
>>> ARC Future Fellow
>>>
>>> Centre for Astrophysics & Supercomputing
>>>
>>> Swinburne University of Technology
>>> John St, Hawthorn VIC 3122 Australia
>>>
>>> phone: +61 3 9214 5307
>>>
>>> fax: +61 3 9214 8797
>>>
>>>
>>>
>>> office days (usually): Mon-Thu
>>> !=============================================================!
>>>
>>
>>
>> _______________________________________________
>> Difx-users mailing list
>> Difx-users at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users
-------------- next part --------------
HDR_VERSION 0.1
TELESCOPE Onsala85
SOURCE R3
RA 01:58:00.7495
DEC 65:43:00.3185
FREQ 1672.0
BW -32.0
DATAFILE /mnt/franz/pr086b/pr086b_o8_no0152_IF15_s1.vdif
INSTRUMENT VDIF
MODE PSR
BASIS Circular
More information about the Difx-users
mailing list