[Difx-users] complex support

Tue Mar 23 21:39:46 EDT 2021

In the 'setup' section of the v2d file you can set, e.g.

freqId=1,2

to select just the second and third frequency setups in the input file. This can be a bit fragile in complex, changing setups, but should be straightforward in your case, assuming all jobs have both VLA and LWA participation.

Cheers,
Cormac
________________________________
From: Jayce Dowell <jdowell at unm.edu>
Sent: Wednesday, March 24, 2021 9:32:54 AM
To: Reynolds, Cormac (CASS, Kensington WA) <Cormac.Reynolds at csiro.au>; adeller at astro.swin.edu.au <adeller at astro.swin.edu.au>
Cc: difx-users at listmgr.nrao.edu <difx-users at listmgr.nrao.edu>
Subject: Re: [Difx-users] complex support

Hi Cormac,

I am not sure which probably means no.  How/where would I do that?

Jayce
--
On 3/23/21 7:27 PM, Reynolds, Cormac (CASS, Kensington WA) wrote:
>    [EXTERNAL]
>
> hi Jayce,
>
> have you taken steps (e.g. by freqId selection) to make sure you are
> not getting correlations between the parent bands - e.g. from VLA-VLA
> or from LWA-LWA stations? By default DiFX will correlate anything it
> can find that seems to match.
>
> cheers,
> Cormac
>
> On Tue, 2021-03-23 at 19:10 -0600, Jayce Dowell via Difx-users wrote:
>> Hi Adam & Chris,
>>
>> I've moved on to the VLA+LWA data and am running into a new
>> problem.  I
>> have my frequency setups as:
>>
>> # FREQ TABLE #######!
>> FREQ ENTRIES:       2
>> FREQ (MHZ) 0:       71.09999999441
>> BW (MHZ) 0:         9.80000000000
>> SIDEBAND 0:         U
>> NUM CHANNELS 0:     392
>> CHANS TO AVG 0:     1
>> OVERSAMPLE FAC. 0:  1
>> DECIMATION FAC. 0:  1
>> PHASE CALS 0 OUT:   0
>> FREQ (MHZ) 1:       71.99999999441
>> BW (MHZ) 1:         8.00000000000
>> SIDEBAND 1:         U
>> NUM CHANNELS 1:     320
>> CHANS TO AVG 1:     1
>> OVERSAMPLE FAC. 1:  1
>> DECIMATION FAC. 1:  1
>> PHASE CALS 1 OUT:   0
>>
>> In my data streams table I have the LWA stations recording complex
>> double sideband with frequency setup 0 and one zoom band of setup 1.
>> The VLA antennas are set real upper sideband with frequency setup 1
>> and
>> no zoom band.  I think this is what I want and mpifxcorr doesn't
>> complain when I run it.  However, difx2fits immediately segfaults
>> when I
>> try to build a FITS-IDI file.  Any suggestions?
>>
>> Jayce
>> --
>> On 3/10/21 4:19 PM, Adam Deller wrote:
>>> *  [EXTERNAL]*
>>>
>>> **
>>> Glad to hear it!
>>>
>>> On Thu, 11 Mar 2021 at 09:58, Jayce Dowell <jdowell at unm.edu
>>> <mailto:jdowell at unm.edu>> wrote:
>>>
>>>      Hi Chris,
>>>
>>>      After some digging I was able to get to the bottom of this and
>>> I am now
>>>      getting results that I am happy with.  It was a combination of
>>>      switching
>>>      over to trunk and fixing a bad assumption in my VDIF conversion
>>> that
>>>      everyone represents four-bit complex integers in the same way.
>>>
>>>      Jayce
>>>      --
>>>      On 2/4/21 3:08 PM, Phillips, Chris (CASS, Marsfield) wrote:
>>>       >    [EXTERNAL]
>>>       >
>>>       > Hi Jayce
>>>       >
>>>       > I'm working the data from the Long Wavelength Array and I am
>>>      trying to see if we can move from our homegrown correlator to
>>> DiFX.
>>>      The two modes I need to support are a LWA-only mode with
>>> homogeneous
>>>      data and an "eLWA" mode with heterogeneous data that combines
>>> the
>>>      LWA stations with the VLA 4-band system.  For the LWA-only mode
>>> I
>>>      have 4+4-bit complex voltage data from two 19.6 MHz wide
>>> spectral
>>>      windows.  I am currently converting these data from the LWA
>>> format
>>>      into two VDIF files, one per window, for correlation with
>>> DiFX.  My
>>>      .input file describes the data as:
>>>       >>
>>>       >> DATA FORMAT:        INTERLACEDVDIF/0:1
>>>       >> QUANTISATION BITS:  4
>>>       >> DATA FRAME SIZE:    7872
>>>       >> DATA SAMPLING:      COMPLEX_DSB
>>>       >> DATA SOURCE:        FILE
>>>       >>
>>>       >
>>>       > I have never personally tested complex_dsb data with
>>> interlaced
>>>      vdif. I have no idea if there could be any “interaction” with
>>> the
>>>      VDIF interleaving and the specific data type (I would have
>>> thought
>>>      not, but you never know).
>>>       >
>>>       > The issue could also be with interpreting the interlaced
>>> VDIF,
>>>      not the complex double sideband.
>>>       >
>>>       > Once you have tested using trunk, not the released version,
>>> I
>>>      would try and use some of the vdifio utilities to try and
>>> convert
>>>      the interlaced VDIF into plain VDIF - either extracting out one
>>>      thread, or merging into multichannel single thread VDIF. You
>>> would
>>>      obviously need to update the DIFX .input (and or .v2d) file and
>>>      vexfile to match the new setup.
>>>       >
>>>       > I would also be using the m5access tools to do things like
>>> make
>>>      the autocorrelation (both before and after fiddling with the
>>> data
>>>      layout). Check DIFX gives the same auto shape as you get with
>>>      m5spec.  If you use a modulated noise source, folding the data
>>>      (m5fold) on the period of the noise cal can be very informative
>>>      also.  Also look at the headers and check that the # frames/sec
>>> etc
>>>      are what you expect (before and after merging threads etc). If
>>> you
>>>      decide it is the interlacing is the issue, I suspect you will
>>> need
>>>      to discuss with Walter.
>>>       >
>>>       >
>>>       >> I am less sure about DiFX supporting our eLWA mode.  For
>>> this we
>>>      have the same 4+4-bit complex voltage data from the LWA
>>> stations but
>>>      with a bandwidth of 9.8 MHz.  We are combining this with 4-bit
>>> real
>>>      voltages from the VLA with 8 MHz of bandwidth and a slightly
>>>      different frequency setup.  We currently process these data by
>>>      shifting the center frequency of the LWA data to match that of
>>> the
>>>      VLA, running two F-engines with different channelization to
>>> arrive
>>>      at the same channel width for both data sources, and then cross
>>>      correlate what overlaps.  Is this mode something that DiFX
>>> supports?
>>>       >>
>>>       >
>>>       > Can you represent this data as VDIF? If you you can probably
>>>      handle the different setup using zoom bands. However if this
>>> can be
>>>      done will really depend on the EXACT setup. Basically you need
>>> to be
>>>      able to channelise the two setups with the same final frequency
>>>      resolution (ie different FFT sizes will be needed, with the
>>> ratio of
>>>      the FFT size matching the ratio of the bandwidth difference.
>>> But the
>>>      sky frequency of the frequency points out of the FFT need to
>>> match
>>>      exactly (ie even if you could achieve the same frequency
>>> resolution,
>>>      if the fine channels are offset by 1/3 of a channel you will
>>> have
>>>      problems (if that offset is a simple integer ratio, you can
>>> just use
>>>      a higher frequency resolution)
>>>       >
>>>       > I hope this makes sense.
>>>       >
>>>       > Cheers
>>>       > Chris
>>>       >
>>>
>>>
>>>
>>> --
>>> !=============================================================!
>>> A/Prof. Adam Deller
>>> ARC Future Fellow
>>> Centre for Astrophysics & Supercomputing
>>> Swinburne University of Technology
>>> John St, Hawthorn VIC 3122 Australia
>>> phone: +61 3 9214 5307
>>> fax: +61 3 9214 8797
>>>
>>> office days (usually): Mon-Thu
>>> !=============================================================!
>>
>> _______________________________________________
>> Difx-users mailing list
>> Difx-users at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/difx-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20210324/26481ec9/attachment-0001.html>