[Difx-users] complex support

Tue Mar 23 21:27:21 EDT 2021

hi Jayce,

have you taken steps (e.g. by freqId selection) to make sure you are
not getting correlations between the parent bands - e.g. from VLA-VLA
or from LWA-LWA stations? By default DiFX will correlate anything it
can find that seems to match.

cheers,
Cormac

On Tue, 2021-03-23 at 19:10 -0600, Jayce Dowell via Difx-users wrote:
> Hi Adam & Chris,
> 
> I've moved on to the VLA+LWA data and am running into a new
> problem.  I 
> have my frequency setups as:
> 
> # FREQ TABLE #######!
> FREQ ENTRIES:       2
> FREQ (MHZ) 0:       71.09999999441
> BW (MHZ) 0:         9.80000000000
> SIDEBAND 0:         U
> NUM CHANNELS 0:     392
> CHANS TO AVG 0:     1
> OVERSAMPLE FAC. 0:  1
> DECIMATION FAC. 0:  1
> PHASE CALS 0 OUT:   0
> FREQ (MHZ) 1:       71.99999999441
> BW (MHZ) 1:         8.00000000000
> SIDEBAND 1:         U
> NUM CHANNELS 1:     320
> CHANS TO AVG 1:     1
> OVERSAMPLE FAC. 1:  1
> DECIMATION FAC. 1:  1
> PHASE CALS 1 OUT:   0
> 
> In my data streams table I have the LWA stations recording complex 
> double sideband with frequency setup 0 and one zoom band of setup 1. 
> The VLA antennas are set real upper sideband with frequency setup 1
> and 
> no zoom band.  I think this is what I want and mpifxcorr doesn't 
> complain when I run it.  However, difx2fits immediately segfaults
> when I 
> try to build a FITS-IDI file.  Any suggestions?
> 
> Jayce
> --
> On 3/10/21 4:19 PM, Adam Deller wrote:
> > *  [EXTERNAL]*
> > 
> > **
> > Glad to hear it!
> > 
> > On Thu, 11 Mar 2021 at 09:58, Jayce Dowell <jdowell at unm.edu 
> > <mailto:jdowell at unm.edu>> wrote:
> > 
> >     Hi Chris,
> > 
> >     After some digging I was able to get to the bottom of this and
> > I am now
> >     getting results that I am happy with.  It was a combination of
> >     switching
> >     over to trunk and fixing a bad assumption in my VDIF conversion
> > that
> >     everyone represents four-bit complex integers in the same way.
> > 
> >     Jayce
> >     --
> >     On 2/4/21 3:08 PM, Phillips, Chris (CASS, Marsfield) wrote:
> >      >    [EXTERNAL]
> >      >
> >      > Hi Jayce
> >      >
> >      > I'm working the data from the Long Wavelength Array and I am
> >     trying to see if we can move from our homegrown correlator to
> > DiFX. 
> >     The two modes I need to support are a LWA-only mode with
> > homogeneous
> >     data and an "eLWA" mode with heterogeneous data that combines
> > the
> >     LWA stations with the VLA 4-band system.  For the LWA-only mode
> > I
> >     have 4+4-bit complex voltage data from two 19.6 MHz wide
> > spectral
> >     windows.  I am currently converting these data from the LWA
> > format
> >     into two VDIF files, one per window, for correlation with
> > DiFX.  My
> >     .input file describes the data as:
> >      >>
> >      >> DATA FORMAT:        INTERLACEDVDIF/0:1
> >      >> QUANTISATION BITS:  4
> >      >> DATA FRAME SIZE:    7872
> >      >> DATA SAMPLING:      COMPLEX_DSB
> >      >> DATA SOURCE:        FILE
> >      >>
> >      >
> >      > I have never personally tested complex_dsb data with
> > interlaced
> >     vdif. I have no idea if there could be any “interaction” with
> > the
> >     VDIF interleaving and the specific data type (I would have
> > thought
> >     not, but you never know).
> >      >
> >      > The issue could also be with interpreting the interlaced
> > VDIF,
> >     not the complex double sideband.
> >      >
> >      > Once you have tested using trunk, not the released version,
> > I
> >     would try and use some of the vdifio utilities to try and
> > convert
> >     the interlaced VDIF into plain VDIF - either extracting out one
> >     thread, or merging into multichannel single thread VDIF. You
> > would
> >     obviously need to update the DIFX .input (and or .v2d) file and
> >     vexfile to match the new setup.
> >      >
> >      > I would also be using the m5access tools to do things like
> > make
> >     the autocorrelation (both before and after fiddling with the
> > data
> >     layout). Check DIFX gives the same auto shape as you get with
> >     m5spec.  If you use a modulated noise source, folding the data
> >     (m5fold) on the period of the noise cal can be very informative
> >     also.  Also look at the headers and check that the # frames/sec
> > etc
> >     are what you expect (before and after merging threads etc). If
> > you
> >     decide it is the interlacing is the issue, I suspect you will
> > need
> >     to discuss with Walter.
> >      >
> >      >
> >      >> I am less sure about DiFX supporting our eLWA mode.  For
> > this we
> >     have the same 4+4-bit complex voltage data from the LWA
> > stations but
> >     with a bandwidth of 9.8 MHz.  We are combining this with 4-bit
> > real
> >     voltages from the VLA with 8 MHz of bandwidth and a slightly
> >     different frequency setup.  We currently process these data by
> >     shifting the center frequency of the LWA data to match that of
> > the
> >     VLA, running two F-engines with different channelization to
> > arrive
> >     at the same channel width for both data sources, and then cross
> >     correlate what overlaps.  Is this mode something that DiFX
> > supports?
> >      >>
> >      >
> >      > Can you represent this data as VDIF? If you you can probably
> >     handle the different setup using zoom bands. However if this
> > can be
> >     done will really depend on the EXACT setup. Basically you need
> > to be
> >     able to channelise the two setups with the same final frequency
> >     resolution (ie different FFT sizes will be needed, with the
> > ratio of
> >     the FFT size matching the ratio of the bandwidth difference.
> > But the
> >     sky frequency of the frequency points out of the FFT need to
> > match
> >     exactly (ie even if you could achieve the same frequency
> > resolution,
> >     if the fine channels are offset by 1/3 of a channel you will
> > have
> >     problems (if that offset is a simple integer ratio, you can
> > just use
> >     a higher frequency resolution)
> >      >
> >      > I hope this makes sense.
> >      >
> >      > Cheers
> >      > Chris
> >      >
> > 
> > 
> > 
> > -- 
> > !=============================================================!
> > A/Prof. Adam Deller
> > ARC Future Fellow
> > Centre for Astrophysics & Supercomputing
> > Swinburne University of Technology
> > John St, Hawthorn VIC 3122 Australia
> > phone: +61 3 9214 5307
> > fax: +61 3 9214 8797
> > 
> > office days (usually): Mon-Thu
> > !=============================================================!
> 
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users