[Difx-users] complex support

[Jan Wagner]

Hi Jayce,

you could use "printDiFXInput.py <inputfile>" from either DiFX 2.6.2
or for more details DiFX Trunk. That will show you the list of
datastreams and their recorded plus zoom frequencies, and a summary
of the baseline table showing the frequencies that are expected to
be found in the *.difx/DIFX_... output files.

Similarly, "printDiFX.py <difxfile>" lists the visibility records in
the mpifxcorr output files. Maybe some frequencies there are
"unexpected"?

Depending on how difx2fits was compiled and whether you have the gdb
debugger installed, you could "gdb --args difx2fits -d test.fits"
and type "run" in the gdb console. After the crash, type "bt" in the
same gdb console. That might produce a more detailed explanation
(for developers) of what could have provoked the segfault.

regards,
Jan

Am 24.03.2021 um 02:10 schrieb Jayce Dowell via Difx-users:
> Hi Adam & Chris,
>
> I've moved on to the VLA+LWA data and am running into a new
> problem.  I have my frequency setups as:
>
> # FREQ TABLE #######!
> FREQ ENTRIES:       2
> FREQ (MHZ) 0:       71.09999999441
> BW (MHZ) 0:         9.80000000000
> SIDEBAND 0:         U
> NUM CHANNELS 0:     392
> CHANS TO AVG 0:     1
> OVERSAMPLE FAC. 0:  1
> DECIMATION FAC. 0:  1
> PHASE CALS 0 OUT:   0
> FREQ (MHZ) 1:       71.99999999441
> BW (MHZ) 1:         8.00000000000
> SIDEBAND 1:         U
> NUM CHANNELS 1:     320
> CHANS TO AVG 1:     1
> OVERSAMPLE FAC. 1:  1
> DECIMATION FAC. 1:  1
> PHASE CALS 1 OUT:   0
>
> In my data streams table I have the LWA stations recording complex
> double sideband with frequency setup 0 and one zoom band of setup
> 1. The VLA antennas are set real upper sideband with frequency
> setup 1 and no zoom band.  I think this is what I want and
> mpifxcorr doesn't complain when I run it.  However, difx2fits
> immediately segfaults when I try to build a FITS-IDI file.  Any
> suggestions?
>
> Jayce
> -- 
> On 3/10/21 4:19 PM, Adam Deller wrote:
>> *  [EXTERNAL]*
>>
>> **
>> Glad to hear it!
>>
>> On Thu, 11 Mar 2021 at 09:58, Jayce Dowell <jdowell at unm.edu
>> <mailto:jdowell at unm.edu>> wrote:
>>
>>     Hi Chris,
>>
>>     After some digging I was able to get to the bottom of this
>> and I am now
>>     getting results that I am happy with.  It was a combination of
>>     switching
>>     over to trunk and fixing a bad assumption in my VDIF
>> conversion that
>>     everyone represents four-bit complex integers in the same way.
>>
>>     Jayce
>>     --
>>     On 2/4/21 3:08 PM, Phillips, Chris (CASS, Marsfield) wrote:
>>      >    [EXTERNAL]
>>      >
>>      > Hi Jayce
>>      >
>>      > I'm working the data from the Long Wavelength Array and I am
>>     trying to see if we can move from our homegrown correlator to
>> DiFX.     The two modes I need to support are a LWA-only mode
>> with homogeneous
>>     data and an "eLWA" mode with heterogeneous data that combines
>> the
>>     LWA stations with the VLA 4-band system.  For the LWA-only
>> mode I
>>     have 4+4-bit complex voltage data from two 19.6 MHz wide
>> spectral
>>     windows.  I am currently converting these data from the LWA
>> format
>>     into two VDIF files, one per window, for correlation with
>> DiFX.  My
>>     .input file describes the data as:
>>      >>
>>      >> DATA FORMAT:        INTERLACEDVDIF/0:1
>>      >> QUANTISATION BITS:  4
>>      >> DATA FRAME SIZE:    7872
>>      >> DATA SAMPLING:      COMPLEX_DSB
>>      >> DATA SOURCE:        FILE
>>      >>
>>      >
>>      > I have never personally tested complex_dsb data with
>> interlaced
>>     vdif. I have no idea if there could be any “interaction” with
>> the
>>     VDIF interleaving and the specific data type (I would have
>> thought
>>     not, but you never know).
>>      >
>>      > The issue could also be with interpreting the interlaced
>> VDIF,
>>     not the complex double sideband.
>>      >
>>      > Once you have tested using trunk, not the released version, I
>>     would try and use some of the vdifio utilities to try and
>> convert
>>     the interlaced VDIF into plain VDIF - either extracting out one
>>     thread, or merging into multichannel single thread VDIF. You
>> would
>>     obviously need to update the DIFX .input (and or .v2d) file and
>>     vexfile to match the new setup.
>>      >
>>      > I would also be using the m5access tools to do things like
>> make
>>     the autocorrelation (both before and after fiddling with the
>> data
>>     layout). Check DIFX gives the same auto shape as you get with
>>     m5spec.  If you use a modulated noise source, folding the data
>>     (m5fold) on the period of the noise cal can be very informative
>>     also.  Also look at the headers and check that the #
>> frames/sec etc
>>     are what you expect (before and after merging threads etc).
>> If you
>>     decide it is the interlacing is the issue, I suspect you will
>> need
>>     to discuss with Walter.
>>      >
>>      >
>>      >> I am less sure about DiFX supporting our eLWA mode.  For
>> this we
>>     have the same 4+4-bit complex voltage data from the LWA
>> stations but
>>     with a bandwidth of 9.8 MHz.  We are combining this with
>> 4-bit real
>>     voltages from the VLA with 8 MHz of bandwidth and a slightly
>>     different frequency setup.  We currently process these data by
>>     shifting the center frequency of the LWA data to match that
>> of the
>>     VLA, running two F-engines with different channelization to
>> arrive
>>     at the same channel width for both data sources, and then cross
>>     correlate what overlaps.  Is this mode something that DiFX
>> supports?
>>      >>
>>      >
>>      > Can you represent this data as VDIF? If you you can probably
>>     handle the different setup using zoom bands. However if this
>> can be
>>     done will really depend on the EXACT setup. Basically you
>> need to be
>>     able to channelise the two setups with the same final frequency
>>     resolution (ie different FFT sizes will be needed, with the
>> ratio of
>>     the FFT size matching the ratio of the bandwidth difference.
>> But the
>>     sky frequency of the frequency points out of the FFT need to
>> match
>>     exactly (ie even if you could achieve the same frequency
>> resolution,
>>     if the fine channels are offset by 1/3 of a channel you will
>> have
>>     problems (if that offset is a simple integer ratio, you can
>> just use
>>     a higher frequency resolution)
>>      >
>>      > I hope this makes sense.
>>      >
>>      > Cheers
>>      > Chris
>>      >
>>
>>
>>
>> -- 
>> !=============================================================!
>> A/Prof. Adam Deller
>> ARC Future Fellow
>> Centre for Astrophysics & Supercomputing
>> Swinburne University of Technology
>> John St, Hawthorn VIC 3122 Australia
>> phone: +61 3 9214 5307
>> fax: +61 3 9214 8797
>>
>> office days (usually): Mon-Thu
>> !=============================================================!
>
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users