[Difx-users] Constraint on the FFT resolution to correlate the wide-band (e.g., 2 GHz) data

[Lupin Lin]

Problem:

In a simple test to correlate with the 2 GHz simulated data (4-sec duration in this example) of the flux in 2 Jy for CARMA (SEFD = 5), SMA (SEFD =10) and SMTO (SEFD =20), a strange trend can be found from the obtained result as shown in “HresFFT_2048MHz.pdf”.

In the correlation obtained for CARMA-SMA and SMA-SMTO, an obvious coherence lost following the increasing of the frequency can be seen on the figure.
But this trend does not appear on the correlation for CARMA-SMTO.
Such a problem may be related to the simulator (e.g., the computation with the earth rotation since this problem occurs on the correlation to the long baseline).
What’s the origin and how to fix it?

———————————————————————————————————————————————————
Adam’s response:

If you have a very large FFT and a high fringe rate, then the assumption that the fractional sample error remains constant for the duration of the FFT becomes incorrect.  This means that the frequency-dependent correction that is applied is also not exactly right, with the error being zero at the reference band edge and maximal at the other band edge.

One can figure it out: work out the duration of an FFT in seconds, multiply that by the fringe rate in ns/s, and divide the result by the sampling time in ns.   If that number is large then you're starting to get substantial decorrelation at the far end of the band.


In the high-resolution FFT plot for correlation (e.g., HresFFT_2048MHz.pdf):
The "FFTSpecRes = 0.00390625” (e.g., the duration for FFT is 2.56x10^{-4} sec) was determined in the .v2d file, so the pointer to examine the fractional sample error for 3 baselines are:
CARMA-SMA: 0.61
CARMA-SMTO: 0.10
SMA-SMTO: 0.71

With a delta fractional sample delay of 0.1 samples, at the high end of the band there will be sin(0.1)/0.1 ~ 99.8% correlation still.  But with 0.71, you have only 91.8% correlation.  That is not so far from what the plots seem to show. So I suspect if you reduce the spectral resolution by at least a factor of 8 you'll see matters improve greatly.

For normal continuum observations I would expect that a channel width of ~0.5 MHz would suffice.  That leads to an FFT length of 2 microseconds.  With a geometric rate of up to 1 microsec/s (maximum due to Earth rotation), then you have a delta geometric delay of 2e-12 seconds from one FFT to the next.  Since the sampling time for a 2 GHz band is 2.5e-10s, you're then looking at ~0.01 sample shifts per FFT, which is back within the range of negligible impact.

It is good to remember that the wider the input band to DiFX, the coarser the best attainable spectral resolution without running into this problem.  With a 2 GHz wide input band, you can't go much better than 62.5 kHz channels (at least on 8,000 km baselines) without accepting some decorrelation.  Whereas with a more typical band, like 128 MHz, you can go 16x narrower.  That is one of several moderately annoying limitations with very wide band inputs in DiFX (the others being mostly logistical, relating to chunk sizes, timekeeping etc).
  
———————————————————————————————————————————————————
Examination on the problem:

Here I put the revised correlated results here (LresFFT_2048M_Hz.pdf). 
To generate the new figure, I only changed "FFTSpecRes = 0.5”, which is 100x larger than the previous setting, and corresponding to the duration of 2x10^{-6} sec.
Then the previous trend, “a clear declination in the amplitude (obtained S/N) following the increasing frequency”, has disappeared in the correlation for the baseline of CARMA-SMA and SMA-SMTO.

———————————————————————————————————————————————————
Extensive Problem:

One more question is about the zoom-band mode for the correlation.
For example, we applied the zoom-band mode to examine the correlated result specified in a specific range (e.g., 1008-1040 MHz of the center band).
For such a case, we saw the similar trend as we correlate the entire full-band (2 GHz) data with the precise “FFTSepcres”.
(It means that S/N obtained at 4-36 MHz > it obtained at 1008-1040 MHz > it obtained at 2012-2044 MHz).
But in the zoom-band mode, the bandwidth is now 32 MHz, not 2048 MHz.
Or the mechanism to apply the zoom-band mode in correlation is still similar to correlate the full-band data, so I cannot set the FFT resolution too high. 

———————————————————————————————————————————————————
Adam’s response:

As the name implies, zoom bands "zoom" in on the recorded bands.  All the standard signal processing (including the fractional sample correction) is done on the recorded band first.  So a zoom band just picks out a segment of the frequency spectrum of a recorded band, including any artifacts that have been introduced in the signal processing up to that point.  So zoom bands are subject to the same limitations regarding fractional sample correction as their parent wide bands.

Getting around this would require adding a new filtering capability upstream of the standard station-based processing.  It would certainly be possible, and would be nice additional flexibility, but would be quite a bit of work.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20160824/db4aca45/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: HresFFT_2048MHz.pdf
Type: application/pdf
Size: 864651 bytes
Desc: not available
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20160824/db4aca45/attachment-0002.pdf>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20160824/db4aca45/attachment-0004.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LresFFT_2048MHz.pdf
Type: application/pdf
Size: 867855 bytes
Desc: not available
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20160824/db4aca45/attachment-0003.pdf>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20160824/db4aca45/attachment-0005.html>