[Difx-users] Constraint on the FFT resolution to correlate the wide-band (e.g., 2 GHz) data

Wed Aug 24 08:12:31 EDT 2016

Thanks for the succinct summary, Ludwig.

Generally speaking, this is not a problem, because one rarely wants to
bother with so many output channels.  The exception comes when a station
can only record wideband subbands due to backend limitations, but the user
still wants spectral line work, and decides to solve the problem with zoom
bands.  The moral of the story is to have at least a bit of flexibility in
the telescope backend!

I'll note for completeness that this is not a problem faced by SFXC, the
software correlator used by JIVE, because they use a separate forwards +
backwards FFT of fixed (small) length bracketing the fractional sample
correction step - it is complete decoupled from the final channelisation.
So they never run into this issue.  The price is two extra FFTs, which is
one of the main reasons why DiFX is a few 10s of percent faster than SFXC
for standard continuum correlations.

Cheers,
Adam

On Wed, Aug 24, 2016 at 2:00 PM, Ludwig Schwardt <schwardt at ska.ac.za> wrote:

> That's an interesting problem...
>
> Let N be the number of wide-band channels, Fs the sampling rate in Hz and
> Fr the fringe rate in seconds per second.
>
> The usual duration of an FFT in seconds = 2 * N / Fs
> The fringe rate in ns / s = 1e9 * Fr
> The sampling time in ns = 1e9 / Fs
>
> So the delta fractional sample delay becomes (2 * N / Fs) * (1e9 * Fr) /
> (1e9 / Fs) = 2 * N * Fr, which is independent of the total bandwidth and/or
> sampling rate.
>
> If you want to keep this delta below say 0.1 (thereby ensuring 99.8%
> efficiency), at the stated worst-case fringe rate of 1 microsec/s, this
> suggests that you cannot have more than 50,000 channels in your wide-band
> FFT, regardless of bandwidth.
>
> Best regards,
>
> Ludwig
>
>
>
>
> On 24 August 2016 at 10:56, Lupin Lin <lupin at asiaa.sinica.edu.tw> wrote:
>
>> Problem:
>>
>> In a simple test to correlate with the 2 GHz simulated data (4-sec
>> duration in this example) of the flux in 2 Jy for CARMA (SEFD = 5), SMA
>> (SEFD =10) and SMTO (SEFD =20), a strange trend can be found from the
>> obtained result as shown in “HresFFT_2048MHz.pdf”.
>>
>> In the correlation obtained for CARMA-SMA and SMA-SMTO, an obvious
>> coherence lost following the increasing of the frequency can be seen on the
>> figure.
>> But this trend does not appear on the correlation for CARMA-SMTO.
>> Such a problem may be related to the simulator (e.g., the computation
>> with the earth rotation since this problem occurs on the correlation to the
>> long baseline).
>> What’s the origin and how to fix it?
>>
>> ———————————————————————————————————————————————————
>> Adam’s response:
>>
>> If you have a very large FFT and a high fringe rate, then the assumption
>> that the fractional sample error remains constant for the duration of the
>> FFT becomes incorrect.  This means that the frequency-dependent correction
>> that is applied is also not exactly right, with the error being zero at the
>> reference band edge and maximal at the other band edge.
>>
>> One can figure it out: work out the duration of an FFT in seconds,
>> multiply that by the fringe rate in ns/s, and divide the result by the
>> sampling time in ns.   If that number is large then you're starting to get
>> substantial decorrelation at the far end of the band.
>>
>>
>> In the high-resolution FFT plot for correlation (e.g.,
>> HresFFT_2048MHz.pdf)
>>
>> :
>> The "FFTSpecRes = 0.00390625” (e.g., the duration for FFT is 2.56x10^{-4}
>> sec) was determined in the .v2d file, so the pointer to examine the
>> fractional sample error for 3 baselines are:
>> CARMA-SMA: 0.61
>> CARMA-SMTO: 0.10
>> SMA-SMTO: 0.71
>>
>> With a delta fractional sample delay of 0.1 samples, at the high end of
>> the band there will be sin(0.1)/0.1 ~ 99.8% correlation still.  But with
>> 0.71, you have only 91.8% correlation.  That is not so far from what the
>> plots seem to show. So I suspect if you reduce the spectral resolution by
>> at least a factor of 8 you'll see matters improve greatly.
>>
>> For normal continuum observations I would expect that a channel width of
>> ~0.5 MHz would suffice.  That leads to an FFT length of 2 microseconds.
>> With a geometric rate of up to 1 microsec/s (maximum due to Earth
>> rotation), then you have a delta geometric delay of 2e-12 seconds from one
>> FFT to the next.  Since the sampling time for a 2 GHz band is 2.5e-10s,
>> you're then looking at ~0.01 sample shifts per FFT, which is back within
>> the range of negligible impact.
>>
>> It is good to remember that the wider the input band to DiFX, the coarser
>> the best attainable spectral resolution without running into this problem.
>> With a 2 GHz wide input band, you can't go much better than 62.5 kHz
>> channels (at least on 8,000 km baselines) without accepting some
>> decorrelation.  Whereas with a more typical band, like 128 MHz, you can go
>> 16x narrower.  That is one of several moderately annoying limitations with
>> very wide band inputs in DiFX (the others being mostly logistical, relating
>> to chunk sizes, timekeeping etc).
>>
>> ———————————————————————————————————————————————————
>> Examination on the problem:
>>
>> Here I put the revised correlated results here (LresFFT_2048M_Hz.pdf).
>>
>> To generate the new figure, I only changed "FFTSpecRes = 0.5”, which is
>> 100x larger than the previous setting, and corresponding to the duration of
>> 2x10^{-6} sec.
>> Then the previous trend, “a clear declination in the amplitude (obtained
>> S/N) following the increasing frequency”, has disappeared in the
>> correlation for the baseline of CARMA-SMA and SMA-SMTO.
>>
>> ———————————————————————————————————————————————————
>> Extensive Problem:
>>
>> One more question is about the zoom-band mode for the correlation.
>> For example, we applied the zoom-band mode to examine the correlated
>> result specified in a specific range (e.g., 1008-1040 MHz of the center
>> band).
>> For such a case, we saw the similar trend as we correlate the entire
>> full-band (2 GHz) data with the precise “FFTSepcres”.
>> (It means that S/N obtained at 4-36 MHz > it obtained at 1008-1040 MHz >
>> it obtained at 2012-2044 MHz).
>> But in the zoom-band mode, the bandwidth is now 32 MHz, not 2048 MHz.
>> Or the mechanism to apply the zoom-band mode in correlation is still
>> similar to correlate the full-band data, so I cannot set the FFT resolution
>> too high.
>>
>> ———————————————————————————————————————————————————
>> Adam’s response:
>>
>> As the name implies, zoom bands "zoom" in on the recorded bands.  All the
>> standard signal processing (including the fractional sample correction) is
>> done on the recorded band first.  So a zoom band just picks out a segment
>> of the frequency spectrum of a recorded band, including any artifacts that
>> have been introduced in the signal processing up to that point.  So zoom
>> bands are subject to the same limitations regarding fractional sample
>> correction as their parent wide bands.
>>
>> Getting around this would require adding a new filtering capability
>> upstream of the standard station-based processing.  It would certainly be
>> possible, and would be nice additional flexibility, but would be quite a
>> bit of work.
>>
>>
>> _______________________________________________
>> Difx-users mailing list
>> Difx-users at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>>
>>
>
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>
>

-- 
!=============================================================!
Dr. Adam Deller
Ph  +31 521595785 / Fax +31 521595101
Staff Astronomer, Astronomy Group
ASTRON, Oude Hoogeveensedijk 4
7991 PD Dwingeloo, The Netherlands
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20160824/b72d009c/attachment.html>