[Difx-users] Question of the loss in Int. time and the phase shift obtained from the correlations
Lupin Lin
lupin at asiaa.sinica.edu.tw
Fri Jun 9 22:47:23 EDT 2017
Dear Adam,
I obtained the approval from Roger Cappallo, and I would like to post his comments following this post.
By Roger Cappallo:
—————————————————
I looked into a similar synthetic data correlation.
It had 2.000 seconds of data, but showed up on the fourfit plot as an integration
time of 1.984 sec, for both auto- and cross-correlations alike.
Looking into the model used, it turned out that the (single station
to earth center) delay was in fact about 16 ms, consistent
with Adam’s hypothesis about the propagation time delay
being the culprit.
Digging into the correlator output (Swinburne file) data, though,
it turns out that the data weight for the 1st second was
0.984, while the weight for the 2nd second was 1.000, for
a sum of 1.984. fourfit uses these weights to calculate integration
time, thus the final result of 1.984 seconds.
So, for my case it appears that the artifact is indeed coming
from difx, and not from the hops part of the data-processing chain.
I suspect the same is true for your example, though a bit of
digging through your files is required to prove it. More simply,
though Adam might be able to explain why the weight of the
first AP is decreased by the earth-center propagation delay.
For completeness, here are two different ways to find the
necessary evidence, either in the difx or hops world:
****** difx-based (e.g. for fileset h_1000):
propagation delay can be found in the h_1000.im file as
the first term of the quintic delay spline, say by grepping
for DELAY:
grep DELAY h_1000.im
data weights can be found by inspecting the Swinburne file
in the .difx directory, e.g. by using the utility program dumpvis
to create an ASCII copy, and then grepping for weight
****** hops-based (e.g. for fileset suffix uvwxyz and baseline AB):
propagation delay splines are in the type 301 records,
and can be examined with:
CorAsc2 301 < A..uvwxyz | grep delay
data weights are in the type 120 records, and can be
looked at with:
CorAsc2 120 < AA..uvwxyz | grep weight
dumpvis actually resides in the hops part of the directory structure.
It should be accessible in the hops executables area. For example,
on my system that would be $HOPS_ROOT/i686-3.15/bin
dumpvis is a "quick and dirty” utility, which shows up
in the fact that you have to specify the number of visibilities,
instead of having it infer it from the data. A typical run might be
dumpvis <long-swinburne-file-name> 128 | less
or of course redirecting into a file for later inspection.
The number of visibility points, 128 in this example, is half of
the number of lags at the bottom of the fourfit plot. You
could also find it empirically by trying different powers of
2, and seeing which one doesn’t cause a hiccup at the
2nd record (this is easier and quicker than it sounds).
Rather than looking at the Swinburne file you could of
course use CorAsc2 to inspect the type 120’s.
—————————————————
Cheers,
--
Lupin Chun-Che Lin
Supporting Scientist of GLT (GreenLand Telescope) project
in Institute of Astronomy and Astrophysics, Academia Sinica,
14F of Astronomy-Mathematics Building (Rm: 1405),
National Taiwan University.
No.1, Sec. 4, Roosevelt Rd, Taipei 10617, Taiwan.
Tel: +886-2-2366-5464
Fax: +886-2-2367-7849
> Adam Deller <adeller at astro.swin.edu.au> 於 2017年6月10日 上午9:39 寫道：
>
> Hi Lupin,
>
> For the benefit of the whole difx-users list, I'm copying into this thread my response to you that expanded on the potential cause of reduced weight of the first integration:
>
> Data files are recorded at the station in the station's reference frame. Correlation is performed in a geocentric reference frame. If the correlation is to start at time X, the first sample that will be requested by the correlator from the station for the correlation will be at time X-\tau_g, where \tau_g is the geometric delay from the station to the geocentre (to obtain the sample that would have arrived at time X on a plane that passes through the geocentre and is perpendicular to the source direction). And if the station data file starts at exactly time X (same time as the requested correlation start, at the geocentre), then the first \tau_g samples won't be available - they are from before the station started recording. And hence the completeness of the first integration will be less than 1.
>
> If the station started recording one or more seconds prior to the start of the correlation time, then the necessary data will be available and the first integration will have full weight.
>
> This is not an error on the part of the correlator: it is expected behaviour, and an unavoidable result of time-tagging the visibilities at the geocentre properly. In the astronomical VLBI schedules that I am familiar with, the scheduling software generally forces the stations to start recording a few seconds prior to the intended scan start (or in the event of a small gap between scans, they never stop recording at all), and this issue doesn't arise.
>
> Cheers,
> Adam
>
>
> On 24 May 2017 at 12:21, Lupin Lin <lupin at asiaa.sinica.edu.tw <mailto:lupin at asiaa.sinica.edu.tw>> wrote:
> Dear Adam,
>
> Thanks for your response again.
>
>> But so you've got two datastreams which you've generated from noise sources, is that right?
>
> Yes.
>
>> They have independent noise and a common signal which has a small but non-zero delay?
> Yes
>
>> And then you're putting both data streams into DiFX and saying they are from the same physical position, so they get the same delay model applied. How are you calculating the single phase number? Is that a simple average over the whole band?
> The phase value was given by the result shown by fourfit/HOPS.
> And it "should be” the average of phase distribution shown in Avgd. Power Spectrum over the whole band (Please refer to the attached Zoom_128_5sec.pdf.
>
> According to the result shown for the cross-correlation, the phase value is -115.4 degree. It is the phase number indicated in my previous “phase.pdf”.)
>
>> What if you reduce the amplitude of the common noise source to something a bit more realistic, like a maximum of 1% of the independent noise amplitude?
> Thanks for the comments, and we will try it.
>
>> I don't have an explanation for why the phase differs between an LO of 86000 MHz vs 220000 MHz when you're extracting a 128 MHz subband, but I can only point the finger at quantisation + numerical precision effects coupled with the very high degree of correlation that appears to be present in your setup.
> "220 GHz" should be a typo because I set for 230 GHz.
> Is there any experiment that we can/should do to further confirm the origin of my problem?
>
>
> Sincerely,
> --
> Lupin Chun-Che Lin
> Supporting Scientist of GLT (GreenLand Telescope) project
> in Institute of Astronomy and Astrophysics, Academia Sinica,
> 14F of Astronomy-Mathematics Building (Rm: 1405),
> National Taiwan University.
> No.1, Sec. 4, Roosevelt Rd, Taipei 10617, Taiwan.
> Tel: +886-2-2366-5464 <tel:+886%202%202366%205464>
> Fax: +886-2-2367-7849 <tel:+886%202%202367%207849>
>
>
>> Adam Deller <adeller at astro.swin.edu.au <mailto:adeller at astro.swin.edu.au>> 於 2017年5月22日 上午10:39 寫道：
>>
>> Hi Lupin,
>>
>> OK, I think I partly understand now. But so you've got two datastreams which you've generated from noise sources, is that right? They have independent noise and a common signal which has a small but non-zero delay? And then you're putting both data streams into DiFX and saying they are from the same physical position, so they get the same delay model applied. How are you calculating the single phase number? Is that a simple average over the whole band? The changing amplitudes due to the requantisation plus the phase ripple could easily account for some of the differences. It also seems like your S/N is absurdly high, which is probably exacerbating a lot of the quantisation-related effects. What if you reduce the amplitude of the common noise source to something a bit more realistic, like a maximum of 1% of the independent noise amplitude?
>>
>> I don't have an explanation for why the phase differs between an LO of 86000 MHz vs 220000 MHz when you're extracting a 128 MHz subband, but I can only point the finger at quantisation + numerical precision effects coupled with the very high degree of correlation that appears to be present in your setup.
>>
>> Cheers,
>> Adam
>>
>>
>> On 22 May 2017 at 12:12, Lupin Lin <lupin at asiaa.sinica.edu.tw <mailto:lupin at asiaa.sinica.edu.tw>> wrote:
>> Dear Adam,
>>
>> Thanks for your repsonse to give the indication.
>> For the 1st question, I will further check with the expert of HOPS developer.
>>
>> For my 2nd question, the Jan’s code is “m5subband” provided in the latest DiFX version to extract the specific frequency range information from a wide/full band data.
>> The original input is also the 2-bit VDIF file.
>> I just compare with "the cross-correlation with two 2-bit wide band VDIF files at a specific frequency range using the zoom-band mode" and "one data subtracted a the same frequency range to cross-correlate another wide band file supported with zoom-band mode.”
>> A simple expression in the .v2d file is the comparison between:
>> (XXXXX and OOOOO are data with 2 GHz, and SSSSS is a data with the band of 256 MHz subtracted from XXXXX)
>> ————————————————————————
>> ANTENNA G1
>> {
>> file = XXXXX.vdif
>> format = VDIF/8224/2
>> phaseCalInt = 0
>> addZoomFreq= freq at 230000.00/bw at 256.0
>> }
>> ANTENNA G2
>> {
>> file = OOOOO.vdif
>> format = VDIF/8224/2
>> phaseCalInt = 0
>> addZoomFreq= freq at 230000.00/bw at 256.0
>> }
>> ———————————————————————— And,
>> ANTENNA G1
>> {
>> file = SSSSS.vdif
>> format = VDIF/8224/2
>> phaseCalInt = 0
>> }
>> ANTENNA G2
>> {
>> file = OOOOO.vdif
>> format = VDIF/8224/2
>> phaseCalInt = 0
>> addZoomFreq= freq at 230000.00/bw at 256.0
>> }
>> ————————————————————————
>>
>> The loss in SNR or the amplitude is expected due to the re-sampling (FFT and IFFT for the original XXXXX.vdif) of the data, but we do not clearly understand the origin to cause the "possible shift" in the obtained phase distribution.
>> (We suspect the shift in phase is due to the adoption of the Ref. Freq., but the series of my tests cannot confirm this point.)
>>
>> The motivation to do it is that we have limited/very slow internet speed in GLT.
>> In order to check the preliminary result via the obtained observation, we would like to send only the band reduced data with a much smaller file size.
>> So to investigate/handle the comparison between the aforementioned two cases is required for GLT team.
>>
>> Sincerely,
>> --
>> Lupin Chun-Che Lin
>> Supporting Scientist of GLT (GreenLand Telescope) project
>> in Institute of Astronomy and Astrophysics, Academia Sinica,
>> 14F of Astronomy-Mathematics Building (Rm: 1405),
>> National Taiwan University.
>> No.1, Sec. 4, Roosevelt Rd, Taipei 10617, Taiwan.
>> Tel: +886-2-2366-5464 <tel:+886%202%202366%205464>
>> Fax: +886-2-2367-7849 <tel:+886%202%202367%207849>
>>
>>
>>
>>> Adam Deller <adeller at astro.swin.edu.au <mailto:adeller at astro.swin.edu.au>> 於 2017年5月22日 上午8:15 寫道：
>>>
>>> Hi Lupin,
>>>
>>> On 19 May 2017 at 13:22, Lupin Lin <lupin at asiaa.sinica.edu.tw <mailto:lupin at asiaa.sinica.edu.tw>> wrote:
>>> To the experienced user of DiFX or HOPS,
>>>
>>> Sorry to spend you some time on two questions.
>>> The first one is related to the integration time presented on the final plot after the auto- or cross- correlations.
>>> (The DiFX I used is trunk ver. and and the HOPS ver. I used is 3.13)
>>>
>>> 1.
>>> Here the GLT team recorded two signals with 3 noise sources (only 1 is the same with a potential delay less than 1 ns) from the same ROACH, and made a test in correlation.
>>> We simply assumed that the Sky (Ref.) frequency is 230 GHz and the data were obtained from the same site (both G1 and G2 were assigned to the same position of GLT).
>>> We correlated the data of 5-sec. supported with the zoom-band mode at 1,200-1,328 MHz.
>>> ————(Contents in .v2d file)———
>>> vex = ext.vex.obs
>>> mjdStart = 2017y124d05h42m00s
>>> mjdStop = 2017y124d05h42m05s
>>> antennas = G1,G2
>>> startSeries = 7000
>>> dataBufferFactor = 16
>>> visBufferLength = 80
>>> nDataSegments = 8
>>> minLength = 0.5
>>> ————————————————
>>> But according to the attached figure of “Zoom128_5sec.pdf”, the int time is only 4.995 sec.
>>> What’s the reason to cause a loss of the obtained Int. time?
>>>
>>>
>>> This may be HOPS-related: I'm not an expert on HOPS so I can't comment. But another possibility is that the correlation runs from 054200 to 054205 at the geocentre, and if the recorded data duration was only 5 seconds long but at the telescope, then the geometric delay will mean than a few milliseconds of data will be chopped off. Maybe one of the HOPS users can chime in on how that integration time number is calculated in the HOPS plot - is it based on the weight? If so then my hypothesis is plausible.
>>>
>>>
>>> 2.
>>> Following the previous test, I used Jan Wagner’s code to do the band subtraction (the original full band is 2GHz) and checked the obtained phase of different cases.
>>> I summarized my tests and my questions in the attached file “phase.pdf”.
>>>
>>>
>>> The 1st case is the same as shown in the attached “Zoom128_5sec.pdf”.
>>> What’s the possible origin to cause the phase shift?
>>> I originally suspect that such a shift is originated from the change in Sky/Ref. freq, but it might be wrong….
>>>
>>>
>>> I'm sorry, but I can't follow what your are trying to do here. I'm not sure what Jan's code is that you're referring to (is it something that filters e.g. your 2 GHz band down to 128 MHz in a chosen window?) If that is the case, what is the input - already quantised 2 bit VDIF data? That will lead to nasty re-quantisation errors - you'll notice that the amplitude goes down, too.
>>>
>>> But the root of the problem is: what are you actually ultimately trying to achieve? A verification of some kind?
>>>
>>> Cheers,
>>> Adam
>>>
>>> Sincerely,
>>> --
>>> Lupin Chun-Che Lin
>>> Supporting Scientist of GLT (GreenLand Telescope) project
>>> in Institute of Astronomy and Astrophysics, Academia Sinica,
>>> 14F of Astronomy-Mathematics Building (Rm: 1405),
>>> National Taiwan University.
>>> No.1, Sec. 4, Roosevelt Rd, Taipei 10617, Taiwan.
>>> Tel: +886-2-2366-5464 <tel:+886%202%202366%205464>
>>> Fax: +886-2-2367-7849 <tel:+886%202%202367%207849>
>>> _______________________________________________
>>> Difx-users mailing list
>>> Difx-users at listmgr.nrao.edu <mailto:Difx-users at listmgr.nrao.edu>
>>> https://listmgr.nrao.edu/mailman/listinfo/difx-users <https://listmgr.nrao.edu/mailman/listinfo/difx-users>
>>>
>>>
>>>
>>>
>>> --
>>> !=============================================================!
>>> Dr. Adam Deller
>>> ARC Future Fellow, Senior Lecturer
>>> Centre for Astrophysics & Supercomputing
>>> Swinburne University of Technology
>>> John St, Hawthorn VIC 3122 Australia
>>> phone: +61 3 9214 5307 <tel:+61%203%209214%205307>
>>> fax: +61 3 9214 8797 <tel:+61%203%209214%208797>
>>>
>>> office days (usually): Mon-Thu
>>> !=============================================================!
>>
>>
>>
>>
>> --
>> !=============================================================!
>> Dr. Adam Deller
>> ARC Future Fellow, Senior Lecturer
>> Centre for Astrophysics & Supercomputing
>> Swinburne University of Technology
>> John St, Hawthorn VIC 3122 Australia
>> phone: +61 3 9214 5307 <tel:+61%203%209214%205307>
>> fax: +61 3 9214 8797 <tel:+61%203%209214%208797>
>>
>> office days (usually): Mon-Thu
>> !=============================================================!
>> _______________________________________________
>> Difx-users mailing list
>> Difx-users at listmgr.nrao.edu <mailto:Difx-users at listmgr.nrao.edu>
>> https://listmgr.nrao.edu/mailman/listinfo/difx-users <https://listmgr.nrao.edu/mailman/listinfo/difx-users>
>
>
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu <mailto:Difx-users at listmgr.nrao.edu>
> https://listmgr.nrao.edu/mailman/listinfo/difx-users <https://listmgr.nrao.edu/mailman/listinfo/difx-users>
>
>
>
>
> --
> !=============================================================!
> Dr. Adam Deller
> ARC Future Fellow, Senior Lecturer
> Centre for Astrophysics & Supercomputing
> Swinburne University of Technology
> John St, Hawthorn VIC 3122 Australia
> phone: +61 3 9214 5307
> fax: +61 3 9214 8797
>
> office days (usually): Mon-Thu
> !=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20170610/70c8abed/attachment-0001.html>
More information about the Difx-users
mailing list