[daip] Problems with DiFX detected PCAL in AIPS.

[Craig Walker]

I started looking at a number of projects to try to see how widespread 
the issue is with the same frequencies for different tones.  I ran into 
a single polarization project which made me realize that I was not 
interpreting the tables correctly.  It seems that each row is not all of 
the tones in one baseband, but rather the data for all of the tones for 
one frequency over the polarizations for which it was measured.  This 
became clear, after a bit of thought, when the single tone project only 
had one set of measurements, not two.  Assuming that the columns are for 
the different polarizations, not frequencies, the DiFX data are probably 
ok and the behavior of PCCOR is understandable.  We still have some 
issues surrounding plotting of PCCOR delay results, but I think that is 
all that is left.

Sorry about the false alarm on the PC table front.

Cheers,

Craig


On 02/03/2014 08:50 AM, Walter Brisken wrote:
>
> I did find something that looks kind of funny. I'll spend some time
> invesigating and will let you know if it turns out to be a real error or
> not. In my quick peak I am finding it hard to see how it explains your
> symptoms (e.g., same frequency labels for different tones).
>
> -W
>
>
>
> On Sun, 2 Feb 2014, R. Craig Walker wrote:
>
>> I am trying to understand some strange behavior of the DiFX detected
>> pulse cal at LA (and other stations to a lesser degree). Basically
>> the DiFX detected tone phases displayed in AIPS show variations that
>> are not see in the on-site detected pcal data from the legacy system.
>>
>> Looking at the PC data in AIPS with PRTAB, I have found that the two
>> tones are billed as having the same frequency. But the phases are
>> different, so they probably really are from different frequencies as
>> they are supposed to be. Having the frequencies billed as the same
>> means that solving for IF delays from the tones should not be
>> possible. But I do have some solutions, although many if not most
>> solution intervals seem to have failed ("INDE" for real and imag and
>> "Infinity" for mbdelay and delay as displayed in PRTAB, but 1.0 for
>> the weights).
>>
>> Walter - what do you know about the tones claiming to have the same
>> frequency? This sounds like a fairly bad bug. I checked and this was
>> true for my runs in Jan 2013 and Dec 2013 with the PFB and in June
>> 2013 with the DDC.
>>
>> It is reasonably easy to fix the tone frequencies with TABED - just add
>> the difference frequency to the appropriate column. I did that using
>> a guess for the second frequency to see what would happen. That
>> created PC 2.
>>
>> I then reran VLBAPCOR. The help for VLBAPCOR says that it uses the
>> highest numbered PC table, which comes from using DEFAULT PCCOR which
>> gives INVER 0. But the messages from my first try claimed that table
>> 1 is used. But later I tried an explicit test with INVER 0 and it
>> seem to have worked, so who knows what happened.
>>
>> While at it, I'll note that DEFAULT TABED sets the outdisk to 1. I'm
>> using disk 5 and wanted my table to default to go on the input file,
>> so I almost got nailed. I'm not sure what was meant to happen there,
>> but I don't think defaulting to 1 is good.
>>
>> After the initial trouble getting PCCOR to use PC 2 using VLBAPCOR, I
>> ran it separately and forced the PC version to 2. This solution got
>> exactly the same DELAY as with PC 1. I even tried deleting PC 1 (I'm
>> using a test data set) and the solutions stayed the same, so it wasn't
>> a problem of using the wrong table. It seems that PCCOR's operation
>> is not sensitive to the frequency of the second tone. I don't
>> understand that. Anyone have a clue?
>>
>> The MBDELAY in the SN table is constant over 7 significant digits
>> throughout the run. That doesn't sound right if it is meant to be
>> based on the pcal data at least in part. The IF DELAYs do vary.
>>
>> So I haven't made much progress yet for trying to figure out if the LA
>> DiFX pcal issue shows just phase variations, indicative of an LO
>> issue, or a delay issue, indicative of a path length change, using
>> PCCOR. I don't understand the lack of sensitivity to the frequency of
>> the second tone and I don't understand why most of the solutions
>> failed.
>>
>> The SN table from PCCOR has single and multi-band delays as noted
>> above and as seen with PRTAB. When I try to plot the DELAY with
>> SNPLT, that program acts like they are all zero. You get the "TICINC:
>> POSITION ROUTINES FAIL ON THIS IMAGE" message and the plot has no
>> scale and a straight line of data. If you force the scale with PIXR,
>> the data continue to act like they are all zero. I am guessing that
>> the DELAY columns in the SN table from PCCOR are not being recognized
>> by SNPLT as delays, but that is only a wild guess. I'm sure I have
>> looked at delays from FRING with SNPLT in the past, so this is rather
>> curious. Is it possible?
>>
>> I suppose I could try the plot with TAPLT, but it has lots of inputs
>> and it's getting late. I'll try later. Time to quit for the day.
>> All of the above is on my Mac at home with a current AIPS installed
>> last week. I could make a stripped down data set if that would be
>> useful, but the current data cannot be seen from the AOC.
>>
>> Cheers,
>>
>> Craig
>>
>>
>>
>>>
>>> I looked into a couple things. First the weights. Contradicting my
>>> earlier proposed source of problem, KPs weights are a bit ratty but PCal
>>> is stable and the reverse is true at LA. I also looked a bit into the
>>> PCal code. Unless there are deep-rooted problems in the PCal extractor
>>> code in DiFX the only problem that I could imaging being excited with
>>> symptoms like we see would be related to dropping a an integer number of
>>> samples. At a sample rate of 64 Msps for your experiment one sample
>>> corresponds to almost 700 turns of phase (do you agree?). Given that I
>>> would expect the phase jumps to be effectively random as a function of
>>> baseband frequency and not consistently at the 10 deg level. It seems to
>>> me we are looking for a jitter somewhere around the 10 degree =
>>> picosecond
>>> level.
>>>
>>> Another thing to look into is the difference between the two tones's
>>> phase
>>> within a subband. That would distinguish between a glitching synthesizer
>>> and pathlength change. It would also be interesting to see if there is
>>> any signal left in the RCP and LCP pcal phase for the same tone.
>>>
>>> -Walter
>>>
>>>
>>> On Sat, 1 Feb 2014, Walter Brisken wrote:
>>>
>>>>
>>>> Looking at the AIPS plot I see your issue. There seems to be some
>>>> correlation between the cable cal variations at LA and the phase cal.
>>>> Do you
>>>> agree with that? But the fact the sniffer plot don't show this is
>>>> troubling.
>>>>
>>>> When we extract tones at the correlator we extract all of them but only
>>>> populate 2 per baseband into FITS files and then the raw values are
>>>> discarded. I'm starting to think we need to preserve them for a bit,
>>>> but the
>>>> files get quite large (text format, one row per antenna per integration
>>>> time).
>>>>
>>>> One thing that is a bit disturbing to me and could point to a DiFX
>>>> issue
>>>> is
>>>> that the pulse cal phases are constant or increasing with punctuated
>>>> jumps.
>>>> If samples being delivered to the pulse cal system are getting dropped
>>>> this
>>>> could be the result depending on how that subsystem actually works.
>>>> I'll
>>>> have to check on that.
>>>>
>>>> Can you see if data weight drops are correlated with these jumps?
>>>>
>>>> -W
>>>>
>>>> On Sat, 1 Feb 2014, R. Craig Walker wrote:
>>>>
>>>>> I think I'm having email problems. An attachment I sent to the
>>>>> geodesy
>>>>> guys got stripped off as a virus because it didn't like the extension
>>>>> - or
>>>>> maybe the fact that there were 3 periods in the name so it thought I
>>>>> was
>>>>> trying to hide an extension. Now this. I am basically sure I got
>>>>> those
>>>>> files attached before sending the mail. Anyway, I'll try again here.
>>>>> One of these has two periods (a gzip file) so maybe I'm being
>>>>> blindsided
>>>>> again. In the other case, the recipient had a place to go to see the
>>>>> reason for the stripped attachment. Do you see anything?
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> I think you failed to attach the attachment.
>>>>>>
>>>>>> I'm going into email darkness for most of today (driving from OVRO
>>>>> to
>>>>>> Pasadena). I'll try to have a look tonight.
>>>>>>
>>>>>> I can't think of any DiFX reason for problems. An obvious test I
>>>>> would
>>>>>> perform is to look at the phase stability implied by visibilities
>>>>> and
>>>>>> compare with pulse cal. If pulse cal is worse then there is
>>>>> something
>>>>>> real to look for. Perhaps could this be a result of integer MHz
>>>>>> sampling?
>>>>>>
>>>>>> -Walter
>>>>>>
>>>>>> On Sat, 1 Feb 2014, R. Craig Walker wrote:
>>>>>>
>>>>>>> Walter,
>>>>>>>
>>>>>>> I'll pass this by you first to see what you think. For my M87 run
>>>>> of
>>>>>>> Dec
>>>>>>> 27, the pulse cal measurements in the legacy system were nice and
>>>>>>> flat.
>>>>>>> However the ones with the data set, measured in the correlator,
>>>>> have
>>>>>>> considerable variation. See the attached plots.
>>>>>>>
>>>>>>> This is disturbing in what it says about the phase stability of
>>>>> either
>>>>>>> the
>>>>>>> RDBE or DiFX. But it is also disturbing because LA is where we
>>>>> plan
>>>>>>> to
>>>>>>> do
>>>>>>> the initial testing of the new synthesizer. If I saw variations
>>>>> like
>>>>>>> I
>>>>>>> see in the AIPS data, I would not be happy with the synthesizer.
>>>>> But
>>>>>>> the
>>>>>>> data here suggest that the synthesizer might not be at fault. Of
>>>>>>> course,
>>>>>>> the first round of tests will use only the legacy system so this
>>>>> won't
>>>>>>> bite us until later.
>>>>>>>
>>>>>>> LA is the worst case. But HN shows a phase ripple in the AIPS
>>>>> (DiFX)
>>>>>>> data
>>>>>>> but not in the legacy system. The others are reasonably flat in
>>>>> both.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Craig
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>>
>>>>>>> R. Craig Walker Array Operations Center
>>>>>>> cwalker at nrao.edu National Radio Astronomy Observatory
>>>>>>> Phone 575 835 7247 P. O. Box O
>>>>>>> Fax 575 835 7027 Socorro NM 87801 USA
>>>>>>> --------------------------------------------------------------web
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> R. Craig Walker Array Operations Center
>>>>> cwalker at nrao.edu National Radio Astronomy Observatory
>>>>> Phone 575 835 7247 P. O. Box O
>>>>> Fax 575 835 7027 Socorro NM 87801 USA
>>>>> --------------------------------------------------------------web
>>>>>
>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> R. Craig Walker Array Operations Center
>> cwalker at nrao.edu National Radio Astronomy Observatory
>> Phone 575 835 7247 P. O. Box O
>> Fax 575 835 7027 Socorro NM 87801 USA
>> --------------------------------------------------------------web
>>
>>

-- 
---------------------------------------------------------------------
     R. Craig Walker            Array Operations Center
     cwalker at nrao.edu           National Radio Astronomy Observatory
     Phone  575 835 7247        P. O. Box O
     Fax    575 835 7027        Socorro NM 87801   USA
---------------------------------------------------------------------