[daip] Problems with DiFX detected PCAL in AIPS.

[Walter Brisken]

I did find something that looks kind of funny.  I'll spend some time 
invesigating and will let you know if it turns out to be a real error or 
not.  In my quick peak I am finding it hard to see how it explains your 
symptoms (e.g., same frequency labels for different tones).

-W



On Sun, 2 Feb 2014, R. Craig Walker wrote:

> I am trying to understand some strange behavior of the DiFX detected
> pulse cal at LA (and other stations to a lesser degree).  Basically
> the DiFX detected tone phases displayed in AIPS show variations that
> are not see in the on-site detected pcal data from the legacy system.
>
> Looking at the PC data in AIPS with PRTAB, I have found that the two
> tones are billed as having the same frequency.  But the phases are
> different, so they probably really are from different frequencies as
> they are supposed to be.  Having the frequencies billed as the same
> means that solving for IF delays from the tones should not be
> possible.  But I do have some solutions, although many if not most
> solution intervals seem to have failed ("INDE" for real and imag and
> "Infinity" for mbdelay and delay as displayed in PRTAB, but 1.0 for
> the weights).
>
> Walter - what do you know about the tones claiming to have the same
> frequency?  This sounds like a fairly bad bug.  I checked and this was
> true for my runs in Jan 2013 and Dec 2013 with the PFB and in June
> 2013 with the DDC.
>
> It is reasonably easy to fix the tone frequencies with TABED - just add
> the difference frequency to the appropriate column.  I did that using
> a guess for the second frequency to see what would happen.  That
> created PC 2.
>
> I then reran VLBAPCOR.  The help for VLBAPCOR says that it uses the
> highest numbered PC table, which comes from using DEFAULT PCCOR which
> gives INVER 0.  But the messages from my first try claimed that table
> 1 is used.  But later I tried an explicit test with INVER 0 and it
> seem to have worked, so who knows what happened.
>
> While at it, I'll note that DEFAULT TABED sets the outdisk to 1.  I'm
> using disk 5 and wanted my table to default to go on the input file,
> so I almost got nailed.  I'm not sure what was meant to happen there,
> but I don't think defaulting to 1 is good.
>
> After the initial trouble getting PCCOR to use PC 2 using VLBAPCOR, I
> ran it separately and forced the PC version to 2.  This solution got
> exactly the same DELAY as with PC 1.  I even tried deleting PC 1 (I'm
> using a test data set) and the solutions stayed the same, so it wasn't
> a problem of using the wrong table.  It seems that PCCOR's operation
> is not sensitive to the frequency of the second tone.  I don't
> understand that.  Anyone have a clue?
>
> The MBDELAY in the SN table is constant over 7 significant digits
> throughout the run.  That doesn't sound right if it is meant to be
> based on the pcal data at least in part.  The IF DELAYs do vary.
>
> So I haven't made much progress yet for trying to figure out if the LA
> DiFX pcal issue shows just phase variations, indicative of an LO
> issue, or a delay issue, indicative of a path length change, using
> PCCOR.  I don't understand the lack of sensitivity to the frequency of
> the second tone and I don't understand why most of the solutions
> failed.
>
> The SN table from PCCOR has single and multi-band delays as noted
> above and as seen with PRTAB.  When I try to plot the DELAY with
> SNPLT, that program acts like they are all zero.  You get the "TICINC:
> POSITION ROUTINES FAIL ON THIS IMAGE" message and the plot has no
> scale and a straight line of data.  If you force the scale with PIXR,
> the data continue to act like they are all zero.  I am guessing that
> the DELAY columns in the SN table from PCCOR are not being recognized
> by SNPLT as delays, but that is only a wild guess.  I'm sure I have
> looked at delays from FRING with SNPLT in the past, so this is rather
> curious.  Is it possible?
>
> I suppose I could try the plot with TAPLT, but it has lots of inputs
> and it's getting late.  I'll try later.  Time to quit for the day.
> All of the above is on my Mac at home with a current AIPS installed
> last week.  I could make a stripped down data set if that would be
> useful, but the current data cannot be seen from the AOC.
>
> Cheers,
>
> Craig
>
>
>
>>
>> I looked into a couple things.  First the weights.  Contradicting my
>> earlier proposed source of problem, KPs weights are a bit ratty but PCal
>> is stable and the reverse is true at LA.  I also looked a bit into the
>> PCal code.  Unless there are deep-rooted problems in the PCal extractor
>> code in DiFX the only problem that I could imaging being excited with
>> symptoms like we see would be related to dropping a an integer number of
>> samples.  At a sample rate of 64 Msps for your experiment one sample
>> corresponds to almost 700 turns of phase (do you agree?).  Given that I
>> would expect the phase jumps to be effectively random as a function of
>> baseband frequency and not consistently at the 10 deg level.  It seems to
>> me we are looking for a jitter somewhere around the 10 degree = picosecond
>> level.
>>
>> Another thing to look into is the difference between the two tones's phase
>> within a subband.  That would distinguish between a glitching synthesizer
>> and pathlength change.  It would also be interesting to see if there is
>> any signal left in the RCP and LCP pcal phase for the same tone.
>>
>> -Walter
>>
>>
>> On Sat, 1 Feb 2014, Walter Brisken wrote:
>>
>>>
>>> Looking at the AIPS plot I see your issue.  There seems to be some
>>> correlation between the cable cal variations at LA and the phase cal.
>>> Do you
>>> agree with that?  But the fact the sniffer plot don't show this is
>>> troubling.
>>>
>>> When we extract tones at the correlator we extract all of them but only
>>> populate 2 per baseband into FITS files and then the raw values are
>>> discarded.  I'm starting to think we need to preserve them for a bit,
>>> but the
>>> files get quite large (text format, one row per antenna per integration
>>> time).
>>>
>>> One thing that is a bit disturbing to me and could point to a DiFX issue
>>> is
>>> that the pulse cal phases are constant or increasing with punctuated
>>> jumps.
>>> If samples being delivered to the pulse cal system are getting dropped
>>> this
>>> could be the result depending on how that subsystem actually works.
>>> I'll
>>> have to check on that.
>>>
>>> Can you see if data weight drops are correlated with these jumps?
>>>
>>> -W
>>>
>>> On Sat, 1 Feb 2014, R. Craig Walker wrote:
>>>
>>>>  I think I'm having email problems.  An attachment I sent to the
>>>> geodesy
>>>>  guys got stripped off as a virus because it didn't like the extension
>>>> - or
>>>>  maybe the fact that there were 3 periods in the name so it thought I
>>>> was
>>>>  trying to hide an extension.  Now this.  I am basically sure I got
>>>> those
>>>>  files attached before sending the mail.  Anyway, I'll try again here.
>>>>  One of these has two periods (a gzip file) so maybe I'm being
>>>> blindsided
>>>>  again.  In the other case, the recipient had a place to go to see the
>>>>  reason for the stripped attachment.  Do you see anything?
>>>>
>>>>
>>>>
>>>>>
>>>>>  I think you failed to attach the attachment.
>>>>>
>>>>>  I'm going into email darkness for most of today (driving from OVRO
>>>> to
>>>>>  Pasadena).  I'll try to have a look tonight.
>>>>>
>>>>>  I can't think of any DiFX reason for problems.  An obvious test I
>>>> would
>>>>>  perform is to look at the phase stability implied by visibilities
>>>> and
>>>>>  compare with pulse cal.  If pulse cal is worse then there is
>>>> something
>>>>>  real to look for.  Perhaps could this be a result of integer MHz
>>>>>  sampling?
>>>>>
>>>>>  -Walter
>>>>>
>>>>>  On Sat, 1 Feb 2014, R. Craig Walker wrote:
>>>>>
>>>>>>  Walter,
>>>>>>
>>>>>>  I'll pass this by you first to see what you think.  For my M87 run
>>>> of
>>>>>>  Dec
>>>>>>  27, the pulse cal measurements in the legacy system were nice and
>>>>>>  flat.
>>>>>>  However the ones with the data set, measured in the correlator,
>>>> have
>>>>>>  considerable variation. See the attached plots.
>>>>>>
>>>>>>  This is disturbing in what it says about the phase stability of
>>>> either
>>>>>>  the
>>>>>>  RDBE or DiFX.  But it is also disturbing because LA is where we
>>>> plan
>>>>>>  to
>>>>>>  do
>>>>>>  the initial testing of the new synthesizer.  If I saw variations
>>>> like
>>>>>>  I
>>>>>>  see in the AIPS data, I would not be happy with the synthesizer.
>>>> But
>>>>>>  the
>>>>>>  data here suggest that the synthesizer might not be at fault.  Of
>>>>>>  course,
>>>>>>  the first round of tests will use only the legacy system so this
>>>> won't
>>>>>>  bite us until later.
>>>>>>
>>>>>>  LA is the worst case.  But HN shows a phase ripple in the AIPS
>>>> (DiFX)
>>>>>>  data
>>>>>>  but not in the legacy system.  The others are reasonably flat in
>>>> both.
>>>>>>
>>>>>>  Cheers,
>>>>>>
>>>>>>  Craig
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>  ---------------------------------------------------------------------
>>>>>>     R. Craig Walker            Array Operations Center
>>>>>>     cwalker at nrao.edu           National Radio Astronomy Observatory
>>>>>>     Phone  575 835 7247        P. O. Box O
>>>>>>     Fax    575 835 7027        Socorro NM 87801   USA
>>>>>>  --------------------------------------------------------------web
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>  ---------------------------------------------------------------------
>>>>     R. Craig Walker            Array Operations Center
>>>>     cwalker at nrao.edu           National Radio Astronomy Observatory
>>>>     Phone  575 835 7247        P. O. Box O
>>>>     Fax    575 835 7027        Socorro NM 87801   USA
>>>>  --------------------------------------------------------------web
>>>>
>>>
>>>
>>
>
>
> ---------------------------------------------------------------------
>    R. Craig Walker            Array Operations Center
>    cwalker at nrao.edu           National Radio Astronomy Observatory
>    Phone  575 835 7247        P. O. Box O
>    Fax    575 835 7027        Socorro NM 87801   USA
> --------------------------------------------------------------web
>
>