[daip] Re: Question about DELZN

[Leonia Kogan]

Hi Richard,

Having read your long message I can point out  your main questions/concern:

1.Can DELZN work with only one calibrator ?????
2. If it can why it fails with your data

I have put the plot of  one of the DELZN try into my web side 
www.aoc.nrao.edu/~lkogan
The file name is DELZN.PS

This is the plot of the polynomial (three term) into the zenith 
atmosphere for the three antennas together with the data
(given antenna -ref antenna.)

You see a rather good fitting.
You wrote:

>If one alters the number
>of parameters fitted for ATM one gets wildly different answers.

Of cource. You can fit the linear polynomial to these data with different parameters.
The fitting will be a little bit worse but still reasonable. So your following statement isong.

>Therefore I am sure that the minimisation for ATM is misbehaving. 

Looking into the data points at the plot, you can see some order.
Look at your data. If they look as a random points you can not expect any good fitting.
By the way the least square subroutine DLESQR can give a warning about the bad fitting

Now you main question:

1.Can DELZN work with only one calibrator ?????

I have always considered the answer is NO. Now I am not sure. I need to 
think.
A problem can appear for a small array when all antennas have almoust 
the same elevation at the given time.
The atmosphere contribution into the given antenn and the reference one 
are almoust identical at this case and therefore
the expected atmosphere difference VAL_ANT-VAL_REF is close to zero for 
any time!!

And finally, Mark Read as I know have used succesfully DELZN!!

LK

Richard Dodson wrote:

>Hi Leonia, 
>
>  
>
>>Let me explain you  what and how DELZN is doing.
>>    
>>
>
>Thanks
>
>  
>
>>Lets note that VAL_IANT -VAL_RANT are the values picked up from the SN
>>table as a function of time.
>>It can be multi band delay phase or ....
>>It is always supposed that the VALs are residuals.
>>    
>>
>
>Of course. But residuals can still contain several several
>contributions. 
>
>  
>
>>THE MAIN POSSIBLE ERRORS (such a geometry, initial model of the
>>atmosphere and clocks and....)  ARE COMPENSATED DURING CORRELATION.
>>THEREFORE  WE CAN CONSIDER THE VALs INCLUDES ONLY THE EFFECT OF THE
>>UNCOMPENSATED ATMOSPHERE AND CLOCKS.
>>    
>>
>
>OK. As I said, we are pushing DELZN. We don't have 500 MHz of bandwidth,
>nor multiple sources. In many ways we are not conforming to the defined
>paths. However one can not do new things on the defined paths, and new
>instruments can't necessarily follow them. I think by leaving the
>defined paths we are uncovering some new insights. 
>
>As an aside the correlator output of a phase referencing experiment, of
>course, does not include the geometry (of the target) as that is what is
>being sought. This does not therefore fit your requirements. This
>certainly effects the Sgr A* work of Mark Reid and the similar VERA
>projects. I take this as meaning that Mark does not use DELZN?
>
>Nevertheless, I continue. 
>
>  
>
>>Additionally  DELZN considers that the atmosphere behaviour is 
>>described as the function of senith angle elevation through
>>the mapping function as COSEC.
>>    
>>
>
>Yes (although we should watch out for the sec(Z)tan(Z)\Delta Z mapping
>function that is required for the phase referencing case, a la Mark
>Reid. It maybe worth adding this mapping function?). However, as you say
>DELZN will not fit for geometry (and I assume is not planned to do so?
>It would be some, but not huge, work to add such a shift to the
>minimisation. In this cases the clocks are zeroed so there are fewer
>parameters), so it is not really suitable for phase referenced images.
>Therefore the manner of its current use for VERA analysis is dubious,
>but should not fail so catastrophically. 
>
>Furthermore I believe it should be able to produce a sensible answer for
>the atmosphere from the reference source alone as it does fit
>tau_atmosphere as a function of time and as a time varying polynomial. 
>
>  
>
>>The SN table VALs are the difference of  antenna ANT and reference
>>antenna RANT. For simplicity lets consider that the reference antenna
>>
>>Therefore the the SN table values can be described by the following
>>equation:
>>
>>VAL_IANT) - VAL_RANT = ZENATM_IANT(time) *MAPFUNC(el)+
>>CLOCK_IANT(time)        (1)
>>                                                -ZENATM_RANT(time)
>>*MAPFUNC(el)
>>DELZN fits polynomials to the   ZENATM_IANT,  ZENATM_RANT    and
>>CLOCK_IANT
>>
>>Actually CLOCK_IANT = CLOCK_IANT - CLOCK_RANT
>>
>>So (for example) if the third order of polynomial (a*t+b*t^2+c) is
>>    
>>
>used
>  
>
>>for both  ZENATM_IANT and  CLOCK_IANT
>>then 6 (six) parameters should be fitted for each antenna. 3 for atm
>>    
>>
>and
>  
>
>>3 for clocks
>>You wrote:
>>
>>    
>>
>>> I believe there are 5 parameters to be fitted per antenna (for 3rd
>>> order polynomial for tau_a) (20 for VERA)
>>>      
>>>
>>I do not know why 5 and what is it (20 for VERA)?
>>    
>>
>
>The help file for DELZN recommends a linear polynomial for the clocks
>(aparm(3)) -- that is why I have 5 not 6 parameters per antenna. 4
>antenna gives 20 parameters, but of course the reference antenna clocks
>(as you point out) are not fitted, so 18. For the phase-only case the
>clocks are solved for on the reference source, so there are few
>parameters again. 
>
>  
>
>>I hope this message will help. 
>>    
>>
>
> It does, but does not answer the bigger questions. I know there are a
>lot of them! These are the most important I think, so I will repeat them
>;): 
>
> I believe that (for some reason) the minimization in DELZN is failing.
>Proof positive of this is the behaviour as one alters the number of
>parameters in the fit. The plot is showing VAL(IANT-RANT)+ATM(RANT)
>(clocks are zero). The data is shown scattered around the model values
>for ATM. ATM is what we get in the text file with OUTFILE set, which is
>what is shown in the plots I attached previous. If one alters the number
>of parameters fitted for ATM one gets wildly different answers.
>Therefore I am sure that the minimisation for ATM is misbehaving. 
>
> I agree that observations of multiple sources should/would avoid this
>problem. However I can not see why multiple observations of one source
>(with a range of elevations from 0 to 80) would not allow a solution for
>the atmosphere, particularly if you allow for a timing varying
>atmosphere.
>
> Can you tell me if there is fundamental problem with such a approach?
>Has it been tried? By who and did they have any comments?
>
> I do not want to have to extract the data from AIPS and try fitting
>externally, as this nullifies the point of what we are trying to achieve
>(i.e. astrometric reduction of VERA data within AIPS). Furthermore
>Honma-san has already succeeded in doing this. 
>
> I think I now understand what the plots are showing, but I would still
>like to see what an expected plot might show. (I would expect a roughly
>constant atmosphere with, perhaps, some linear trend.)
>
> The ATM polynomial is fitted against time, which is what one would
>expect. It is therefore the fitting that is problematic. What
>minimization method do you use? I find leasqr mentioned, but can not
>find where it is called from -- this is a bog standard matrix inversion
>and so one needs treat it with care. 
>
>In particular (From the code)
>C     Notes:
>C       1) Strictly speaking, the design matrix will usually contain
>C          rows of zeroes and therefore be singular.  This arises if no
>C          observations sensitive to a particular parameter have been
>C          done.
>C             In practice, any such singularities are ignored and the
>C          associated parameters remain undetermined.
>
> I have a gut instinct this is where we are hitting the problem. 
>
> And perhaps most importantly -- as it implies mis-use of DELZN can
>bugger up the data reduction -- do you think that DELZN as we are using
>it is just performing a selfcalibration? 
>
>
>
>
> I have some suggestions: 
>
>a) DELZN could possibly be used for the atmosphere correction from the
>phase of the CALIBed _target source_, if the mapping function was
>sec(z)tan(z)\Delta Z, rather than sec(z) as currently. The extra terms
>are smoothly varying, so a polynomial fit for a (constant) atmosphere
>could follow them -- however high order polynomials imply more unknowns
>than freedoms. 
> I could add this new mapping function. 
>
> But one would really want to solve for geometry at the same time -- a
>simple offset from the centre is all that is required. In this current
>approach the geometrical offset is _assumed_ to be identifiable in the
>dirty map. This was not the case for Sgr A* so may be a poor assumption.
>
> Addition of the geometrical terms would allow DELZN to be used for the
>phase residuals of phase referencing experiments. 
> 
>b) DELZN should work for the reference source only (i.e. a single
>source) with MB delays. There are a more unknowns (~5 per antennae) than
>observables (~the number of antennae) per time interval, but assuming
>solutions are independent (=significantly different elevation?) on a
>reasonably short (1 hour say?) time interval this should not be a
>problem. Has anyone tried DELZN for a single source with the VLBA? 
>
> As I said previously, this problem is hard to explain. 
>
>
>      Richard
>
>
>
>  
>
>>Richard Dodson wrote:
>>
>>    
>>
>>>Hi Amy, Leonia,
>>>
>>>Thanks for your quick answer to Maria and my questions. We have more.
>>>
>>>Also I started this email last week -- so somethings have been
>>>answered. I hope my re-writing has not introduced any incoherency.
>>>
>>>
>>>
>>>      
>>>
>>>>In an ideal world, what you did would work fine.  The problem is
>>>>        
>>>>
>that
>  
>
>>>>128MHz will, in most cases, be too small a bandwidth to obtain an
>>>>accurate multiband delay.  As you see in Figure 2 of Memo 110 a
>>>>        
>>>>
>narrow
>  
>
>>>>bandwidth can easily give errors of a few nsec, which is why your
>>>>multiband delays are large and give you nonsensical results from
>>>>        
>>>>
>DELZN.
>  
>
>>>>There is basically no way to fix your data, next time you have to
>>>>observe with a wider bandwidth.
>>>>
>>>>
>>>>        
>>>>
>>> With the VERA setup that is not possible, or more accurately not
>>>easy, to implement at this time. Therefore we are still looking to
>>>find some way to apply the correction. You will have to be patient
>>>with us, as we are stretching DELZN in ways that were never imagined
>>>or intended.
>>>
>>> What may be possible, or even necessary, is to introduce a short
>>>section to the VERA schedules with scans of multiple known
>>>calibrators. Currently there is a single scan of a fringe finder,
>>>      
>>>
>then
>  
>
>>>the dual beams track (continuously) the reference and the target
>>>sources.
>>>
>>> We are pretty confident, however, that the delay values should good
>>>enough to support this analysis. In fact it is being done at the
>>>moment (by others), by exporting the data and fitting externally.
>>>      
>>>
>What
>  
>
>>>we are trying to find is a route which is within AIPS.
>>>
>>> 128 MHz is only a factor of 4 less than your 500 MHz. The _scatter_
>>>in the solutions is less than 1 nsec (typically 0.3). The solutions
>>>      
>>>
>of
>  
>
>>>course include clock offsets and rates, so the actual values are much
>>>greater than this. However, if we have managed to understand your
>>>memo, this can be accounted for.
>>>
>>> However, where VERA will run into problems is the number of
>>>      
>>>
>freedoms
>  
>
>>>with the fitting routines. Recall that VERA has only 4 antennae, thus
>>>only 3 MultiBand delays.  VERA has H-masers clocks,  but my
>>>      
>>>
>experience
>  
>
>>>with the Mitaka correlator (with VSOP data I admit) is that they
>>>      
>>>
>don't
>  
>
>>>usually bother to (or need to) produce perfect clock models for the
>>>correlation run, so there could be considerable offsets and non-zero
>>>rates.
>>>
>>> Non-zero delays are also common with the VLBA. How have you removed
>>>these in your analysis? Or does the multi-band delay fitting mean the
>>>delays in the multiband column are somehow smaller than those in the
>>>single band delays. Memo 110 does not make this clear (to me ;).
>>>
>>>The values for the atmosphere we derive from the MB delays are very
>>>silly. The depth calculated is mega-millimeters! Now some questions.
>>>
>>>- Could you show us an example of a successful Atmospheric Zenith
>>>      
>>>
>Depth plot?
>  
>
>>>- When you say in the help file "a polynomial of the chosen degree
>>>      
>>>
>for
>  
>
>>>the zenith atmosphere delay" for the fitting against Zenith. Is this
>>>the fit to the point by point solution? I.e. the variation of delay
>>>with time, or is there a polynomial mapping function you are using?
>>>- What might have happened to destroy the fitting algorithm?
>>>
>>>Basically I am suspicious about the numbers of parameters in the fit
>>>and the consequences of it.
>>>
>>>You must be fitting
>>>
>>>\Delta \tau_{ij} = \tau_{atmos_i} sec Z_i - \tau_{atmos_j} sec Z_j +
>>>                                   \tau_{clock_i}  - \tau_{clock_j}
>>>
>>> Where \Delta \tau is the MBD (to reference j) and \tau_{clock} is
>>>the linearly varying clock and \tau_{atmos} is the polynomially (in
>>>time?) varying atmospheric delay. Z is the zenith angle.
>>>
>>>So plotting against elevation (fig4) is unlikely to show much unless
>>>one assumes that the clocks are zero, and the plot is dominated by
>>>      
>>>
>the
>  
>
>>>effects from one antenna?
>>>
>>> I believe there are 5 parameters to be fitted per antenna (for 3rd
>>>order polynomial for tau_a) (20 for VERA). There are {the number of
>>>antenna less one} delays per source and solution interval (3 per sol
>>>interval for VERA). This makes it clear why multiple sources should
>>>      
>>>
>be
>  
>
>>>scheduled. Depending on how the solution interval is defined and used
>>>it maybe possible to find solutions for our case. One would imagine
>>>that if one had (say) 7 independent solution intervals over the
>>>experiment one would be over determined for both of these kinds of
>>>delay.
>>>
>>>This is what I would expect, but this is what would appear to not be
>>>      
>>>
>happening.
>  
>
>>>We will experiment with trying to merge the calibrators into one
>>>      
>>>
>file
>  
>
>>>-- the sources are co-observed so have common antennae and time
>>>      
>>>
>stamps
>  
>
>>>-- but have not done this yet.
>>>
>>> Does this make sense? I find it hard to express clearly. But there
>>>is no doubt that we would prefer to use the cleaner MDEL solution as
>>>the phase solution is a can of worms we don't want to open. But we
>>>have.
>>>
>>>
>>> Part 2. The Phase based corrections
>>>
>>>
>>>
>>>      
>>>
>>>>I have never used the phase so I cannot tell you how accurate it
>>>>        
>>>>
>should be
>  
>
>>>>        
>>>>
>>>Oh dear. Is there some one we can ask about these? Mark Reid perhaps?
>>>I am aware that these questions are beyond the usual scope.
>>>
>>> Maria has successfully generated a DELZN solution from the CALIB
>>>phases, but this is (possibly) contaminated by the dual beam phase
>>>referencing approach used by VERA. It is important in this VERA
>>>check-out-phase to ensure that this is not the case. In particular we
>>>want to be sure that the expression used by M. Reid for the Sgr A*
>>>analysis is that used. I.e. for the _differential_ zenith delay. They
>>>were using the phases from the target source to calculate both the
>>>atmospheric corrections and the source offset from zero. The
>>>differential atmosphere delay correction is a function of \tau_0
>>>sec(Z) tan(Z) \Delta Z. (ApJ 524 p816). This is a very similar
>>>      
>>>
>problem
>  
>
>>>and you suggest in memo 110 that Mark `initiated' the DELZN approach.
>>>Does he use DELZN for this correction? In particular Mark fitted for
>>>the atmosphere and the geometry at the same time.
>>>
>>> But as far as we can tell DELZN does not fit for geometry. Is this
>>>      
>>>
>correct?
>  
>
>>>Anyway, In the phase case the clocks are definitely zero, and so do
>>>not need fitting for. The zenith depths calculated are a reasonable
>>>4cm or so. The solutions have an odd curvature tho'. I attach the
>>>plots of the values. Is this behaviour expected? I would have hoped
>>>for a constant -- or nearly constant -- value for each site.  Further
>>>more if one changes the order of the polynomial one would expect a
>>>similar result -- I.e. the zeroth order would be the mean of the
>>>higher order curves -- but this is not what we find.
>>>
>>> When we tried reducing the order of the polynomial we got very
>>>different solutions. What is _really_ being plotted as the routine
>>>runs. When we limit the number of terms for the atmosphere we get
>>>      
>>>
>very
>  
>
>>>different ranges of values plotted (aparm(2)=1 is a few millimeters
>>>(the attached zen_delay_0_order.ps), aparm(2)=2 is a some
>>>      
>>>
>millimeters,
>  
>
>>>aparm(2)=0 is a centimeters (the attached zen_delay.ps)). However at
>>>all times the line goes through the data. How can this be so? Is the
>>>data somehow different for each fit with different numbers of
>>>polynomial terms.
>>>
>>> The default solutions, however, do do a good job of correcting the
>>>image quality: Reduced sidelobes, larger peak. However I worry that
>>>this is very similar to what we would have got had one done a simple
>>>selfcal. How is this different from a selfcal, other than smoothing
>>>with a polynomial fit to the data?
>>>
>>> Thanks for your help with these many questions,
>>>
>>>                    Richard
>>>
>>>----------------------------------------
>>>Richard Dodson : Marie Curie Fellow
>>>email : r.dodson at oan.es
>>>Observatorio Astronomico Nacional
>>>Apartado 112 E-28803 Alcala de Henares
>>>Phone: (+34) 91-8855060/61, 62 (fax)
>>>----------------------------------------
>>>
>>>
>>>      
>>>