[evla-sw-discuss] System design decisions needed for Widar

[Barry Clark]

A couple of Michael's comments deserve a recomment

Michael Rupen wrote:
> A few comments on Barry's discussion points...
> 
>> There are several issues to be decided about what we want the system
>> to look like in the long and intermediate term so we know what
>> general direction to take in the short run.
>>
>> 1.  Flags from the telescopes.
>> The long term route is clear - they should be collected by MCAF
>> and inserted into the SDM.  There are problems with doing this
>> in the short term because of a breakdown in the colaboration
>> with ALMA.
> 
> This is not quite fair.  The ball is very definitely in our court
> to propose a definition for the Flag Table.  Obviously that needs a 
> higher priority, and should not be that hard.  I'll try to put something 
> together
> this week.

I don't think we have time to go through a formal coordination with
ALMA, given the cycle time these have had in the past.  I think we need
to start something now.

[snip]

> 
>> 4.  Are correlation coefficients useful?
>> From a sufficiently high level abstract viewpoint, the answer is no.
>> Calibration is done by dividing each subband by the subband gain, and
>> then multiplying that whole combined spectrum by a number that converts
>> it to Janskys.  This number is derived by looking at the "best"
>> subband (least interference, best known cal) and dividing by the
>> measured cal power (that is, cal on minus cal off), and multiplying
>> by the Tcal for that subband.  This converts the measure correlations
>> into kelvin.  One then multiplies by the antenna effective area which
>> converts them into Janskys.  (but see footnote)  The power level out
>> of the requantizer nowhere enters in this description of calibration.
>> On the other hand, for various intermediate operations, particularly
>> in telcal, having correlation coefficients, derived by dividing the
>> measured correlations by the power out of the requantizer, really
>> simplifies things by not having to worry about changes in power level
>> across the time of its calculations.
> 
> I would phrase the above as "Calibration CAN BE done..." rather than
> "Calibration is [implying must be] done...".  Certainly for instance we
> do not currently use the effective area for most VLA calibrations, instead
> referencing to sources of known flux density.
> 
> Here are my arguments for the utility of "true" (rather than scaled)
> correlation coefficients:

The main points of my otherwise rather mad description of the
calibration process were that
1.  The "sideband stitching process" is nonsense.  In the extremely 
unlikely case Michael mentions of wanting to average across subbands,
one can just divide by the respective requantizer gains.  I'm dubious
enough that this might be useful that I'm inclined not even to worry
about getting requantizer gains to the CBE.
2.  The way we have been thinking about calibration is by dividing
chip outputs by the power into the chip (Psys) and, very shortly into
post-processing, dividing by Pcal/Psys.  I am suggesting that we do
the algebra first, noting that Psys^-1 * (Pcal/Psys)^-1 = Pcal^-1, and
send along the chip output and Psys, rather than C/Psys and Tcal/Psys.
It saves a lot of bother.

> 
> 1- Correlation coefficients are a standard data product which is easy to
>   explain, and for which many people already have a gut-level 
> understanding.

That's because they haven't thought about it deeply enough.  Correlation
coefficients can be even more confusing than chip outputs in some
cases.  In fact, the things we are used to calling correlation
coefficients are in fact not so.  What we make in delay space are
correlation coefficients.  The fourier transform of these correlation
coefficients is correlation coefficients only under fairly extreme
assumptions of a flat spectrum and low SNR.  But, if you want to label
them as 'correlation coefficients', merely dividing by 2.5 (or whatever
the magic number is for target power out of the requantizer) will get
them close enough that only real sticklers would notice the difference.

> 
> 2- The SSRs for both ALMA and the EVLA have always required that we store
>   the visibilities as correlation coefficients.  So does the BDF 
> definition.
>   Many other interferometers (including the current VLA and the VLBA)
>   report correlation coefficients.  There has to be a mighty strong reason
>   to move away from this, at this late date.
> 
>   - Both AIPS and CASA already happily deal with corr'n coefficients.
> 

As noted above, the software does not have to change to deal with chip
outputs and Pcal, although to be scrupulous about it, one might want
to change the column headings in listings.

> 3- TelCal really needs some uniform scaling across all baselines, which
>   at a minimum means correlation coefficients.  If we do not put these
>   directly into the BDF we'll have to send the scalings across to TelCal
>   and do them there.

I am unable to understand this.  Antsol is perfectly happy if different
antennas have different scalings.  What it needs is a little stability
across subscans, so we should not reset T304 attenuators or requantizer
gains at subscan boundaries.  Otherwise it is happy enough.
> 
> 4- If we do any averaging in the CBE (e.g., to give subscan averages to
>   TelCal, or to limit data rates) it's more proper to average correlation
>   coefficients.  [Barry points out that you've chosen your integration
>   time assuming that nothing changes on shorter timescales, so if things
>   do change very fast you've got bigger problems than improper averaging!]
> 
Nor do I understand this statement.  But if we tried to implement it,
measuring powers more frequently than the output integration time, it
would be a bear.

> 5- We don't need the zero lags (power levels) for anything else, so why
>   not use them where they're needed and then throw them away?
>   [I stand ready to hear counter-arguments on this point!]

I've made the stronger argument that we don't need them anywhere.
> 
> 
>> If we decide correlation coefficients are useful, it raises the question
>> of when they are calculated - in the CBE or later in the stream.  (If
>> this division is done in the CBE, we will have to have the power of
>> undoing it later, since, as noted above, correlation coefficients are
>> not used in the calibration process.)
> 
> I do not understand this.  Currently we and many others happily give
> corr'n coeffs, which we happily calibrate.  What is different about WIDAR?
> 
The different thing about EVLA/WIDAR is that there is no analog ALC in
the system.  We have failed to recognize the effect of that, like
generals fighting the last war.  We are going from a power based system
to a gain based system.

> 
> Another question is how we should derive the normalization to get to corr'n
> coeffs, assuming we want them.  Above I've implicitly assumed we're using
> the filter-based power measurements, which requires some care in ensuring
> everything (including data blanking) is treated identically e.g. when using
> recirculation.  Another approach would be to use the autocorrelation
> spectra directly, as calculated on the Baseline Board.
>>
[snip]