[evla-sw-discuss] considerations toward an interim DCAF

[Ken Sowinski]

To be able to replace the Modcomps an interim version of Dcaf
(data capture in e2e parlance) is needed which creates archive 
records from the data presented by the old correlator.  This 
document outlines a proposal for how to create such a Dcaf.
'Archive Record' is not a term in the e2e glossary, but as it 
is used in the current VLA system to represent the final data
product, it is retained in describing the interim EVLA system.
'Interim' will always be used to mean observing with a combination
of VLA and EVLA antennas with the old correlator.  The words 'scan'
and 'subscan' will be used in the sense of the e2e glossary.  
Ultimately, software developed to support interim observing will be 
discarded. The rest of this document is a high level description of 
the work needed to be able to produce a working interim Dcaf; a more 
detailed description of the proposed changes to the archive record 
will be written in the next few weeks.

The primary requirement is that the interim Dcaf produce archive
records that differ as little as possible from the current archive
record in both syntax and semantics.  The current archive record
is described in VLA Computer Memo #188 and that memo is required
reading if you intend to understand this document.  Inputs to 
Dcaf are the correlator data stream and auxilliary data from the
Executor, Flagger and the Monitor Data Stream.  The output is the
archive record presented to at least one medium.  It is expected
that the archive record will be presented to the real time VLA
archive and to a tape device as a precaution against network or 
archive failure.  

More information than is currently available to the Executor is 
needed to create the archive record.  As the Executor learns how to
control the correlator more of the information will be available, but
not all.  The critical information lacking is the intent that is
currently embodied in .obs file fields such as cal code and observing 
mode, and observing sub-mode which is maintained in the observing 
system in realtime.  Obs2cript understands observing mode well enough 
to be able to generate a sequence of jython which can cause the antennas, 
and eventually the correlator as well, to carry out the actions required 
of any mode.  In the hybrid system now running, the Modcomp which is 
executing the .obs file is aware of intent and knows how to associate it 
with the correlator data to construct the archive record.  All this 
meta-information is lost in the combined obs2script and Executor 
implementations.  To remedy this will require extensive enhancements to 
both components.

obs2script
This program should really be called obs2py (or obs2jy) to allow for
a possible future executor which may accept some kind of script
which is not expressed in jython.  If that comes to pass we will need
translators for both systems for a while and it seems easier to deal
with if they have different names.  To deal with Dcaf obs2script
will have to generate a sequence of jython interspersed with the 
existing antenna control sequence which tells Dcaf what is happening.
this sequence will be ignored by the Executor except to pass it along 
to Dcaf.  Doing this will require a precise description of what the
Modcomps do with each of the observing modes we plan to support.  I
argue that we cannot take liberties with the implentation of the observing
modes because there are already existing downstream programs which depend
on the current implentation.

Executor
An obvious way to deal with this is to invent a dcafObject which
will mostly be a home for methods which can be used to send information 
to Dcaf.  We will send information as XML on sockets and do so at scan
and subscan events.  It is hoped that it will not be necessary to send
information at each integration, but a list of things that are expected
to change at integration events will be given below.  

Flagger
Flagger will have to learn to send information to Dcaf as well as the
Modcomps.  (Modcomps don't do multicast.)  The method and semantics of 
the communication can be private to Flagger and Dcaf.

Dcaf
Dcaf must be able to,
1.  Ingest correlator data.  To do this it needs to be able to determine
how much data to expect and when to expect it.  For each subarry it needs
to know how many antennas are in the subarray, Correlator Mode, BW, 
Hanning Flag, integration time.
2.  Ingest auxilliary information from Executor, Flagger and, very likely,
the Monitor Data Stream to be able to generate a representation of the
archive record.
3.  Create an archive record and publish it.  'Publish' can mean make it
available to one or more processes which will actually do I/O, or can
mean that Dcaf actually does the I/O.

Archive Record
For now I limit myself to listing the items in the archive record which
potentially change at each integration.  There are examples where the
cause of the change is very subtle.
RCA:  MJAD and IAT at the time the record is assembled.  This will be
      generated by Dcaf if it has a good enough idea of time.  
SDA:  Average (Integration) Time (*1)
      Stop Time (*2)
      Epoch and Apparant positions  (the former may change for moving objects)
      IAT, LST and geometry reference time
      Trig functions of az, el and parallactic angle
      Weather data captured during the integration
ADA:  Antenna control bits
      Flagging bits
      Nominal Sensitivity  (converts \rho to DekaJy)
      Peculiar phase (*3)
      Total Delay (*4)
      U, V and W (*5)
      FE and BE T_sys (*6)
      IF control bits  (I haven't examined these in detail,
                        they may be static)

*1  This can change during a scan if another subarray changes the
    integration time.
*2  This can change during a scan if the operator does an extend.
*3  This will change during the scan if the subarray is in a phased
    array mode.
*4  In addition to geometric changes, this will change during a scan 
    while in delay finding mode.
*5  When doing holography, U and V are usurped to represent the antenna 
    pointing offsets from the reference position.  This will change on
    subscan boundaries.
*6  FE T_sys will not be created for EVLA antennas.