[evla-sw-discuss] Notes from Scripting Walk-through Meeting

[Barry Clark]

Notes from Boyd Waters, expanded and beautified in a few cases by me.  Lines
without the leading '>' I've messed with.

> From bwaters+ole at aoc.nrao.edu Thu Feb 13 16:20 MST 2003
> From: "Boyd Waters" <bwaters+ole at aoc.nrao.edu>
> To: "Boyd Waters" <bwaters at cv3.cv.nrao.edu>
> Cc: "Tim Cornwell" <tcornwel at zia.aoc.NRAO.EDU>,
>    "Ken Sowinski" <ksowinsk at zia.aoc.NRAO.EDU>,
>    "Barry Clark" <bclark at zia.aoc.NRAO.EDU>,
>    "Bill Sahr" <bsahr at cv3.cv.nrao.edu>,
>    "Rich Moeser" <rmoeser at cv3.cv.nrao.edu>
> Subject: Notes from Scripting Walk-through Meeting
> Date: Thu, 13 Feb 2003 16:20:40 -0700
> 
> Scripting Walk-Through Meeting, 2/13/2003 4:00:00 PM
> 
The Antenna object sets up the hardware in the antenna, via the AntennaPhysical
object.  It also sets up the stuff in the correlator front-end - the
station board.  The Subarray object sets up the correlator back-end.
Does the correlator object knows about the antenna? Or the other way round?
> Ken has been thinking that the antenna knows nothing of the correlator.
Bclark says that the AntennaPhysical sends the station boards the delay polynomial,
and sets up the crossbar switch on the station board.
Ken:  Where is system temperature applied to data?  Barry:  Maybe not until data
reduction software.
> You can hide or expose various things
> 
> 
> ANTENNA VS. ANTENNA PHYSICAL
> What's interesting about those example scripts is that there is nothing
> antenna-specific in them. It's very rare that you will need to tell a
> particular antenna to do something. Ken means knowing that it's an EVLA
> antenna rather than a VLBA antenna.
> Barry says that you have to be a little bit aware of these differences; but
> not too much.
> Antenna-physical will of course know about the actual antenna in order to do
> the actual antenna setup. Antenna-physical picks up the 
state of the antenna hardware, and knows how to issue commands to make the
hardware correspond to the desired state.
> 
> Rmoeser says: can we just have one class? 
instead of both Antenna and AntennaPhysical.  Barry says it is convenient 
to separate them for subarraying.  The script interpreter then doesn't 
need to know what antennas are in the subarray it is controlling, and an
AntennaPhysical can conveniently be added to the subarray at any time and 
end up properly initialized.
> Can "antenna" be a generic
> interface?
Barry: Python does not have the concept of Interfaces.  It can operate
on an object in the java world, or it can import a java class and make
objects of its own of that class, which are then a little messy and 
inconvenient to access from the java world. But it has no concept like
a java Interface - it deals with objects.
> It has to instatiate
> something.
> Boyd: I don't believe you. Let's take a look at this. Anyhow perhaps it is
> not important.
> 
> Interpreter will be started with Antenna objects in it. The antenna object
> is a better place than the antenna-physical to store such things as the
> model params and the pointing. So the Java world will create the antenna
> objects, and then make it known to the Python interpreter thread (jython
> runs in-process).
> 
> Rmoeser says: did you ever consider making the control script an XML file?
> Barry: it's very wordy to do so; this way, you can sit down at a terminal,
> connect your terminal to the interpreter, type a few lines and have the
> array go do something useful.
Current VLA has a lot of special files with standard setups, which the operator
can switch to to find out if the array is misbehaving or if the astronomer
has screwed it up.  This facility could replace that, with additional clarity
and flexibility both.
> Boyd: e2e will need markup of intent.
> Barry: you'll note that we are waiting for feedback from e2e about what is
> needed.
> Rich: xml and script are two different things. 
> Barry: there are ways of using the python that are just as transparent as
> using the xml.
> Rich: if we just look at the xml as the data that's necessary, then we have
> an xml interpreter in m&c. we can restrict changes to the API to be outside
> the M&C system in this way.
> Barry: we think that we can maintain the same separation in python.
> Barry: In XML you start out with making stupid mistakes; in Python we're
> dealing with a much higher level of abstraction, so a blank page in python
> is at a place that you'd get with a sophisticated editor tool in XML. So in
> python we start out by making sophisticated mistakes instead.
> 
> SUBARRAYS
> 
> Ken: I have a hard time with one script dealing with multiple sub-arrays.
> Barry: Yes, this is going to be very difficult for the observing tool to get
> its head around.
Ken:  In the long term we are talking about having the WIDAR correlator
process recorded VLBI data while also doing eVLA correlating.  So there
have to be two separate executives controlling the correlator at the 
same time.
Barry:  I hope it can be separated so cleanly.
> Ken: There is a set of resources that can be shared by sub-arrays.
> Barry: well, it's divided in a different dimension...
> 
> POLYNOMIALS
> Barry: the data rate for polynomials is surprisingly low; three-terms for
> ten seconds is adequate; it's marginal (probably all right) for a global
> array, though hopeless for a spacecraft.
> 
> SCHEDULING BLOCKS
> Ken: the re-packaging of the intent of the observation into the scheduling
> block will be handled by the system.
> 
> Barry: well, let's go there:
> 1) PI submits PROPOSAL
2) TAC makes PROJECTS from PROPOSALS
3) a PROJECT may encompass observations in more than one array configuration.
a PROGRAM is that part of a PROJECT done in a single configuration.
> 4) associated with a PROGRAM, the OT generates SCHEDULING BLOCKS which
> consist of a CONTROL SCRIPT and "a lot of other stuff", such as an iteration
> count, how many times its already been run, sensitivities...
> 5) a SESSION sort-of sits outside the hierarchy here; a SESSION is a
> collection of EXECUTIONS of SCHEDULING BLOCKS, contiguous in time, within
> the same PROGRAM. There may be only one SCHEDULING BLOCK in a SESSION, there
> may be multiple executions of SBs in a SESSION, or 
a single SB may be executed in multiple SESSIONs.
> 6) within the CONTROL SCRIPT inside the SB, there is the notion of a SCAN:
> an observation that is trying to accomplish a particular purpose. There are
> POINTING SCANS, MOSAICING SCANS... the SCAN may consist of multiple
> OBSERVATIONS.
> 7) an OBSERVATION is an antenna at a particular configuration, looking at a
> particular thing. (May include antennas moving uniformly while the phase center
> does not change.
> 8) INTEGRATION
> 9) CORRELATOR DUMP (might be faster than integration rate).
> that's about as far as I care to go... there's pulsar binning...
> 
> Ken: the distinction between OBSERVATION and SCAN is a bit artificial, I
> think.
> Barry: it matters to the guy who is reducing that to produce a pointing
> offset. The m&c does not care; only knows about OBSERVATIONS (and deep in
> its guts it knows about INTEGRATIONS and DUMPS).
> 
> SCHEDULING TOOL
> Is the thing that decides which is the next thing to do. The M&C system does
> not need to care.
> Barry: for the EVLA, it may be sufficient to look at the site test
> interferometer to find out when the phase is going to pot. We have a much
> easier time of it than ALMA. Also, we can simply go away and do something
> else for a while, whereas in ALMA this is harder to do. We will simply flag
> the data as invalid and go do something else; ALMA may try to struggle with
> getting the data out.
> 
> ------
> WINDING DOWN...
> 
> Ken: SUBARRAYS won't know or care which ANTENNAS are a part of them?
> Barry: There's two ways of using the word SUBARRAY (and of ANTENNA). These
> SUBARRAYS know ANTENNAs belong to them, but not which PHYSICAL_ANTENNAS
> belong to them unless they ask.
> The SUBARRAY object needs to have some mechanism of getting into the Java
> world.
to setup the correlator backend.
> 
> The primary thing that is missing is the protocol on the MIB service ports.
> When we've got any DEVICE that we know what it does, we can go ahead and
> write ANTENNAS, and then we can write the PHYSICAL_ANTENNA (which translates
> between these).
> 
> Ken: the polynomials can drive the fringe rotators?
> Barry: yep, it's all there. I worry a little bit about that one, since it
> uses CALC which is in FORTRAN, which is non-reentrant, so we may have to use
> the Synchronisable feature of Java to talk to that, which has problems of
> its own. It might not work very fast, but it will work.
> Ken: The idea is to run CALC for every antenna, for every pointing?
> Barry: yes, it wants to group its invocations by time. For one time, all
> antennas, all the pointings. This will be a bit tedious. It's main
> computational effort is figuring out where the Earth is...
> Ken: If SBs are not fixed-length, then you could run CALC every half-hour
> for the next half-hour...
> Barry: well, VLBA does something like this, but it's an organizational mess,
> hard to kept track of what's going on.
> Ken: run time for one CALC invocation?
> Barry: two-tenths of a second, but I could be way wrong either way. It could
> be as long as a second. Distressing, but possible.
> Ken: but it's unlikely to be longer in the future.
> Barry: quite. If it were not for Moore's Law, I would be considering a
> different design.
> 
> 
> Ken: when we speak of the MIB flowing data upward to the higher layer, we
> are talking of this big java observing layer? Yes. My idea in the past was
> that you needed enough info to do flagging and not much more....
> Barry: well, the critical flag is that the state of the antenna is not what
> it was commanded to be. So that's exactly this procedure. There will be a
> few interesting things, I'm sure, that there will be some things that we
> cannot deduce simply by looking at the monitor data...
> Boyd: ...we learned that on the GBT...
> Barry: ... so we'll have to pass those things along wholesale.
>