[evla-sw-discuss] [Fwd: EVLA Engineering Software Requirements - A few comments]

[Bill Sahr]

This document, from Bob Hayward, is his response to a 
request for input from the Electronics Division for
use in the Engineering Software Requirements document
being put together by Bryan Butler.  I am posting
it on evla-sw-discuss as a preview of the issues
we must address for the EVLA M&C system.  I encourage
people working on the EVLA M&C System to give read
this posting as it raises quite a number of interesting
issues, not all of which are necessarily addressed by
our current design.

Bill

-------- Original Message --------
Subject: EVLA Engineering Software Requirements - A few comments
Date: Fri, 07 Mar 2003 16:30:46 -0700
From: Robert Hayward <rhayward at aoc.nrao.edu>
Organization: NRAO
To: Bryan Butler <bbutler at zia.aoc.NRAO.EDU>,
bsahr at zia.aoc.NRAO.EDU,"Jackson, Jim(NRAO-AOC)" <jjackson at zia.aoc.NRAO.EDU>
CC: "Lilie, Paul (NRAO-AOC)" <plilie at zia.aoc.NRAO.EDU>,"Mertely, Dan
(NRAO-AOC)"<dmertely at zia.aoc.NRAO.EDU>,"Szpindor, Ed
(NRAO-AOC)"<eszpindo at zia.aoc.NRAO.EDU>

Hi All,

Here are some comments for the EVLA Engineering Software Requirements 
document. They are not in any particular order and might be considered a 
random collection of thoughts (or perhaps just a random firing of neurons):

Philosophy:
---------
We will want a paragraph or so on the philosophy of just what the guys 
in the basement need for their real-time and archived software 
requirements. If there is any particular difference between the 
philosophy the various groups adopt (FE, LO/IF, DCS, Correlator, Digital 
Samplers & Data Transmission), that should be clearly described.

One philosophical question to be debated is the issue of controlling 
modules manually or when we need to over-ride remote control. We will 
need to address best to do this. The classic example for the FE & Cryo 
Groups is being able to take a receiver off-line to purge it for half a 
day and not have the M&C system say, "Whoops, you're getting warm so 
I'll just start cooling you down again". Yet we don't want a system to 
be locked out and forgotten forever.

M&C Points:
----------
Rather than have the ESR call out every control and monitor point, it 
might be better to just reference the Hardware Definition Document (HDD) 
for each receiver/module. This will make the ESR more readable plus it 
will allow us to have just one location where the specifications reside. 
As far as I know, the HDD is suppose to list all these points along with 
their sample rates, flag conditions, control information and even 
discuss the software algorithms.

Local/Remote Control Cosiderations:
----------------------------------
We will need to interface to the module in the lab. We will have to be 
able to control the module & monitor the test points independent of the 
rest of the EVLA M&C system. Can we just hook our bench PC directly to 
the Ethernet port and open up some sort of Java browser or do we need a 
special piece of software running on the lab PC? Or should we think 
about talking to the module over its serial port?

In the "EVLA Test Rack", we will want to be able to configure the entire 
system (ie: FE, LO, IF, samplers and presumably the digital transmission 
sub-systems) just like we would on an antenna. For example, to check out 
a receiver, we need to be able to set the LO's and IF band switches. We 
will want to be able to monitor the TP detectors to measure Y-factors 
and frequency response, plus be able to exercise all the controllable 
hardware functions and check out all of the monitor points prior to 
sending a repaired or spare receiver/module out into the field.

On the antenna, we will need stand alone operation. Again, we will need 
to be able to control and monitor all of the functions and test points. 
For example, to check out a receiver we will have to set the LO's and 
band switches as well as move the sub-reflector so the receiver under 
test will see cold sky. It would be nice if there was a default 
configuration one could just call up so all of this could be more or 
less  automated. When evaluating receivers we will also need to be able 
the see the Total Power and Synchronous Detectors. If we are talking 
about total stand-alone control and monitoring at an antenna (ie: 
independent of the link to the Control Building), does that means the 
MIB would keep sending out its logged data even if the link to the CB 
was disconnected or will the antenna just shut down and everything freezes?

Do we need an Ethernet port on the surface so a tech can plug in his 
Lap/Palm-Top? This would make it much easier to do hot/cold load 
measurements (you currently need to have 2 people or a lone tech has to 
climb back and forth to the Vertex cabin to look at the A-Rack sync det 
values). It would also be nice to have the M&C test program calculate 
and display the Y-factor and/or Trx in real-time.

Is it worth implementing some sort of messaging system between a Laptop 
on the surface with a Laptop in the Vertex Room or one in the Pedestal 
Room or another antenna or the Control Building? This could make 
communication between sites easier (my kids seem to be able to spend 
hours using instant messaging between friends rather than just picking 
up a phone).

It would be nice to be able to monitor everything on another antenna. 
This will facilitate maintenance procedures by allowing a tech to 
eavesdrop on another antenna in order to decide on whether it needs to 
be visited. Whether he should be able to control items in the antenna is 
less certain because you don't want him to accidentally clobber some 
astronomical observation. However, if he has permission from the 
Telescope Operator, than he might be able to do a quick test on some 
module remotely before deciding to run out and visit the antenna. This 
could be important if the Array is in the A-configuration and he's half 
way out an arm. It would be a shame to make him travel all the way back 
to the Control Building just to remotely monitor the antenna at the far 
end of the arm he was on.

 From the Control Building, we'd obviously want to have full access to 
everything on an antenna remotely.

 From the AOC, we should be able to do anything we would be able to do 
at the Control Building (seeing how the Array may well be operated from 
the AOC sometime in the future). Again, we have to decide on some 
mechanism to obtain permission from the Telecope Operator if the 
engineering staff  needs to change some parameter which may affect 
astronomical data.

 From home, or other remote location, it would also be desirable to 
monitor M&C data from any antenna. The question of whether a member from 
the technical could be allowed to modify a control function needs to be 
discussed further. One can think of many scenarios where it would come 
in handy. A password authorization scheme might overcome the security 
issue but it doesn't solve the problem of accidentally upsetting the 
antenna cart in the middle of an observation. That could be a necessary 
option but it might make some sense to just inform the Telescope 
Operator and have him do it if he/she agrees.

Data Sampling:
-------------
All of the engineering groups will have to consider how fast they need 
to have their monitored points sampled. A default archive rate should be 
set. Unless it really needs it, it should be reasonably slow (maybe once 
per minute or even longer) but some will need to be higher (once per 
second) and some even quicker (tenth of a second). For the receivers, 
the pseudo sampling scheme allows only a few the most important receiver 
parameters to be monitored at a decent rate. The rest can only be 
sampled when the telescope is not taking data (so the F317 module needs 
to get a Data Ready bit of some sort).

What about parameters that you want to catch and flag quickly (like a LO 
out of lock), which sometimes can oscillate in and out of lock? If the 
response time between the unlock condition and the sampling, flagging 
and transmission time is too long, it will screw up the correlation, 
meaning the astronomical data will have to be edited. Is it better to 
catch it quick and gate the correlator or just edit it afterwards?

Test Software:
-------------
Who will write & maintain the software which the engineering staff will 
need for their Lap/Palm-Tops at the antenna?

Each group will required its own test routines. For example, the FE 
Group would like to have Trx calculated and displayed when performing 
Y-factor measurements on an antenna.

We may want to consider adding the capability of plugging in a GPIB 
device (like a power meter, synthesizer, etc) into the Vertex Cabin. 
This would most likely be done through a GPIB-to-Ethernet interface box 
of some sort. What kind of S/W hooks would we need to access the 
instrument or would we just have an independent PC control it remotely?

Data Archive:
------------
Those of us supporting the VLBA very much like the VLCj program that 
allows us to not only look at real-time data as well as graphically 
display historical data (alas, only up to 144 hours). Although I have no 
idea how stale the real-time data really is, but it can't be more than a 
few seconds old. We'd like a much longer archive memory - I can think of 
cases where it would be nice to look at data stretching back weeks, 
months and even years.

The FE Group would like to see things like the following from the archive:

- Look at the cryo temperatures and pressures on a receiver for a 
lengthy period (a leaky receiver usually starts to warm up slowly or its 
cold surfaces may start to oscillate). It might be nice to graph 
receivers on other antennas for a comparison. Or we might want to plot 
the temperatures on all of the receivers on a single antenna to diagnose 
a compressor problem.

- Look at Tsys over time for a receiver. It would be nice to see how a 
receiver performs over a year or more. Since the receiver is rarely used 
24/7, the plot would have to edit out those times when other receivers 
were in use.

- Look a the Tsys of a particular receiver (ie: track the serial 
number). We sometimes move a receiver around the Array a lot. It might 
be nice to see how Tsys changes from antenna to antenna.

Data Logger:
-----------
The Data logger, as envisioned by Steve Durand, would come in handy for 
  attempting to measure fluctuations in some monitored parameter which 
vary faster than the (default) sample time of the M&C system. While we 
could just inform the M&C to change the archive rate, there will likely 
be some instances where we don't really want the data to fill up the 
archive.

 From the FE point of view, we might want to use it to capture rapid 
variations in bias voltages. We would probably also want to be able to 
grab the data file and port it over to our own spreadsheet program (like 
Excel) to be fiddled with further. At some point, if this application 
becomes more of a standard test procedure, we might want it to become a 
ESR test utility.

Calibration:
-----------
The FE Group is assuming that we will provide Tcal data to Operations as 
we've done in the past for the VLA, but in a table form (say every 100 
MHz across a receiver band) rather than the single Tcal value that we 
give them now. What kind of format will be required? Will we be able to 
read them back on-line to check them independently?

Since we have measured Trx also, is it worth having Ops include this 
data also and allow it to be accessed. Nowadays we have to keep track of 
all the SOIDA plots ourselves. It might make some sense to have it 
available on-line so that astronomers could also refer to it, say, to 
correlate a poor Tsys with a nasty bump in the Trx plot.

That's enough for a Friday afternoon,
Bob