[evla-sw-discuss] NRAO Common Software

[Bill Sahr]

The following is a cut & paste of an 8 month old
email exchange between myself & Richard Prestage 
of Green Bank concerning the use/resuse of software 
for the monitor and control system.  The issues 
raised in this exchange are now immediately relevant.

I am not familiar with the current organization
& staffing at Green Bank.  I would appreciate it
if Green Bank recipients of this posting would 
further distribute it to any concerned or affected
parties.

Thank you.

Bill


-------- Original Message --------
Subject: NRAO common software?
Date: Tue, 16 Oct 2001 22:02:04 -0400 (EDT)
From: Richard Prestage <rprestag at gb.nrao.edu>
To: <pnapier at sadira.gb.nrao.edu>, Bill Sahr <bsahr at sadira.gb.nrao.edu>,Tim
Cornwell <tcornwel at sadira.gb.nrao.edu>,Gareth Hunt
<ghunt at sadira.gb.nrao.edu>
CC: Phil Jewell <pjewell at sadira.gb.nrao.edu>,
<pvandenb at sadira.gb.nrao.edu>,Joe Brandt <jbrandt at sadira.gb.nrao.edu>,"Mark
H. Clark" <mclark at sadira.gb.nrao.edu>,John Ford <jford at sadira.gb.nrao.edu>


Peter, Bill et al -

At the scientific staff meeting yesterday, Peter made the remark that
the EVLA project is having a hard time hiring real-time programmers.
At the same time, Gareth Hunt and the GB monitor and control group
spent some time a couple of weeks ago discussing why NRAO software
groups appear to start over, almost from scratch, for each new
telescope project.  We agreed that, while there might have been good
technical reasons for this in the past, there are no good technical
reasons now. I didn't follow up at the time after our discussions with
Gareth, but Peter's statements have motivated me to do so now.

Let me be explicit: I can think of no compelling technical reason why
the NRAO does not develop "common" telescope control software which is
applicable to all of its telescopes. (There may be non-technical
reasons. Some of them are pretty valid, other less so.) The logical
extension of my previous statement is that the GBT telescope control
software should be adopted as the basis for development for the EVLA
project (I'll discuss some questions of "software re-use" below).

After the scientific staff meeting, I took the time last night to go
to the EVLA web site, and read the "EVLA Architecture and Design"
document (six months after it was written!). The implication is that,
although the concepts may be re-used, none of the existing GBT M&C
software will be. If that decision has been made, is it too late to
influence it? If not, should we not commit ourselves to develop a plan
whereby the GBT telescope control software forms the basis of the EVLA
control system, to the mutual benefit of both projects?

This would be a really lousy time to start such an initiative from the
GBT point of view. I've wrestled with whether to send this email out
or not. After careful consideration, we might decide this is not a
good idea. However, I believe the long-term advantages to the GBT (and
the EVLA, and NRAO) would outweigh the short-term costs (and I can
suggest ways to mitigate those). There will never be a good time to
start this. If we wait for the perfect time, it will never happen.

Comment welcome.

Regards,
			Richard


A note on the issue of software re-use
--------------------------------------

Many scientific organisations have made the conscious decision to develop
"common software" to use on successive projects. The ESO common software
is an example well known to NRAO. A more relevant example to the GBT/EVLA
case might be the UK's use of the Anglo-Australian Telescope's "DRAMA"
software (see http://www.aao.gov.au/drama/html/dramaintro.html for
details).

DRAMA is a tool-kit which provides a tasking model, inter-task
communication software, user interfaces, an error reporting library,
and so on. DRAMA was developed in the early nineties, and is an
evolutionary upgrade to the UK "ADAM" environment developed in the
eighties, It is used on the AAT (a 3.9 meter vintage 1970s equatorial
mount optical telescope), the WHT (a 4.2m alt-az telescope
commissioned in the late 1980s), and the JCMT (a 15m alt-az submm
telescope which adopted DRAMA in the mid-1990s).

There are three reasons why I put DRAMA forward as a good example.
Firstly, it is without question a demonstration that common software can
be used very successfully across telescopes of different types and
vintages (and with software groups and budgets *far* smaller than that
of ESO).  Secondly, the use of DRAMA at the different sites is selective;
some have adopted the complete package, others only the bits they like
best (needed the most). But, where adopted, this is explicitly use of the
same physical code (not just adopting similar philosophies). Finally, for
all of the telescopes that use it, DRAMA has been used to add to
(replace) existing telescope control software in an evolutionary fashion.

In my opinion, there is no question that (appropriate) "common
software" is a good thing. The "EVLA Architecture and Design" document
suggests (page 4) "Literal code re-use is not only impossible, but to
be avoided."  I disagree rather strongly (although it may be a problem
of semantics, and I have singled out a single sentence). For example,
all of the AAT, WHT, UKIRT and JCMT use the identical inter-task
communication software, and this is extremely successful. The correct
statement, in my opinion, is that "common software implementations
must evolve to provide new functionality and correct deficiences;
interfaces should be well-defined and backwards compatability
maintained".

To use a hypothetical example, GB could develop the first version of
NRAO's common communications library (call it RPC++ 1.0). The EVLA
could, and should adopt this. EVLA developers would undoubtably
discover some bugs, which could be fixed to the benefit of both
groups. Eventually, EVLA might say "RPC++ is good, but we need some
new functionality. We will develop RPC++ 2.0." Older applications
could link against RPC++ 2.0 without modification, but wouldn't be
able to take advantage of the new features. (In the UK we went
further; new releases were installed by the use of new dynamically
linked libraries; the need for a relink was frowned upon.) GB could
use RPC++ 2.0 for new tasks; these could communicate to older, version
1.0 tasks without any problems.

Eventually, we might decide that the RPC mechanisms were too long in
the tooth and we needed to adopt CORBA. In this case, an RPC++ 2.0
interface wrapper should be provided as well as the new native CORBA
interface. Old tasks might now need to be relinked to make use of the
new implementation, but the applications themselves would not need to
be re-written. Alternatively, an RPC++ 2.0 to CORBA translator task
could be provided.

What would EVLA get if NRAO adopted the GBT software as the basis for
its common software? A modern, object-oriented, modular system
developed in C++ and portable across Solaris, VxWorks and (more or
less) Linux. The benefit of tens of staff years of development
effort. A tasking model, an inter-task communication library, a
parameter system, and a powerful message handling system. The ability
to develop GUIs in Glish/Tk or Tcl/Tk, including in the Glish case a
well developed "table parser" allowing processing of "observe files".

What would the GBT get if EVLA were to use the GBT software? The
benefit of additional developers working on the same code base, as
opposed to developing their own software to tackle the same
problems. Perhaps assistance in completing the port to RT-Linux. A
next-generation MCB replacement.  The possibility that EVLA might
develop a Java user interface, for example, which could then
immediately be adopted for use on GBT. And so on...


-------- Original Message --------
Subject: Re: NRAO common software?
Date: Fri, 19 Oct 2001 19:05:55 -0600
From: Bill Sahr <bsahr at cv3.cv.nrao.edu>
Organization: National Radio Astronomy Observatory
To: Richard Prestage <rprestag at sadira.gb.nrao.edu>
CC: pnapier at sadira.gb.nrao.edu, Bill Sahr <bsahr at sadira.gb.nrao.edu>,Tim
Cornwell <tcornwel at sadira.gb.nrao.edu>,Gareth Hunt
<ghunt at sadira.gb.nrao.edu>,Joe Brandt <jbrandt at sadira.gb.nrao.edu>,"Mark H.
Clark" <mclark at sadira.gb.nrao.edu>,John Ford
<jford at sadira.gb.nrao.edu>,Gustaaf Van Moorsel
<gvanmoor at cv3.cv.nrao.edu>,Boyd Waters <bwaters at cv3.cv.nrao.edu>
References: <Pine.LNX.4.33.0110162159050.5550-100000 at etamin2.gb.nrao.edu>

Richard,

Your note contains many good points.  I will attempt to respond to some
of them by injecting my comments within the included text of your
message.

Bill

Richard Prestage wrote:
> 
[snip]
>
At this point I am going to take the liberty of doing a bit of
"cut & paste" in order to group various points to which I wish
to respond.

Richard Prestage continues:
> At the same time, Gareth Hunt and the GB monitor and control group
> spent some time a couple of weeks ago discussing why NRAO software
> groups appear to start over, almost from scratch, for each new
> telescope project.  We agreed that, while there might have been good
> technical reasons for this in the past, there are no good technical
> reasons now...
> 
> Let me be explicit: I can think of no compelling technical reason why
> the NRAO does not develop "common" telescope control software which is
> applicable to all of its telescopes...
> 
> After the scientific staff meeting, I took the time last night to go
> to the EVLA web site, and read the "EVLA Architecture and Design"
> document (six months after it was written!). The implication is that,
> although the concepts may be re-used, none of the existing GBT M&C
> software will be. 

& from a later section

> In my opinion, there is no question that (appropriate) "common
> software" is a good thing. The "EVLA Architecture and Design" document
> suggests (page 4) "Literal code re-use is not only impossible, but to
> be avoided."  I disagree rather strongly (although it may be a problem
> of semantics, and I have singled out a single sentence).

Yes, your interpretation of that sentence is a problem of semantics.
My position on code reuse is that if possible and if done appropriately,
it is a "good thing".  It is a very "bad thing" to commit to code reuse
before a determination has been made that the code is appropriate to
the requirements of the design.  The EVLA is in the requirements
gathering stage.  As the process of requirements gathering proceeds, 
high level design will begin & progress.  Once a partial, but fairly 
complete, high level overview of what is needed has emerged, then 
we will examine pre-existing code bases with an eye toward code reuse.  
I can & will attach a timeline to this scenario.  Decisions concerning 
reuse of code should be made within 1 to 3 months after the EVLA M&C 
PDR that is currently scheduled for Feb 2002.  In large part, this 
particular sequencing of steps is an issue of pragmatics.  Until we 
have done the work needed for the PDR, any detailed, and, therefore, 
time consuming examination of other bodies of code would drain our 
currently undermanned effort of the resources needed to accomplish 
the tasks that are properly our 1st priorities at this time - 
requirements, initial design, and recruitment.  _If_ we had the 
staffing, I would make the examination of the GBT code, DRAMA, the
ALMA TI, and some other initiatives (such as a code generation 
package for M&C systems that has been developed for the LMT) a 
parallel effort.  However, the staffing is not available at this 
time, so the process must be serialized in order to meet the EVLA
project deadlines.

& , a bit more cut and paste of Richard's remarks:

> ... If that decision has been made, is it too late to
> influence it? If not, should we not commit ourselves to develop a plan
> whereby the GBT telescope control software forms the basis of the EVLA
> control system, to the mutual benefit of both projects?

> 
> To use a hypothetical example, GB could develop the first version of
> NRAO's common communications library (call it RPC++ 1.0). The EVLA
> could, and should adopt this. EVLA developers would undoubtably
> discover some bugs, which could be fixed to the benefit of both
> groups. Eventually, EVLA might say "RPC++ is good, but we need some
> new functionality. We will develop RPC++ 2.0." Older applications
> could link against RPC++ 2.0 without modification, but wouldn't be
> able to take advantage of the new features. (In the UK we went
> further; new releases were installed by the use of new dynamically
> linked libraries; the need for a relink was frowned upon.) GB could
> use RPC++ 2.0 for new tasks; these could communicate to older, version
> 1.0 tasks without any problems.
> 
> Eventually, we might decide that the RPC mechanisms were too long in
> the tooth and we needed to adopt CORBA. In this case, an RPC++ 2.0
> interface wrapper should be provided as well as the new native CORBA
> interface. Old tasks might now need to be relinked to make use of the
> new implementation, but the applications themselves would not need to
> be re-written. Alternatively, an RPC++ 2.0 to CORBA translator task
> could be provided.
> 
> What would EVLA get if NRAO adopted the GBT software as the basis for
> its common software? A modern, object-oriented, modular system
> developed in C++ and portable across Solaris, VxWorks and (more or
> less) Linux. The benefit of tens of staff years of development
> effort. A tasking model, an inter-task communication library, a
> parameter system, and a powerful message handling system. The ability
> to develop GUIs in Glish/Tk or Tcl/Tk, including in the Glish case a
> well developed "table parser" allowing processing of "observe files".
> 
> What would the GBT get if EVLA were to use the GBT software? The
> benefit of additional developers working on the same code base, as
> opposed to developing their own software to tackle the same
> problems. Perhaps assistance in completing the port to RT-Linux. A
> next-generation MCB replacement.  The possibility that EVLA might
> develop a Java user interface, for example, which could then
> immediately be adopted for use on GBT. And so on...

Ever since Boyd Waters, myself, and Bruce Rowen visited Green Bank,
we have hoped that it would be possible to develop exactly the sort
of dynamic you describe.  For over a year now, we have discussed this
sort of possibility among ourselves and very much hope that it will
come to pass.  I will encourage it.  However, when I read statements 
about specific pieces of code that the EVLA "could and should adopt" 
now, prior to a determination of the needs of the EVLA, I begin to dig 
in my feet like a mule that is being pulled too hard.  I will not
pre-constrain the scientific, engineering, and technical aspects of
the design before investigating the needs of the project.

I am not against the idea of code reuse.  I do not suffer from the 
"not invented here" syndrome.  I simply want to proceed in a 
reasonable manner that takes first things first, within the 
current constraints of our schedule and staffing levels.  If those 
constraints change, I will parallelize more of our efforts.

Bill


-------- Original Message --------
Subject: Re: NRAO common software?
Date: Mon, 22 Oct 2001 08:07:17 -0400 (EDT)
From: Richard Prestage <rprestag at gb.nrao.edu>
To: Bill Sahr <bsahr at cv3.cv.nrao.edu>
CC: Richard Prestage
<rprestag at sadira.gb.nrao.edu>,<pnapier at sadira.gb.nrao.edu>,Tim Cornwell
<tcornwel at sadira.gb.nrao.edu>,Gareth Hunt <ghunt at sadira.gb.nrao.edu>,Joe
Brandt <jbrandt at sadira.gb.nrao.edu>,"Mark H. Clark"
<mclark at sadira.gb.nrao.edu>,John Ford <jford at sadira.gb.nrao.edu>,Gustaaf
Van Moorsel <gvanmoor at cv3.cv.nrao.edu>,Boyd Waters
<bwaters at cv3.cv.nrao.edu>


Hi Bill -

your response was great - all that I could have hoped for. I must admit
that as a result of our discussions with Gareth, we had a more pessimistic
picture of where the EVLA was headed (hence in part my note). I'm glad
that this is not the case. I'm glad also to hear that your recruiting
situation sounds more positive. (A minor point; we found trawling
unsolicited Monster resumes pretty much a waste of time. The response to a
specific ad of our own on Monster was much more profitable - although in
the end we still haven't filled a position from that route).

We are suffering from a similar effort crunch to that you describe (we
currently have a total of four unfilled computing positions). I was very
hesitant about committing the GB group to even more work. However, if
we can build a fruitful collaboration, it will be more than worthwhile.
The timescale for when detailed discussions might begin (Spring next year)
also looks much more doable from our point of view.

I would encourage you if possible to start the dialog in the early design
stage (perhaps even before the PDR). While I completely understand that
you do not wish to constrain the design in advance, it may be that you get
to a choice of two (or more) possible approaches, one of which would lend
itself to collaboration, while the other would not. While it certainly
shouldn't drive the design, I think knowledge of the fact that a
(modifiable) NRAO implementation already exists should certainly influence
it.

I look forward to further interchange over the coming months.

Regards,
				Richard