[evla-sw-discuss] RTOSes ?

[John Ford]

(I've read all the replies so far, and address several of them in
here...)

Bill Sahr writes:
 > 
 > I noticed in the notes from the EVLA Coordination Meeting
 > of 1/28/2002, that Gareth is on record as finding the notion
 > of using more than one rtos "unacceptable".
 > 
 > I am wondering what the opinion of other members of this
 > discussion group might be on this matter.
 > 
 > I see three contexts in which an rtos might be used - 
 > 1) the MIBs for the antenna hardware and ancillary
 > devices (such as the API), 2) the MIBs on the correlator 
 > boards, and 3) the real-time crates that will run the 
 > applications which perform tasks such as geometry 
 > calculations, checker functions, data-flagging, higher 
 > level, overall correlator monitor and control and 
 > communication with the correlator backed, etc, etc.

This is a good analysis of the situation.  It is my opinion (bolstered
by a bit of experience) that there is a very large gulf between the
requirements for the MIB-like device and the other computer systems
(see below). 

I also think that the MIB should be more like the VLBA SIB and less
like a full-blown station computer.  There are many reasons, but chief
among them is that hardware developers need a simple reliable
interface to attach to their designs *early* in the project, so that
they can do useful debugging and testing of their designs without a
lot of software support.  I would err on the side of making the MIB
dumber, rather than smarter.

It is much easier to upgrade the software in a crate or rack-mount
computer than to upgrade 27 flash ROMS of hardware when a bug is
found in code embedded in some MIB in the system.

 > 
 > Frankly, my own position on the matter is that we might
 > end up with more than one rtos, and that we it would not
 > be a disaster if such were the case.  Would I prefer
 > one rtos ?  Yes.  However, we have a great deal of
 > real-time experience within the group.  Nearly all of
 > us have worked with several different rtoses over the
 > course of our careers.  All of them must do pretty much
 > the same sort of thing, and most of them can be learned 
 > rather quickly.  In my experience, when working with 
 > a small, modular, lean rtos one can begin to do useful 
 > work in a little as 4 weeks.

I agree with this sentiment, to a point.  However, it takes a long
time sometimes to find out the best way to approach a problem with
OS's of differing feature sets, and all RTOS's have undocumented
features to be feared.

 > 
 > The same rtos may simply not be the right tool for all
 > three contexts.  For the antenna MIB we want an rtos
 > that is as compact as possible, offers a minimum of
 > real-time functionality and incorporates a TCP/IP
 > protocol stack.  The correlator MIB presents a situation
 > that is considerably less limited with respect to resources,
 > and may benefit from a more varied and extensive set
 > of capabiliites.  The real-time crates are yet another
 > class of applications.

I agree entirely here.  Bad publicity for many systems is due to
misapplication.  I would think that you would need to have an RTOS
with one set of features (small, low context switch times, low
interrupt latency, ???) for the MIB, and another set of features (high
availability, driver availability, virtual memory, CORBA, etc.)  I
suppose it *might* make sense to use the same thing, if it was
configurable to that extent.

 > 
 > My deepest reservation concerns the use of real-time linuxes 
 > of one sort or another.  Not only is the programming style 
 > quite different from the approach taken for non-Linux work, 
 > but also, I feel that Linux has technical characteristics 
 > which make it fundamentally inappropriate for real-time work.  
 > The  patches and modifications made to Linux to adapt it to
 > real-time strike me a kludges which tend to mire one
 > in systems level rather than applications level 
 > considerations.  (If real-time Linux were appropriate
 > to any component of the system, I would use it for
 > item 3 as listed above - the real-time crates and not
 > in an embedded context.

We have written a TaskLib that is API compatible with VxWorks that
allows us to compile and run much of our system on Solaris, Linux, or
VxWorks.  The programming approach that we take is the same in any
case, that is, separate out the RT from non RT parts of the system,
use standard Ygor libraries, use POSIX functions where possible, etc.

I would not be frightened of Linux.  We are moving away from VxWorks
to Real-Time Linux (the RTAI flavor) for the GBT systems.  We already
run Linux on many critical systems, and it seems to work just fine.
Joe Brandt is porting our tracking LO system to RTAI right now. The
kernel patch is not really that big a hack.  The technique has been
used for years (in spite of Yodiaken's patent.)

We have, at this moment, the GBT antenna controller running under
Solaris on an Ultra-1.  Runs rings around the VxWorks version.  Less
jitter, better development environment, cheaper hardware, more
reliable, etc.

The good thing about using Linux, Solaris, or another "real" OS is
that you can separate out the real-time stuff (which is typically a
small percentage of the tasks to be done) and run all the non-realtime
stuff in user space, which means that all the standard development
tools are available, experienced developers are available (Rats.  I
sound like a Microsoft drone!), and the real-time stuff can run by
itself on the (nearly) bare iron.

It seems to me that the Correlator MIB is a different beast from the
MIB that you'd embed in, say a receiver front-end.  I would design
something with a standard communications interface and little
intelligence to talk to the hardware, and put the smarts in a higher
level controller.  I think having the ability to put some intelligence
on the card is a good idea, but it seems it should be the exception
rather than the rule to use it.

Barry's summary of the whole situation seems correct.

One other requirement that I would levy on *any* software that you
use, but especially the RTOS, is that you get source code.  This will
save your bacon someday.  The speed of obsolescence nowadays is
incredible.

If I've run off the tracks regarding where you are in this process and
your philosophy about what the MIB is, I apologize.  I've read much of
the archive, but not all of it.

FWIW.

John