[evla-sw-discuss] Antenna MIB software & toolset
Bill Sahr
bsahr at cv3.cv.nrao.edu
Thu May 2 00:20:02 EDT 2002
Included below is a report on the MIB software,
including my recommendation for purchase and
a justification of that recommendation.
Bill
-------------- next part --------------
Date: 05/01/02
Subject: MIB Hardware & Software, Risks & Choices
Author: Bill Sahr
Soon, a decision will be made on the rtos, networking software, and
tool set for the antenna MIB. The purpose of this report is to
summarize the two main choices that have emerged, and to make explicit
some of the risks involved.
RISKS
The basic problem can be summed up in one sentence. We must choose
some $80K to $100K worth of software before we know that the processor
chosen is actually suitable. Suitable, in this case, is defined as
RFI quiet. The TC11IB chip has three characteristics which make it
desirable - 1) sufficient on-chip RAM (1.5 Mbytes) to avoid execution
of code and data fetches from off-chip RAM, 2) an input clock of 12
MHZ which may be a sinusoid rather than a square wave, and 3) on-chip
peripherals & peripheral interfaces. The combination of these
characteristics should make for an RFI quiet processor. The risk is
in the word "should". We will not actually know the RFI
characteristics of the MIB until the actual MIB board has been
designed, and a prototype has been tested in the RFI chamber. The
evaluation board, due the 1st week of May 2002, will not provide a
true test case. It is the actual MIB board which must be tested, and
a MIB board prototype is not scheduled to be ready until 7/15/02. The
schedule for the test antenna does not permit that we delay the
decision and purchase of the software until tests of the MIB prototype
have been completed.
SCHEDULE
Installation of the prototype EVLA M&C hardware on the test antenna is
currently scheduled to begin in Q2 2003. Take that date as April
2003. I do not currently have a date for the start of bench testing,
but I will guesstimate it to start 6 months before installation of the
hardware on the test antenna - October 2002. Barry & I both estimate
that it will take 5 to 6 months to write the software to support bench
testing. If we accept the figure of 6 months, then we should have
begun software development in April 2002. We are already late.
Clearly, delaying our ability to even begin software development until
testing of the MIB prototype is completed would absolutely guarantee
a delay of several months for the test antenna.
MORE ON SCHEDULES
Once we decide on the MIB software, we cannot simply order it and have
it sent FedEx. We must 1) have multiple copies of the evaluation
board for the TC11IB in-house, 2) if the ethernet port has not been
implemented on the evaluation board, then we must add the needed
hardware to the evaluation board, 3) send an evaluation board to the
rtos vendor for the port, and 4) the rtos vendor, after completion of
the port, must forward the board along with the ported rtos to the
group or company that will write the ethernet driver. These steps
assume that we have already requested and received formal quotes for
the work, and have made a decision on the vendor(s) who will receive
the contract for the work. Timelines for the port and the ethernet
driver will be given in the description of the two alternatives for
the software.
THE CHOICES
I. Accelerated Technology Inc (ATI), Altium: Required components
Nucleus PLUS rtos ~ 45KB $12.5K, ? # of seats
Nucleus Net ~ 87KB, code + $15.0K, ? # of seats
add'l RAM
Ethernet driver ~ 5.5KB code+RAM included in port costs
Nucleus WebServ ~ 35KB, code+RAM $10.0K, ? # of seats
(HTTP server) not incl content
see comment 1
Comments:
1. Basic capabilities of Nucleus WebServ include serving static files,
building dynamic pages, DES encryption, and the ability to respond to
function execution requests from a browser. A file system is optional.
The utility that allows one to forgo use of a file system is included in
the purchase price. WebServ does NOT include a server side push
capability that would allow continuous updating of browser screens. They
do have a workaround to make an HTML page refresh automatically which can
be used if we have control over (can modify) the HTML content that is being
browsed. The workaround takes the form of placing the statement <meta
http-equiv="Refresh" content="20;URL=index.htm"> before the body of the
HTML page. Content is the amount of time, in seconds, before the page
refreshes. URL is the page to refresh. The claim is that this tag, used
in combination with the CGI interface capabilities, will provide continuous
update of a browser page with the desired values.
Porting costs: ATI is offering a package deal which includes the port of
the kernel, development of the ethernet driver, integration of all products
purchased (rtos, network stack, the http server, any/all elements of the
extended protocol package, the shell, etc, plus updated documentation that
has been passed through their verification process, and integration with
the tool set (at a minimum, ATI will need to develop the module that
provides kernel aware debugging for the TASKING debugger) for a total of
$30K to $50K. They are conducting negotiations with Infineon for partial
financial support of the porting effort. If these negotiations are
successful, the cost may be lower. If cost becomes the determining factor,
they are willing to be flexible on the pricing.
Time to port: As of 4/22/2002, if they had received the P.O. that
day, the work would have begun on 5/20 and concluded on 7/5/2002.
Scheduling of work is on a 1st come, 1st serve basis, with 6 to 7
weeks needed for this job + several (? 4) weeks lead time before the work
can begin. Call it a total of 10 to 12 weeks.
Tool Set
I have allowed the rtos vendors to specify their preferred tool set. ATI
has a close integration with the TASKING tool set by Altium, and prefers
to work with the TASKING tool set. The debugger included with the TASKING
tool set will be kernel aware w.r.t. the Nucleus PLUS rtos. Altium seems
to have a close relationship with Infineon. The current release of the
TASKING tool set already includes support for the TC11IB, and Infineon used
the TASKING tool set to develop a set of publicly released benchmarks for
the TC11IB. (See www.eembc.org/benchmark/benchmain.asp)
The floating point libraries provided by TASKING for the TC11IB are
SINGLE PRECISION. Will single precision floating point be adequate for our
needs ? The runtime libraries are all thread-safe. Source code for the
runtime libraries is not available.
The tool set includes capabilities for source code management and version
control. There are a few version managers to select, such as PVCS, but CVS
is not one of the version managers available. The source/version
management scheme used by TASKING will not integrate with CVS.
TASKING by Altium $5.4K node-locked
$7.1K float
Not including maintenance
ATI required components - 172.5 KB RAM
ATI required components - $37.5K
ATI port costs - $30.0K to $50K
Tool set, 3 floats - $21.3K
--------
$88.8K to $108.8K
II. Express Logic, Interniche, Green Hills: Required components
ThreadX rtos ~ 32KB $12.5K, 3 seats
NicheStack ~ 45.5KB $12.0K, ? # of seats
Ethernet driver ~ 5.5KB code+RAM see port costs
(Interniche)
WebPort ~ 23.5KB code+RAM $12.0K, ? # of seats
(Interniche) not incl content
see comment 1
HTML compiler N.A. Normally $3K
see comment 2 see comment 2
Comments:
1. Basic capabilities appear to be the same as Nucleus WebServ with the
addition of a server side push capability that allows for the continuous
update of a browser screen. This capability could be useful to us. We
might be able port WebPort to Nucleus PLUS, but I don't know that it would
be worth the effort. The literature claims that all that is required is a
TCP/IP stack with a Sockets API and a single timer call (1-1000 times per
second).
2. As with Nucleus WebServ a file system is optional, but the utility which
allows one to omit a file system is an extra cost item. The cost is
normally $3K. Interniche is willing to include the HTML Compiler in a
software bundle at no cost if we buy directly from Interniche. However,
Green Hills also resells the InterNiche software. If we buy the InterNiche
software through Green Hills they will supply the Kernel Aware Package
(KAP) for ThreadX at no charge. If we buy the InterNiche software directly
from InterNiche Green Hills states that there will be a $10K to $25K charge
for the KAP. Dueling vendors and arm twisting. Exactly the sort of
situation we wish to avoid.
Porting Costs: $10K to port the rtos. Express Logic does not have the
expertise to write the ethernet driver, but Interniche does. Interniche
will write the Ethernet driver at a cost of $2K. This arrangement is not
as fully integrated as the ATI deal. Interniche will integrate the driver
with NicheStack. We will be required to integrate the other components -
rtos with stack with http server, etc. My best estimate, based on a
reading of the porting instructions and a conversation with an individual
who has done this sort of work is integration will consume several weeks to
a month or more.
Time to port: Express Logic requires 10 - 14 weeks after receipt of the
purchase order and Interniche needs 7 working days. Call it 11 to 15 weeks
total.
Tool Set
Green Hills is the preferred tool set for Express Logic. Most of us know
Green Hills from the days when they were a compiler company. Their
compilers had and still have a good reputation, and I would be surprised if
their tool set was anything less than a quality product. I have been
informed by their sales rep that the Green Hills tool set is used by an
Infineon group working in Colorado Springs. However, Infineon makes many
chips, and I do not know if that particular Infineon group works with
either the TC11IB chip or the TriCore architecture. Green Hills supports
the TriCore V1.3 as a generic core. It does not include support
specifically for the TC11IB. The Green Hills debugger is kernel aware
w.r.t. ThreadX.
The Green Hills tool set provides _both_ single and double precision
floating point libraries. The runtime libraries are NOT thread safe, but
can be made to be so by the user. Source code is "available" for the
runtime libraries, but I do not have a price.
Green Hills uses a proprietary system for source code/version management,
but they have stated that "CVS integration is fairly automatic in newer
versions" of the tool set.
There is one annoying gotcha with the Green Hills tool set. For floating
licenses they use a license manager (ELAN) that does not work correctly
under Windows 2000. There is no anticipated fix. The company which
produces the license manager was acquired by a competitor who has no
intention of fixing the bugs. Green Hills is evaluating other license
managers, but no decision is expected soon. Since our defacto Windows OS
is now Windows 2000, we would have to run the license manager on a
non-Windows 2000 machine. Marie Glendenning and Allan Poindexter tell me
that running the license manager on Solaris is no problem as long as a unix
daemon for the license manager is supplied. Green Hills says ELAN does run
under Unix so I assume the needed daemon is included.
Green Hills Multi IDE $5.9K node-locked
$7.9K float
Not incl maintenance
Express Logic + Interniche, required components - 106.5 KB
Express Logic + Interniche,
required components - $36.5K
HTML Compiler - $ 3.0K to $ 0K
port costs - $12.0K
Green Hills KAP - $0 to $10K - $25K
tool set, 3 float - $23.7K
--------
$75.2K if Interniche software purchased
from Green Hills
$82.2K to 97.2K if Interniche software
purchased from Interniche
Note we pay either the $3K for the HTML Compiler _or_ a $10K to $25K fee
for the KAP, but not both.
OPTIONAL COMPONENTS, Accelerated Technology Inc.
TFTP Server 7.0 - 9.0KB code + RAM
FTP Server 16.5 - 21.5KB code + RAM
FTP Client 15.0 - 19.0KB, code + RAM
Telnet Server 28.0 - 38.0KB, code + RAM
note: TFTP Client Support is provided as a part of Nucleus NET.
Comments:
1. The FTP Client & Server require a file system. The FTP Client can be
configured to use the Nucleus File Abstraction Layer (FAL) and a block of
memory as the file medium, or it can use Nucleus FILE. The Nucleus FAL is
provided as a part of Nucleus NET. The FTP Server requires a true physical
file medium and a true file system. The FTP Server is compatible with
Nucleus FILE, which includes a RAM disk driver.
2. Nucleus Telnet integrates with Nucleus SHELL (discussed later) to allow
a remote user to perform runtime kernel aware debugging via a Telnet
session.
ATI bundles the TFTP Server, the FTP Server, the FTP Client, and the Telnet
Server into one package, the Nucleus Extended Protocol Package. The cost
for the package is $7495.00.
Nucleus FILE
Requires 30 - 45KB of RAM for the code, plus ~ 7KB for data. In a recent
document I used 47KB as my figure for the RAM footprint. Nucleus FILE does
come with a RAM disk driver, and can use those flash devices which emulate
an ATA/IDE disk. To use raw flash a media manager layer is required. ATI
does not offer a media manager, but can refer us to several third party
sources.
Nucleus FILE Cost - $5995.00
MS-DOS 4.0 and higher compatible (ugh!). "Mult-tasking aware", i.e., user
tasks must register with the file system before it can be used, and multiple
tasks can request services from the file system at the same time.
Nucleus SHELL
Intended to complement use of a source level debugger. Installs as a
Nucleus PLUS task. The shell can be invoked either by entering a status
command or by a user entering the C interface calls and responding to the
prompts for the parameters. Its features and capabilities include display
of information on all kernel objects, display and modification of memory,
and remote invocation of Nucleus PLUS services.
RAM footprint - ~ 35KB, code + data
cost - $1,295 per seat
Nucleus MNT & the MNT binary bundle
Nucleus MNT is a port of Nucleus PLUS to the Windows environment. The port
includes setup of interrupt vectors for the timer and terminal interface,
scheduling via the Windows threads model for task switching, and a timer
tick for Nucleus PLUS task sleep, time slicing, time-out and timer thread
capabilities. Using Nucleus MNT it is possible to develop Nucleus
PLUS-based applications that execute as native Windows processes under
Windows 95/98, NT, and 2000. Nucleus MNT requires Microsoft Visual C++ as
the tool set. The complete MS Visual C++ IDE is available including the
editor, project/make capabilities, compiler, librarian, assembler, linker,
and debugger. A MS Visual Studio plugin which provides kernel awareness
for Nucleus MNT is also included. A prebuilt demo application that
exercises most of the Nucleus PLUS capabilities is provided.
The MNT binary bundle is a port of all additional Nucleus products to the
Windows environment. It includes binaries of Nucleus NET, Nucleus WebServ,
Nucleus GRAFIX, Nucleus FILE, Nucleus C++ (a C++ wrapper of Nucleus PLUS,
NET, and FILE), the servers and client in the Extended Protocol Package,
the SHELL, Nucleus email, and SNMP that have been ported to Nucleus MNT.
The Nucleus binary bundle will run under Windows NT or Windows 2000.
The MNT version of Nucleus NET allows Nucleus MNT processes to communicate
with one another via a shared memory area. Multiple versions of Nucleus
MNT can be executed on a Windows NT/2000 machine, each with its own IP
address, providing the ability to simulate a network within a single
NT/2000 machine. This particular capability may not be useful to us. In
the context of our application it would model MIB to MIB communications,
which is not currently planned. Of greater interest is the fact that
winsock applications can communicate with the MNT environment, giving us
the ability to create data sources and sinks for MNT applications. Another
useful feature is that Windows applications such as browsers can
communicate with the MNT environment via the Windows NDIS driver, and MNT
applications can communicate with nodes on a physical network using NT/2000
routing facilities.
ATI claims that they use MNT in-house for initial software development.
Ram footprint - N.A.
Cost - Nucleus MNT, $995.00 per seat
Nucleus binary bundle, $995.00 per seat
Nucleus GRAFIX
GRAFIX is a GUI package for embedded systems. Since we do not plan to
embed a GUI I have not investigated this product.
CEE-J
Pronounced "siege". CEE-J is Java. To be more specific, CEE-J is a suite
of Java virtual machines. Four Java configurations are supported - CLDC
(Connected Limited Device Configuration), MIDP (Mobile Information Device
Profile), EmbeddedJava, and PersonalJava. CLDC is intended for devices that
have minimal display capabilities, and includes selected classes from
java.lang, java.io, and java.util. MIDP is a platform that contains a
specialized library tailored to small displays. CEE-J is marketed by ATI,
but actually manufactured by a company named Skelmir. Kevin has been in
contact with Skelmir. The information Kevin has supplied to me indicates
that
- the VM for the CLDC version is ~ 100KB in size
- network services can be added at the cost of ~ another 100KB
- Skelmir is interested in porting to the TriCore architecture
- the estimated port time is 10 weeks
- the port would be done after the Nucleus port is completed
- initial pricing must be obtained through ATI.
The CLDC platform with network services is worth considering. We must
investigate the capabilities of the CLDC platform more thoroughly, and we
must obtain a price.
OPTIONAL COMPONENTS, Express Logic & InterNiche
Express Logic
FileX. FileX is Express Logic's embedded file system. A file system is
required if we wish to include an FTP server in our system. FileX would be
one possibility. It is the usual DOS-compatible file system, supporting
12, 16, and 32 bit FAT, long filenames, etc.
RAM footprint - ~ 26KB RAM + code for one open media
Price - $5250
Express Logic does NOT offer a shell task.
Interniche
FTP server 32KB code + RAM + 3.5KB per session $3.0K
Telnet server 4KB code + ~ 3.2KB per connection $6.0K
PURCHASE RECOMMENDATION
My personal recommendation is that we go with Nucleus PLUS by ATI for the
software and the TASKING tool set by Altium. I do have some questions
about exactly which optional components we may desire. I suggest a
possible configuration below, with some questions I would like help in
answering.
Nucleus PLUS rtos ~ 45.0KB $12.5K
Nucleus NET ~ 87.0KB $15.0K
Ethernet Driver ~ 5.5KB part of port costs
Nucleus WebServ ~ 35.0KB $10.0K
port costs $30.0K to $50.0K
TASKING tool set,
3 seats, floating licenses $21.3K
Nucleus SHELL ~ 35KB
3 seats (?) $ 3.9K
Nucleus MNT, 2 seats $ 2.0K
Nucleus MNT binary bundle,
2 seats $ 2.0K
Microsoft Visual C++, 6.0,
2 seats for Nucleus MNT cost unknown at this time
This combination of components gives a RAM footprint of 207.5KB, and a cost
of between $96.7K and $116.7K depending on the port costs (not including
the cost of MS Visual C++ for Nucleus MNT). ATI has made it clear that
they have some flexibility in pricing. Probably, they would prefer to
reduce the costs of the software components rather than the port cost
unless they get a firm commitment of financial assistance from Infineon.
We will not have a true, final price until we get a formal quote from ATI.
The price should be no higher than $116.7K, and may be lower.
Additionally, if we are all agreed on the desirability of purchasing MNT
then we will need a Windows NT/2000 workstation for Kevin, and if Wayne and
George are currently running some version of Windows other than NT or 2000,
we may need a Windows NT/2000 box for the electronics area.
My questions are as follows:
- Is the single precision floating point support provided by the TASKING
tool set adequate to our purposes ?
- Do we need 3 seats for the SHELL ?
- Are we interested in the servers and client that are included in the
Nucleus Extended Protocol Package? They are: TFTP server, FTP server,
FTP client, and a Telnet Server. If we think we will want the FTP
server, then we must also purchase a file system & be prepared to
accommodate its resource/media needs. The cost of the Extended Protocol
Package is ~ $7.5K. The cost of Nucleus FILE is ~ $6K.
Toolset Host Environment
The Electronics Division uses Windows based machines. The MNT package
requires Windows NT/2000. The toolset to JTAG port cable required for
downloads of the binaries from the toolset to the target is a parallel port
to JTAG connector cable. The need for a parallel port on one end of the
cable seems to exclude Sun hardware, but Linux on Wintel hardware could
still be used. However, MNT still requires Windows. All-in-all I think we
would avoid unnecessary complications and delays if we settled on one host
environment and the weight of the accumulating facts seems to favor Windows
2000.
Java
CEE-J is worth exploring. I suggest that we do so after we have begun the
paperwork, contract negotiations, etc for the above mentioned components
in order not to lose more time.
JUSTIFICATION OF RECOMMENDATION
It is very reasonable to ask why I have recommended a choice that is
the more expensive by, worst case, some $20K to $40K ($96.7K-$75.2K, &
$116.7K-$75.2K). There are several reason why I favor the ATI-Altium
combination over Express Logic-Interniche-Green Hills.
First and foremost, ATI has offered a degree of integration that is,
basically, an out-of-the-box solution. For the $30K to $50K port cost they
will port the rtos to the TC11IB chip, write the ethernet driver, integrate
the driver with the network stack, the network stack and HTTP server with
the rtos, and create the module needed to provide kernel aware debugging
for the toolset. They will supply fully updated and verified
documentation. For us, this degree of integration means time saved. My
best guess is that the ATI-Altium solution will save us at least one month
over the alternative. Time is our most precious resource. We cannot afford
to sacrifice that month (or more).
ATI offers a much fuller simulation environment. We can take delivery of
Nucleus MNT and the MNT binary bundle immediately, install it on a Windows
box, and begin software development, even while the development board is at
the vendor and they are working on the rtos port and ethernet driver.
Again, we save time, perhaps as much as another month. Express Logic does
offer a Win32 port of ThreadX, but that is the rtos only - no network
stack, no http server, and the cost for the Win32 port of ThreadX is
$12.5K. We could not do effective software development with ThreadX with a
Windows port of only the rtos.
The ATI solution offer the possibility of Java in the MIB. CEE-J is
already ported to Nucleus PLUS, but not to the TriCore architecture. I am
not currently aware of any Java virtual machine which is ported to ThreadX.
We might be able to find a company that would be willing to undertake a
port of a Java virtual machine to both ThreadX & the TriCore architecture,
but the port would involve more work (rtos as well as processor
architecture) which would likely make it more expensive. Also, we might
not be able to find such a company. With CEE-J we have a company (Skelmir)
that is already partnered with ATI, has already ported to Nucleus PLUS and
has already stated that they are interested in doing a TriCore port.
The TASKING toolset already supports both the TriCore architecture and the
TC11IB chip specifically. Green Hills supports the TriCore architecture,
but without specific support of the TC11IB chip.
The dueling vendors problem. With the ATI-Altium solution we deal with two
companies. Altium has been very low profile throughout my investigations.
They promptly answer my queries, but have raised no obstacles or issues.
Altium's work is already done. Their tool set is TC11IB ready. Green
Hills is a reseller of both ThreadX and Internche software. Green Hills
acknowledges that we must buy the rtos directly from Express Logic because
it is a special port, but they are engaging in arm twisting w.r.t. to the
purchase of the Interniche software. On the one hand, Interniche has
indicated that they would substantially discount the price of their
software if we buy a bundled package directly from them (above and beyond
dispensing with the $3K charge for the HTML Compiler). On the other hand,
Green Hills has indicated that they will charge an additional $10K to $25K
for the ThreadX kernel aware package/module if we do not buy the Interniche
software through them (while offering a 5% discount on the Green Hills
tools and no charge for the ThreadX kernel aware module if we do buy the
Interniche software from them). I consider the Green Hills-Interniche
disagreement to be a sample of things to come if we go with the Express
Logic/Interniche/Green Hills solution. We do not need the aggravation
and lost time that would be involved.
Finally there is a grab bag of miscellaneous issues. Several of the people
who will use this software are concerned that the ThreadX rtos may be a bit
too minimal for our application. On the other hand, the code size of the
Nucleus products has been a concern. Both ATI & Express Logic have
provided references to customers who have used their tools. I am still
trying to contact the ThreadX reference. I was able to contact the ATI
reference and was pleased by what I discovered. The reference supplied to
me by ATI is to a consultant who had been contracted to work on a high
speed, network based communication box whose purpose was to perform ATM
transactions, credit card charges, and the like. The emphasis was on the
ability of the box to perform a high volume of short transactions. The
performance of the network stack was crucial to the success of this
application so a performance evaluation was done which compared the network
stacks offered by Fusion, Interniche, and ATI. The tests were performed
using a PowerPC 860 based system. Without going into a great deal of
detail, I will give a quick summary table of a subset of the results -
total transaction time, and transactions per second (cache not used) where
a transaction was defined as the time it took for an ethernet message to
move from the wire up to the application level, and then down through all
of the software layers and back onto the wire.
Interniche Nucleus Net
total transaction time 3.05 ms 1.77 ms
transactions/second 328 564
While my report of these results is informal to the point of being
anecdotal, the results do indicate that the Nucleus Net software gave
superior performance. It should be noted that the figures reported for
Interniche apply to the case of having corrected a bug found in the code
(the evaluater had source code) which was causing the Interniche stack to
underperform by a factor of 3. I was also struck by the comments made
during our discussion of the software. This particular reference felt that
the Nucleus source code was superior in structure and commenting. More
specifically, the Nucleus code is structured in such a manner that is was
easy to separate control structures from buffers in such a way that the
PowerPC cache could be used to gain significant performance improvements.
The Interniche code had so co-mingled local variables, buffers, buffer
descriptors, etc, that it was not possible to effectively utilize the
PowerPC cache for the control structures. Again because of superior code
structure, he also found the Nucleus code easier to integrate. Typically
he could get the Nucleus PLUS rtos up and running on PowerPC, X86, and
68XXX architectures in half a day, while the code from other vendors
required a week or more of work.
In summary, my choice is the ATI-Altiun (Nucleus-TASKING) combination
because that solution offers superior integration, and a superior
simulation environment that taken together could save us months of time.
Time is crucial. The greater amount of invested time that seems to be
required by the Express Logic/Interniche/Green Hills solution would
virtually guarantee that we will miss, by months, the deadlines for bench
testing and test antenna support of the EVLA monitor and control system.
With the ATI-Altium combination we have at least a chance of meeting those
deadlines. When the possibility of Java in the MIB, specific support for
the TC11IB chip in the TASKING tool set, a basically one vendor solution,
and possibly superior performance are added, the choice is clear and the
extra cost justified.
More information about the evla-sw-discuss
mailing list