<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Michael,<br>
see my comments below.<br>
Sonja <br>
<br>
<div class="moz-cite-prefix">On 04/01/2014 11:32 AM, Michael Rupen
wrote:<br>
</div>
<blockquote
cite="mid:Pine.LNX.4.64.1404011231560.25240@scamper.aoc.nrao.edu"
type="cite">
<pre wrap="">[Daniel/Rick: please forward this to evla-sss-discuss, to which I'm not
allowed to submit...or maybe I have the name wrong??]
This is a proposal for a simple approach to scientific subarrays which I
believe would handle the most scientifically interesting cases, without being
too painful for the SSA group to implement (the latter based on conversations
with Daniel).
1- Define a linked set of a master and multiple slave SBs, one SB
per subarray.
- This is done in the OPT.
- Master and slave SBs are written independently by the observer, and
use completely independent resources.
- It is the observer's responsibility to obey subarray restrictions
(in much the same way that she's responsible for putting in dummy scans,
ensuring sufficient time on-source, etc., for regular observing).
: If it's easy it would be a good idea to automatically check that
the lengths of each SB in the set are consistent, and to COMPLAIN
LOUDLY (i.e., don't let the observer submit the SBs) if any
slave SB is longer than the master SB.
- The master and slave SBs are clearly marked as such in the OPT
displays.
- There is an additional display in the header tab of subarray SBs,
showing which antennas to use for this subarray. Ideally the OPT
would check that these are consistent (i.e., that no antennas are
duplicated, and that restrictions on the distribution of antennas
between subarrays are obeyed), but this is a nicety.
: The simplest display would simply show the set of antennas with
check boxes, ideally with checks of different colors for the
different subarrays:
ea01 ... ea09
ea10 ... ea18
ea19 ... ea27
ea28 (PT, someday)
: Another nicety would be to allow the observer to specify antennas
by location instead of number. Until this is available it would
be nice to have a link to the antenna positions file, which is
available on the VLA home page:
<a class="moz-txt-link-freetext" href="http://www.vla.nrao.edu/operators/CurrentPos.pdf">http://www.vla.nrao.edu/operators/CurrentPos.pdf</a>
This is not ideal, as that page generally shows the current
configuration rather than the configuration for which the SB is
written (e.g., one might write an A cfg SB before that
configuration actually begins), but that's life.
: If listing of antennas is allowed we could accurately calculate
the data output rates for the subarrays. The SBs in the
set could report both that SB's data rate and the total for the
entire set; the latter would then be used to check whether the
observer exceeds currently-allowed data rates. We have already
had one subarray observation which requested faster dump rates
than the OPT/RCT allowed, but which were fine given the reduced
number of baselines (Osten et al. stellar observations).
...This would require some interaction between OPT and RCT. One
approach would be to allow resources to be marked as "Subarray
resources", and have the observer enter the maximum number of
antennas to be used with that resource in the RCT. The reported
data rate (and RCT restrictions on that data rate) would then be
calculated using that maximum number of antennas. Such subarray
resources could only be used in SBs which are part of a set of
subarray SBs, and the maximum number of antennas allowed for
that SB could be compared with the number of antennas actually
selected for that SB, with errors resulting if they are not
consistent.
...Such an approach could be readily extend to those wishing
to trade antennas for integration time (e.g., to allow
faster dumps as GO or OSRO for resources which use only
20 antennas).
2- The master SB (only!) is used for scheduling the experiment.
- The OST only looks at the master's scheduling requirements (weather,
LST start times, etc.).
: Ideally the slave SBs do not even show scheduling requirements
as options in the OPT, instead showing "tied to Master SB #XXXXX".
- When the OST displays the master (i.e., when it's put on the
suggested schedule), the master is marked using a special
(colorful?) label, to show this is a multi-SB script.
3- When the master is selected by the operator:
- m2s is run automatically for all SBs in the set (both master and
slaves) -- see also below!
- The operator is then prompted as usual to run each SB in the set.
Ideally the antennas in that subarray are pre-populated in the
screen that pops up for each SB.
: Daniel tells me this is easy. If it's not, just do a separate
popup for each SB, with some text telling the operator to use
the list of antennas in that SB's comments (which could also pop
up automagically). This sounds unwieldy, but note that it's
basically what we ask the operator to do now!
- When the SBs are successfully executed, charge the observing program
for the master SB's time (only).
4- Staggered starts & simultaneous reconfigurations
The most common (only?) current failure mode for subarrays occurs
when the correlator is reconfigured at the same time in multiple
subarrays. The worst case appears to be at the beginning of the
script, when typical high-frequency subarrays have a dummy scan with
one configuration followed by a dummy scan with another; the first
scan of a script is special because both of those configurations are
sent to the CM in rapid succession, so the CM (and the CBE and
everything else) is very busy.
Currently we try to minimize this problem by offsetting the start
times of the various subarrays: we tell the operator in the nightly
note to start SB 1 on these antennas, then wait 30sec before starting
SB 2 on these other antennas, etc.
Ideally we would fix the underlying problem, but that has proved
elusive, and subarrays have only occasionally risen to a sufficient
priority that we look seriously at debugging and/or fixing this. So
for now I assume we remain in the current state. How do we handle
this in the proposed scheme?
I see several possible approaches.
A. Do what we do now: when the operator selects a subarray experiment
from the OST list, pop up a message telling him:
(1) do not queue up these subarray SBs
(2) wait 30secs (-ish) between submitting each SB in the set to
the Executor
B1. Modify m2s to automatically insert n*30sec of blank time (doing
NOTHING -- definitely not talking to the CM!) in the .evla scripts
for the slave SBs.
...Ken tells me this is simple to do.
...This has the advantage that the operator could submit them
all at once, and in advance.
B2. Modify m2s to insert an n*30sec-long dummy scan using the
same resource as the 1st scan of the observation, to avoid piling
up multiple configuration messages at the start of an SB.
...Sounds good but I would want to test this before going too crazy.
...Like B1, the operator could then submit the SBs just like normal
ones.
C. Enforce non-overlapping reconfigurations in the OPT.
...This sounds nice but also sounds painful; and I really hope that
in the end we eliminate this restriction, so this would become
"throwaway code".
There are doubtless many other possibilities. I'm leaning towards B
at the moment but we should do some tests to be sure this actually
works.
Note that both A and B assume that the subarrays are changing
configurations (if at all) at the same time in the SB (before we put
in any blanking). If the subarrays are truly independent this is not
a given. However, all the scientific subarrays I've seen so far have
indeed changed configurations in lock step, if they change
configurations at all.
Another problem with A and B is that the execution time of all the
subarrays is increased by up to a couple minutes, because
we've added dead time. We could calculate this in the OPT (by
charging each "slave" SB the extra n*30sec) or just live with it until
we fix the fundamental problem. Those who know more about the OST
will have to comment on any scheduling issues (e.g., we used to have
to start SBs on 15min boundaries, which would mean that staggered starts
effectively add 15min to the total observing time).</pre>
</blockquote>
<br>
Example: VCI documents (messages) for a set of three subarrays,
where master SB uses subarray A, while 'slave' SBs 1 and 2 use
subarrays B and C. <br>
<br>
Please correct me if I am wrong, but I believe that the Executor now
transmits the following: <br>
<br>
<ol>
<li>subArray-A-Obs123-Master, ActivationID=startSubA</li>
<li>actTrigger, activationID=startSubA, actTime=12:30:00</li>
<li>subArray-B-Obs123-Slave1, ActivationID=startSubB</li>
<li>actTrigger, activationID=startSubB, actTime=12:30:00</li>
<li>subArray-C-Obs123-Slave2, ActivationID=startSubC</li>
<li>actTrigger, activationID=startSubC, actTime=12:30:00</li>
</ol>
<br>
(actually activation time is staggered by 30sec or so to avoid
problems).<br>
<br>
If I am right, that there is a way around this problem, but I am not
sure how difficult it would be to make changes in the Executor to
use a single Activation Trigger for all SBs in a set (master +
slaves) and to transmit this: <br>
<ol>
<li>subArray-A-Obs123-Master, ActivationID=startObs123</li>
<li>subArray-B-Obs123-Slave1, ActivationID=startObs123</li>
<li>subArray-C-Obs123-Slave2, ActivationID=startObs123</li>
<li>actTrigger, activationID=startObs123, actTime=12:30:00</li>
</ol>
CM places received configuration messages (create/delete subarray)
in the configuration queue. <br>
Configuration messages are removed from the queue (and processed)
when Activation Trigger is received. <br>
To process all 3 messages as a single configuration, Executor would
have to assign the same Activation ID to all SBs (configuration
messages) that should be activated at the same time, and transmit a
single Activation Trigger with that Activation ID<i> after all
messages are transmitted</i> (master + slaves).<br>
<br>
Note: the order in which messages 'create subarray' are transmitted
does not matter, but the Activation Trigger must be transmitted
last, to ensure that CM has received all subarrays before Activation
Trigger is received. <br>
<p>In this case, CM would process all three subarrays one after the
other, forward configuration to CBE for all three subarrays one
after the other; but this<i> should not </i>cause the type of
problems that we see now (or should at least minimize them).<br>
</p>
<br>
<br>
<blockquote
cite="mid:Pine.LNX.4.64.1404011231560.25240@scamper.aoc.nrao.edu"
type="cite">
<pre wrap="">
5- Subarray resource restrictions
I claim that our current restriction on subarrays (that they not share
rows/columns on any Baseline Board) means that subarrays are truly
independent in terms of correlator use -- subarrays can use any
currently-allowed GO or OSRO (not RSRO!) setup, and those setups are
completely independent from subarray to subarray. Similarly I claim
that there are no additional restrictions related to the crossbar
boards. [But see below, as reality does intrude on these ideals...]
- I do not include phased array or VDIF outputs in these claims.
- Contrary to the above...
: the CBE's assignment of BlB outputs to CBE node NICs might set
some restrictions on the use of BlB stacking
: the limited number of DUMPTRIGs (and current limits on routing those
DUMPTRIGs) may lead to restrictions on recirculation (as well as on
phase binning, intermittent dumps, and very fast dumps -- but those
are all RSRO anyhow).</pre>
</blockquote>
<br>
When you refer to "the limited number of DUMPTRIGs" isn't that the
number of DUMPTRIGs per Station Board, i.e. per antenna. <br>
Nominally 16, but CMIB is not able to support that many. But that
just limits the nummber of different DUMPTRIGS for the subbands of
the same antenna. <br>
<pre wrap="">Not sure what are the restrictions on routing the DUMPTRIGs. Perhaps I forgot.
As far as the correlator is concerned, there should be no restriction for DUMPTRIGs in different subarrays.
</pre>
<br>
<blockquote
cite="mid:Pine.LNX.4.64.1404011231560.25240@scamper.aoc.nrao.edu"
type="cite">
<pre wrap="">
For now I believe it's safe to allow any continuum experiments
(3-bit or 8-bit), and I think we could write down some simple (and
not overly onerous) restrictions on spectral line setups which would
avoid these problems. To my knowledge there have yet to be any
approved spectral line subarray experiments, so simply not allowing
those is probably fine for quite a long time. [Basically this
translates to disallowing recirculation and BlB stacking when using
subarrays. If we use the "subarray" tag in the RCT, as outline
above, the RCT could enforce those restrictions on all subarray
resources.]
- We may have to explicitly call out the requested autoCorrelation
request in the .vci scripts for subarrays, to ensure they do not
"leak over" into extra rows/columns. I'm not sure where this stands
at the moment.
I believe this proposal covers all science cases we have seen thus far
except those which violate non-subarray restrictions (e.g., very fast
dump times). For non-RSRO experiments the proposed implementation:
- avoids manual m2s/edits
- allows dynamic scheduling
- allows proper (automatic) accounting of observing hours
- covers almost all of the interesting science cases
- does not place too great a burden on our very limited programming
and testing/commissioning staff
- should be reasonably easy to explain to garden-variety observers
Comments?
Michael
_______________________________________________
evla-sw-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:evla-sw-discuss@listmgr.cv.nrao.edu">evla-sw-discuss@listmgr.cv.nrao.edu</a>
<a class="moz-txt-link-freetext" href="http://listmgr.cv.nrao.edu/mailman/listinfo/evla-sw-discuss">http://listmgr.cv.nrao.edu/mailman/listinfo/evla-sw-discuss</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Sonja Vrcic
Team Lead Software
National Research Council Of Canada
Dominion Radio Astrophysical Observatory
<a class="moz-txt-link-abbreviated" href="mailto:Sonja.Vrcic@nrc-cnrc.gc.ca">Sonja.Vrcic@nrc-cnrc.gc.ca</a>
(250)497-2309 or (250)497-2300 ext.309
<a class="moz-txt-link-freetext" href="http://www.nrc-cnrc.gc.ca/">http://www.nrc-cnrc.gc.ca/</a>
</pre>
</body>
</html>