[evlatests] Failures during last night's tests

[Michael Rupen]

Note that the CM also complained that the CBE was not responding to
configurations; I believe this shows up on the operators' screen these days,
but would have to check to be sure.

          Michael

>
> the references to the times when the operator gets the message about
> missing scans are only because my cron job to try to detect them only
> runs every 5 minutes.  that is the granularity of the messages.  there
> are alerts which we could pull out of the alert database (sent by MCAF)
> which would be more informative, but i don't think it's worth the
> effort, since we know what scans were missed now already.
>
> i think this is all attributable to the CBE not getting configured
> properly - both the zeroes and the missing scans.  we should test, going
> back to slightly longer scans.
>
> 	-bryan
>
>
> Rick Perley wrote, On 1/5/12 10:33 AM:
>>      Another 'flux densities' test last night suffered multiple
>> failures.  Although similar to those from two nights ago, there are
>> significant differences.
>>
>>      I attempted to observe six sources, in this order:
>>
>>      3C84, 3C138, 3C147, 3C454.3, Uranus, J0006-0623.
>>
>>      For each source, the following sequence of bands was made:
>>
>>      L, XRefPnt, X, C, S, U, K, Q.
>>
>>      Except for the referenced pointing, the times were arranged to
>> provide between 15 and 20 seconds of good data.  Typically, this means
>> each band is observed for 20 to 35 seconds (the difference being due to
>> how long it takes to reconfigure the correlator and/or move the
>> subreflector).
>>
>>      The major fault (from the observer's point of view) was missing scans:
>>
>>      3C84 missed Ku, K, Q, and Ka bands.
>>      3C138 missed Ku band.
>>      3C454.3 missed S, Ku, K, Q, and Ka bands.
>>      The last two sources provided no data at all.
>>
>>      The problem which affected the data from two days ago -- zero
>> amplitude and zero weight data on approximately half of the antennas --
>> did *not* occur in last night's test.  The only issue (from the
>> observer's perspective) is the missing scans.
>>
>>      The operator's message, sent following the run, is reproduced
>> below.  The first alert he references is 44 seconds following the last
>> valid data (2:55:36).  Presumably, these missing data are all 'pure
>> zeros', and hence are stripped out upon filling.
>>      The operator's message also references emails about missing BDFs --
>> he lists three messages, but there are five missing scans.  The times
>> noted are a poor match to the missing scans:  at 2:35, we were getting
>> good data at X-band on 3C138; at 2:40, we were doing a referenced
>> pointing run on 3C147 -- which gave excellent solutions, and at 3:00, we
>> were on or near Uranus (hard to know for sure, since no data showed up).
>>
>>
>>
>> -------- Original Message --------
>> Subject: 	CBE config failure during TCAL0005
>> Date: 	Wed, 04 Jan 2012 20:20:23 -0700
>> From: 	EVLA Operations<observe at nrao.edu>
>> Organization: 	National Radio Astronomy Observatory
>> To: 	Michael Rupen<mrupen at aoc.nrao.edu>, Bryan Butler
>> <bbutler at nrao.edu>, Rick Perley<rperley at nrao.edu>, Ken Sowinski
>> <ksowinsk at nrao.edu>
>>
>>
>>
>> Hello.  The following alert appeared at 2:56:20, 2:58:23, 3:03:57,
>> 3:05:56, 3:10:45, and 3:14:27 UTC:
>>
>> mccccm timerExpiredFor:TCAL0005.sb7289738 config-failure-cbe
>>
>> Also received 3 missing BDF e-mails at 2:35, 2:40, and 3:00 UTC.
>>
>> After TCAL0005 ended I ran the clearCorrelator script and then X_Osro.
>>
>> The CM screen for X_osro displays Failed for the CBE configure status.
>>
>> Dave
>> EVLA Ops
>>
>> _______________________________________________
>> evlatests mailing list
>> evlatests at listmgr.cv.nrao.edu
>> http://listmgr.cv.nrao.edu/mailman/listinfo/evlatests
> _______________________________________________
> evlatests mailing list
> evlatests at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/evlatests
>