[evlatests] Subreflector rotation statistics

Wed Apr 3 16:32:16 EDT 2019

Rick,

Yes, this analysis looked exclusively at rotation flags (which 
unfortunately have the generic label "SUBREFLECTOR_ERROR" in the SDM).  
Focus flags are counted separately and are labeled "FOCUS_ERROR".  
That's not to say there are no problems with focus, I just haven't 
looked at it yet.

-Paul

On 2019-04-03 14:28, Rick Perley wrote:
>     Paul, et al.:
> 
>     Are you sure it is only *rotation*, as opposed to focus?  In the
> various test I do, both are involved.  I'll soon have better
> information, as I'm nearly ready to seriously reduce the 'flux
> density' test data.
> 
>     Rick
> 
> 
> On 04/03/2019 02:15 PM, Paul Demorest wrote:
>> hi all,
>> 
>> This is a report on an analysis of VLA subreflector rotation times I 
>> recently did (some of you will have seen a version of this already; 
>> there is a little new info in here but no change in basic 
>> conclusions).  This was motivated by recent anecdotal reports from 
>> operators and analysts about specific antennas often being flagged due 
>> to subreflector rotation for much longer than expected, sometimes 
>> resulting in their missing calibrators, etc.  I thought it would be 
>> useful to take a more systematic look at recent data for problems like 
>> this.
>> 
>> First, the main conclusions are:
>> 
>>  - There are several "bad" antennas that frequently spend >~10x the 
>> time flagged due to subreflector rotation as the rest, often for 
>> minutes at a time.  These are ea05, ea11, ea22, ea23, and ea25.  These 
>> should be prioritized for FRM maintenance if possible.
>> 
>>  - There are a few more "marginal" ones that show similar behavior but 
>> not quite as severe (ea09, ea10, ea12, ea13, ea15).
>> 
>>  - All the "bad" and "marginal" antennas have old ACUs.
>> 
>>  - Not all old-ACU antennas act badly, for example ea03 and ea04 look 
>> generally pretty well-behaved.  But even these "good" examples spend 
>> typically ~50% more time flagged than new-ACU antennas.  So the new 
>> ACUs and associated mechanical overhaul are clearly an improvement 
>> (this is probably not news to many of you!).
>> 
>> More details about this analysis:
>> 
>> I gathered data on this from the SDMs currently available in the MCAF 
>> workspace.  Right now this goes back to the beginning of the year.  To 
>> avoid confusion from test/maint time, I only counted real science 
>> observations, identified as those datasets that start with '1' or 'V'.
>> 
>> For each day (MJD) I add up all the time each antenna is listed as 
>> being in the SUBREFLECTOR_ERROR state in Flag.xml.  This only counts 
>> rotation errors (I haven't looked at focus but could in the future).  
>> Since there will be different numbers of band changes each day, I then 
>> divide all the times by the median of the 10 best (least flagged) 
>> antennas for that day.
>> 
>> For a second statistic, I also looked at the duration of each flag 
>> event.  For reference, a typical subreflector rotation for a band 
>> change should take somewhere between 5 and 25 seconds depending on 
>> which bands are in use; Rick took a close look at this recently, see 
>> his emails to this list in Nov 2018 titled "Band Change Times."  The 
>> assumption that band changes take ~20s is baked into our software in 
>> several places (OPT, observing scripts).  I counted up all the 
>> instances where an antenna was flagged for >30s or >120s, these will 
>> be potentially bad for observations.
>> 
>> Both of these metrics are plotted versus antenna number for a week's 
>> worth of data at a time (starting on Wednesday evenings). The rotation 
>> time plot has one point per antenna per day for a week.  The flag 
>> duration counts are cumulative for the whole week.  See attached png 
>> showing the most recent week, and pdf showing all available data.
>> 
>> This analysis has an implicit assumption that all antennas are getting 
>> commanded to do the same thing.  This will occasionally not be true, 
>> for example if an antenna is removed from observing for part of a day 
>> for some reason.  So isolated data points away from 1.0, or small 
>> non-zero numbers of long-duration flags can probably be ignored.  But 
>> long-term patterns where certain antennas have consistently 
>> high/scattered points or many long-duration flags are meaningful, for 
>> example the "bad" ones I mentioned above.
>> 
>> The other situation that may confuse this analysis somewhat is 
>> subarray observations.  To help avoid this, I've excluded all datasets 
>> that used less than 24 antennas.  There may be some residual effect on 
>> the first full-array observation following a subarray project since 
>> the antennas will have different starting subreflector positions.  
>> These have not been removed since they are more difficult to 
>> automatically identify.  But I think this happens infrequently enough 
>> that it's not a big problem.
>> 
>> Please let me know if you have comments, suggestions, or questions 
>> about any of this.
>> 
>> Cheers,
>> Paul
>> 
>> 
>> 
>> _______________________________________________
>> evlatests mailing list
>> evlatests at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/evlatests
> 
> 
> _______________________________________________
> evlatests mailing list
> evlatests at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/evlatests