[evlatests] Subreflector rotation statistics

[Paul Demorest]

hi all,

This is a report on an analysis of VLA subreflector rotation times I 
recently did (some of you will have seen a version of this already; 
there is a little new info in here but no change in basic conclusions).  
This was motivated by recent anecdotal reports from operators and 
analysts about specific antennas often being flagged due to subreflector 
rotation for much longer than expected, sometimes resulting in their 
missing calibrators, etc.  I thought it would be useful to take a more 
systematic look at recent data for problems like this.

First, the main conclusions are:

  - There are several "bad" antennas that frequently spend >~10x the time 
flagged due to subreflector rotation as the rest, often for minutes at a 
time.  These are ea05, ea11, ea22, ea23, and ea25.  These should be 
prioritized for FRM maintenance if possible.

  - There are a few more "marginal" ones that show similar behavior but 
not quite as severe (ea09, ea10, ea12, ea13, ea15).

  - All the "bad" and "marginal" antennas have old ACUs.

  - Not all old-ACU antennas act badly, for example ea03 and ea04 look 
generally pretty well-behaved.  But even these "good" examples spend 
typically ~50% more time flagged than new-ACU antennas.  So the new ACUs 
and associated mechanical overhaul are clearly an improvement (this is 
probably not news to many of you!).

More details about this analysis:

I gathered data on this from the SDMs currently available in the MCAF 
workspace.  Right now this goes back to the beginning of the year.  To 
avoid confusion from test/maint time, I only counted real science 
observations, identified as those datasets that start with '1' or 'V'.

For each day (MJD) I add up all the time each antenna is listed as being 
in the SUBREFLECTOR_ERROR state in Flag.xml.  This only counts rotation 
errors (I haven't looked at focus but could in the future).  Since there 
will be different numbers of band changes each day, I then divide all 
the times by the median of the 10 best (least flagged) antennas for that 
day.

For a second statistic, I also looked at the duration of each flag 
event.  For reference, a typical subreflector rotation for a band change 
should take somewhere between 5 and 25 seconds depending on which bands 
are in use; Rick took a close look at this recently, see his emails to 
this list in Nov 2018 titled "Band Change Times."  The assumption that 
band changes take ~20s is baked into our software in several places 
(OPT, observing scripts).  I counted up all the instances where an 
antenna was flagged for >30s or >120s, these will be potentially bad for 
observations.

Both of these metrics are plotted versus antenna number for a week's 
worth of data at a time (starting on Wednesday evenings).  The rotation 
time plot has one point per antenna per day for a week.  The flag 
duration counts are cumulative for the whole week.  See attached png 
showing the most recent week, and pdf showing all available data.

This analysis has an implicit assumption that all antennas are getting 
commanded to do the same thing.  This will occasionally not be true, for 
example if an antenna is removed from observing for part of a day for 
some reason.  So isolated data points away from 1.0, or small non-zero 
numbers of long-duration flags can probably be ignored.  But long-term 
patterns where certain antennas have consistently high/scattered points 
or many long-duration flags are meaningful, for example the "bad" ones I 
mentioned above.

The other situation that may confuse this analysis somewhat is subarray 
observations.  To help avoid this, I've excluded all datasets that used 
less than 24 antennas.  There may be some residual effect on the first 
full-array observation following a subarray project since the antennas 
will have different starting subreflector positions.  These have not 
been removed since they are more difficult to automatically identify.  
But I think this happens infrequently enough that it's not a big 
problem.

Please let me know if you have comments, suggestions, or questions about 
any of this.

Cheers,
Paul


-------------- next part --------------
A non-text attachment was scrubbed...
Name: 58570.subrot.png
Type: image/png
Size: 56976 bytes
Desc: not available
URL: <http://listmgr.nrao.edu/pipermail/evlatests/attachments/20190403/a36a96c2/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: subrot.all.pdf
Type: application/pdf
Size: 101759 bytes
Desc: not available
URL: <http://listmgr.nrao.edu/pipermail/evlatests/attachments/20190403/a36a96c2/attachment-0001.pdf>