[evlatests] Subreflector rotation (and focus) statistics

Tue Apr 9 13:30:24 EDT 2019

It'd be good to verify that it's worn gears (and not, say, bad bearings) 
on those antennas though.

	-Bryan

Gene Cole wrote on 4/9/19 09:19:
> Paul,
> 
> With regard to rotation, we suspect worn gears to be the problem. Many of these antennas use aluminum gears that wear down over time.  As with the new ACU upgrade, steel gears are used as replacements.
> 
> Doug Whiton will conduct FRM repeatability tests on all the antennas in question.  After reviewing this data, the mechanics can start servicing the antennas in the array.  We need to order new gears, so the lead-time of this will drive any progress toward this.  Ea9 and ea22 are due for ACU upgrades this year, so we will wait until then to address them.
> 
> --Gene
> 
> -----Original Message-----
> From: evlatests <evlatests-bounces at listmgr.nrao.edu> On Behalf Of Paul Demorest
> Sent: Wednesday, April 03, 2019 5:39 PM
> To: Rick Perley <rperley at nrao.edu>
> Cc: evlatests at listmgr.nrao.edu
> Subject: Re: [evlatests] Subreflector rotation (and focus) statistics
> 
> So it was easy enough to run the same script and look at the focus flag rather than rotation.  Attached is the same set of plots as previously, just for focus now.
> 
> There do some to be some consistently poor antennas here as well (ea03, ea05, ea13, ea26).  These are also all old-ACU antennas, but annoyingly this set is almost completely disjoint from the rotation bad-list, aside from ea05.  However, the scale of the problem overall seems less severe in focus, so I'd still say the "bad" list from my original email are the ones to pay attention to first.
> 
> -Paul
> 
> On 2019-04-03 14:32, Paul Demorest wrote:
>> Rick,
>>
>> Yes, this analysis looked exclusively at rotation flags (which
>> unfortunately have the generic label "SUBREFLECTOR_ERROR" in the SDM).
>>   Focus flags are counted separately and are labeled "FOCUS_ERROR".
>> That's not to say there are no problems with focus, I just haven't
>> looked at it yet.
>>
>> -Paul
>>
>> On 2019-04-03 14:28, Rick Perley wrote:
>>>      Paul, et al.:
>>>
>>>      Are you sure it is only *rotation*, as opposed to focus?  In the
>>> various test I do, both are involved.  I'll soon have better
>>> information, as I'm nearly ready to seriously reduce the 'flux
>>> density' test data.
>>>
>>>      Rick
>>>
>>>
>>> On 04/03/2019 02:15 PM, Paul Demorest wrote:
>>>> hi all,
>>>>
>>>> This is a report on an analysis of VLA subreflector rotation times I
>>>> recently did (some of you will have seen a version of this already;
>>>> there is a little new info in here but no change in basic
>>>> conclusions).  This was motivated by recent anecdotal reports from
>>>> operators and analysts about specific antennas often being flagged
>>>> due to subreflector rotation for much longer than expected,
>>>> sometimes resulting in their missing calibrators, etc.  I thought it
>>>> would be useful to take a more systematic look at recent data for
>>>> problems like this.
>>>>
>>>> First, the main conclusions are:
>>>>
>>>>   - There are several "bad" antennas that frequently spend >~10x the
>>>> time flagged due to subreflector rotation as the rest, often for
>>>> minutes at a time.  These are ea05, ea11, ea22, ea23, and ea25.
>>>> These should be prioritized for FRM maintenance if possible.
>>>>
>>>>   - There are a few more "marginal" ones that show similar behavior
>>>> but not quite as severe (ea09, ea10, ea12, ea13, ea15).
>>>>
>>>>   - All the "bad" and "marginal" antennas have old ACUs.
>>>>
>>>>   - Not all old-ACU antennas act badly, for example ea03 and ea04
>>>> look generally pretty well-behaved.  But even these "good" examples
>>>> spend typically ~50% more time flagged than new-ACU antennas.  So
>>>> the new ACUs and associated mechanical overhaul are clearly an
>>>> improvement (this is probably not news to many of you!).
>>>>
>>>> More details about this analysis:
>>>>
>>>> I gathered data on this from the SDMs currently available in the
>>>> MCAF workspace.  Right now this goes back to the beginning of the year.
>>>> To avoid confusion from test/maint time, I only counted real science
>>>> observations, identified as those datasets that start with '1' or
>>>> 'V'.
>>>>
>>>> For each day (MJD) I add up all the time each antenna is listed as
>>>> being in the SUBREFLECTOR_ERROR state in Flag.xml.  This only counts
>>>> rotation errors (I haven't looked at focus but could in the future).
>>>> Since there will be different numbers of band changes each day, I
>>>> then divide all the times by the median of the 10 best (least
>>>> flagged) antennas for that day.
>>>>
>>>> For a second statistic, I also looked at the duration of each flag
>>>> event.  For reference, a typical subreflector rotation for a band
>>>> change should take somewhere between 5 and 25 seconds depending on
>>>> which bands are in use; Rick took a close look at this recently, see
>>>> his emails to this list in Nov 2018 titled "Band Change Times."  The
>>>> assumption that band changes take ~20s is baked into our software in
>>>> several places (OPT, observing scripts).  I counted up all the
>>>> instances where an antenna was flagged for >30s or >120s, these will
>>>> be potentially bad for observations.
>>>>
>>>> Both of these metrics are plotted versus antenna number for a week's
>>>> worth of data at a time (starting on Wednesday evenings). The
>>>> rotation time plot has one point per antenna per day for a week.
>>>> The flag duration counts are cumulative for the whole week.  See
>>>> attached png showing the most recent week, and pdf showing all available data.
>>>>
>>>> This analysis has an implicit assumption that all antennas are
>>>> getting commanded to do the same thing.  This will occasionally not
>>>> be true, for example if an antenna is removed from observing for
>>>> part of a day for some reason.  So isolated data points away from
>>>> 1.0, or small non-zero numbers of long-duration flags can probably
>>>> be ignored.  But long-term patterns where certain antennas have
>>>> consistently high/scattered points or many long-duration flags are
>>>> meaningful, for example the "bad" ones I mentioned above.
>>>>
>>>> The other situation that may confuse this analysis somewhat is
>>>> subarray observations.  To help avoid this, I've excluded all
>>>> datasets that used less than 24 antennas.  There may be some
>>>> residual effect on the first full-array observation following a
>>>> subarray project since the antennas will have different starting
>>>> subreflector positions.  These have not been removed since they are
>>>> more difficult to automatically identify.  But I think this happens
>>>> infrequently enough that it's not a big problem.
>>>>
>>>> Please let me know if you have comments, suggestions, or questions
>>>> about any of this.
>>>>
>>>> Cheers,
>>>> Paul
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> evlatests mailing list
>>>> evlatests at listmgr.nrao.edu
>>>> https://listmgr.nrao.edu/mailman/listinfo/evlatests
>>>
>>>
>>> _______________________________________________
>>> evlatests mailing list
>>> evlatests at listmgr.nrao.edu
>>> https://listmgr.nrao.edu/mailman/listinfo/evlatests
>>
>> _______________________________________________
>> evlatests mailing list
>> evlatests at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/evlatests
> 
> _______________________________________________
> evlatests mailing list
> evlatests at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/evlatests
> 