[evlatests] Subreflector rotation (and focus) statistics

Dave Parker dparker at nrao.edu
Tue Apr 9 15:25:15 EDT 2019 

I believe Antenna 25 is a completely different issue then these problems 
we are seeing with the sub-reflectors. My understanding of the problems 
being addressed here are that the sub-reflector is slow getting to position.

Dave

On 4/9/2019 11:30 AM, Bryan Butler wrote:
> 
> It'd be good to verify that it's worn gears (and not, say, bad bearings) 
> on those antennas though.
> 
>      -Bryan
> 
> 
> Gene Cole wrote on 4/9/19 09:19:
>> Paul,
>>
>> With regard to rotation, we suspect worn gears to be the problem. Many 
>> of these antennas use aluminum gears that wear down over time.  As 
>> with the new ACU upgrade, steel gears are used as replacements.
>>
>> Doug Whiton will conduct FRM repeatability tests on all the antennas 
>> in question.  After reviewing this data, the mechanics can start 
>> servicing the antennas in the array.  We need to order new gears, so 
>> the lead-time of this will drive any progress toward this.  Ea9 and 
>> ea22 are due for ACU upgrades this year, so we will wait until then to 
>> address them.
>>
>> --Gene
>>
>> -----Original Message-----
>> From: evlatests <evlatests-bounces at listmgr.nrao.edu> On Behalf Of Paul 
>> Demorest
>> Sent: Wednesday, April 03, 2019 5:39 PM
>> To: Rick Perley <rperley at nrao.edu>
>> Cc: evlatests at listmgr.nrao.edu
>> Subject: Re: [evlatests] Subreflector rotation (and focus) statistics
>>
>> So it was easy enough to run the same script and look at the focus 
>> flag rather than rotation.  Attached is the same set of plots as 
>> previously, just for focus now.
>>
>> There do some to be some consistently poor antennas here as well 
>> (ea03, ea05, ea13, ea26).  These are also all old-ACU antennas, but 
>> annoyingly this set is almost completely disjoint from the rotation 
>> bad-list, aside from ea05.  However, the scale of the problem overall 
>> seems less severe in focus, so I'd still say the "bad" list from my 
>> original email are the ones to pay attention to first.
>>
>> -Paul
>>
>> On 2019-04-03 14:32, Paul Demorest wrote:
>>> Rick,
>>>
>>> Yes, this analysis looked exclusively at rotation flags (which
>>> unfortunately have the generic label "SUBREFLECTOR_ERROR" in the SDM).
>>>   Focus flags are counted separately and are labeled "FOCUS_ERROR".
>>> That's not to say there are no problems with focus, I just haven't
>>> looked at it yet.
>>>
>>> -Paul
>>>
>>> On 2019-04-03 14:28, Rick Perley wrote:
>>>>      Paul, et al.:
>>>>
>>>>      Are you sure it is only *rotation*, as opposed to focus?  In the
>>>> various test I do, both are involved.  I'll soon have better
>>>> information, as I'm nearly ready to seriously reduce the 'flux
>>>> density' test data.
>>>>
>>>>      Rick
>>>>
>>>>
>>>> On 04/03/2019 02:15 PM, Paul Demorest wrote:
>>>>> hi all,
>>>>>
>>>>> This is a report on an analysis of VLA subreflector rotation times I
>>>>> recently did (some of you will have seen a version of this already;
>>>>> there is a little new info in here but no change in basic
>>>>> conclusions).  This was motivated by recent anecdotal reports from
>>>>> operators and analysts about specific antennas often being flagged
>>>>> due to subreflector rotation for much longer than expected,
>>>>> sometimes resulting in their missing calibrators, etc.  I thought it
>>>>> would be useful to take a more systematic look at recent data for
>>>>> problems like this.
>>>>>
>>>>> First, the main conclusions are:
>>>>>
>>>>>   - There are several "bad" antennas that frequently spend >~10x the
>>>>> time flagged due to subreflector rotation as the rest, often for
>>>>> minutes at a time.  These are ea05, ea11, ea22, ea23, and ea25.
>>>>> These should be prioritized for FRM maintenance if possible.
>>>>>
>>>>>   - There are a few more "marginal" ones that show similar behavior
>>>>> but not quite as severe (ea09, ea10, ea12, ea13, ea15).
>>>>>
>>>>>   - All the "bad" and "marginal" antennas have old ACUs.
>>>>>
>>>>>   - Not all old-ACU antennas act badly, for example ea03 and ea04
>>>>> look generally pretty well-behaved.  But even these "good" examples
>>>>> spend typically ~50% more time flagged than new-ACU antennas.  So
>>>>> the new ACUs and associated mechanical overhaul are clearly an
>>>>> improvement (this is probably not news to many of you!).
>>>>>
>>>>> More details about this analysis:
>>>>>
>>>>> I gathered data on this from the SDMs currently available in the
>>>>> MCAF workspace.  Right now this goes back to the beginning of the 
>>>>> year.
>>>>> To avoid confusion from test/maint time, I only counted real science
>>>>> observations, identified as those datasets that start with '1' or
>>>>> 'V'.
>>>>>
>>>>> For each day (MJD) I add up all the time each antenna is listed as
>>>>> being in the SUBREFLECTOR_ERROR state in Flag.xml.  This only counts
>>>>> rotation errors (I haven't looked at focus but could in the future).
>>>>> Since there will be different numbers of band changes each day, I
>>>>> then divide all the times by the median of the 10 best (least
>>>>> flagged) antennas for that day.
>>>>>
>>>>> For a second statistic, I also looked at the duration of each flag
>>>>> event.  For reference, a typical subreflector rotation for a band
>>>>> change should take somewhere between 5 and 25 seconds depending on
>>>>> which bands are in use; Rick took a close look at this recently, see
>>>>> his emails to this list in Nov 2018 titled "Band Change Times."  The
>>>>> assumption that band changes take ~20s is baked into our software in
>>>>> several places (OPT, observing scripts).  I counted up all the
>>>>> instances where an antenna was flagged for >30s or >120s, these will
>>>>> be potentially bad for observations.
>>>>>
>>>>> Both of these metrics are plotted versus antenna number for a week's
>>>>> worth of data at a time (starting on Wednesday evenings). The
>>>>> rotation time plot has one point per antenna per day for a week.
>>>>> The flag duration counts are cumulative for the whole week.  See
>>>>> attached png showing the most recent week, and pdf showing all 
>>>>> available data.
>>>>>
>>>>> This analysis has an implicit assumption that all antennas are
>>>>> getting commanded to do the same thing.  This will occasionally not
>>>>> be true, for example if an antenna is removed from observing for
>>>>> part of a day for some reason.  So isolated data points away from
>>>>> 1.0, or small non-zero numbers of long-duration flags can probably
>>>>> be ignored.  But long-term patterns where certain antennas have
>>>>> consistently high/scattered points or many long-duration flags are
>>>>> meaningful, for example the "bad" ones I mentioned above.
>>>>>
>>>>> The other situation that may confuse this analysis somewhat is
>>>>> subarray observations.  To help avoid this, I've excluded all
>>>>> datasets that used less than 24 antennas.  There may be some
>>>>> residual effect on the first full-array observation following a
>>>>> subarray project since the antennas will have different starting
>>>>> subreflector positions.  These have not been removed since they are
>>>>> more difficult to automatically identify.  But I think this happens
>>>>> infrequently enough that it's not a big problem.
>>>>>
>>>>> Please let me know if you have comments, suggestions, or questions
>>>>> about any of this.
>>>>>
>>>>> Cheers,
>>>>> Paul
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> evlatests mailing list
>>>>> evlatests at listmgr.nrao.edu
>>>>> https://listmgr.nrao.edu/mailman/listinfo/evlatests
>>>>
>>>>
>>>> _______________________________________________
>>>> evlatests mailing list
>>>> evlatests at listmgr.nrao.edu
>>>> https://listmgr.nrao.edu/mailman/listinfo/evlatests
>>>
>>> _______________________________________________
>>> evlatests mailing list
>>> evlatests at listmgr.nrao.edu
>>> https://listmgr.nrao.edu/mailman/listinfo/evlatests
>>
>> _______________________________________________
>> evlatests mailing list
>> evlatests at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/evlatests
>>
> 
> 
> _______________________________________________
> evlatests mailing list
> evlatests at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/evlatests

More information about the evlatests mailing list