<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>We took a look at the bad antennas (ea05, ea11, ea22, ea23, and
ea25). 5 and 23 Rotation showed the typical signs of worn gears.
11, 22 and 25 performed well (consistently reached desired
position within 0.1 degrees and no drive faults). The test was a
52 position script that includes large and small moves and covers
the full range of motion.</p>
<p>Doug<br>
</p>
<div class="moz-cite-prefix">On 4/9/2019 9:19 AM, Gene Cole wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8759ad3f702445b589e3103501eadad0@nrao.edu">
<pre class="moz-quote-pre" wrap="">Paul,
With regard to rotation, we suspect worn gears to be the problem. Many of these antennas use aluminum gears that wear down over time. As with the new ACU upgrade, steel gears are used as replacements.
Doug Whiton will conduct FRM repeatability tests on all the antennas in question. After reviewing this data, the mechanics can start servicing the antennas in the array. We need to order new gears, so the lead-time of this will drive any progress toward this. Ea9 and ea22 are due for ACU upgrades this year, so we will wait until then to address them.
--Gene
-----Original Message-----
From: evlatests <a class="moz-txt-link-rfc2396E" href="mailto:evlatests-bounces@listmgr.nrao.edu"><evlatests-bounces@listmgr.nrao.edu></a> On Behalf Of Paul Demorest
Sent: Wednesday, April 03, 2019 5:39 PM
To: Rick Perley <a class="moz-txt-link-rfc2396E" href="mailto:rperley@nrao.edu"><rperley@nrao.edu></a>
Cc: <a class="moz-txt-link-abbreviated" href="mailto:evlatests@listmgr.nrao.edu">evlatests@listmgr.nrao.edu</a>
Subject: Re: [evlatests] Subreflector rotation (and focus) statistics
So it was easy enough to run the same script and look at the focus flag rather than rotation. Attached is the same set of plots as previously, just for focus now.
There do some to be some consistently poor antennas here as well (ea03, ea05, ea13, ea26). These are also all old-ACU antennas, but annoyingly this set is almost completely disjoint from the rotation bad-list, aside from ea05. However, the scale of the problem overall seems less severe in focus, so I'd still say the "bad" list from my original email are the ones to pay attention to first.
-Paul
On 2019-04-03 14:32, Paul Demorest wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Rick,
Yes, this analysis looked exclusively at rotation flags (which
unfortunately have the generic label "SUBREFLECTOR_ERROR" in the SDM).
Focus flags are counted separately and are labeled "FOCUS_ERROR".
That's not to say there are no problems with focus, I just haven't
looked at it yet.
-Paul
On 2019-04-03 14:28, Rick Perley wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap=""> Paul, et al.:
Are you sure it is only *rotation*, as opposed to focus? In the
various test I do, both are involved. I'll soon have better
information, as I'm nearly ready to seriously reduce the 'flux
density' test data.
Rick
On 04/03/2019 02:15 PM, Paul Demorest wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">hi all,
This is a report on an analysis of VLA subreflector rotation times I
recently did (some of you will have seen a version of this already;
there is a little new info in here but no change in basic
conclusions). This was motivated by recent anecdotal reports from
operators and analysts about specific antennas often being flagged
due to subreflector rotation for much longer than expected,
sometimes resulting in their missing calibrators, etc. I thought it
would be useful to take a more systematic look at recent data for
problems like this.
First, the main conclusions are:
- There are several "bad" antennas that frequently spend >~10x the
time flagged due to subreflector rotation as the rest, often for
minutes at a time. These are ea05, ea11, ea22, ea23, and ea25.
These should be prioritized for FRM maintenance if possible.
- There are a few more "marginal" ones that show similar behavior
but not quite as severe (ea09, ea10, ea12, ea13, ea15).
- All the "bad" and "marginal" antennas have old ACUs.
- Not all old-ACU antennas act badly, for example ea03 and ea04
look generally pretty well-behaved. But even these "good" examples
spend typically ~50% more time flagged than new-ACU antennas. So
the new ACUs and associated mechanical overhaul are clearly an
improvement (this is probably not news to many of you!).
More details about this analysis:
I gathered data on this from the SDMs currently available in the
MCAF workspace. Right now this goes back to the beginning of the year.
To avoid confusion from test/maint time, I only counted real science
observations, identified as those datasets that start with '1' or
'V'.
For each day (MJD) I add up all the time each antenna is listed as
being in the SUBREFLECTOR_ERROR state in Flag.xml. This only counts
rotation errors (I haven't looked at focus but could in the future).
Since there will be different numbers of band changes each day, I
then divide all the times by the median of the 10 best (least
flagged) antennas for that day.
For a second statistic, I also looked at the duration of each flag
event. For reference, a typical subreflector rotation for a band
change should take somewhere between 5 and 25 seconds depending on
which bands are in use; Rick took a close look at this recently, see
his emails to this list in Nov 2018 titled "Band Change Times." The
assumption that band changes take ~20s is baked into our software in
several places (OPT, observing scripts). I counted up all the
instances where an antenna was flagged for >30s or >120s, these will
be potentially bad for observations.
Both of these metrics are plotted versus antenna number for a week's
worth of data at a time (starting on Wednesday evenings). The
rotation time plot has one point per antenna per day for a week.
The flag duration counts are cumulative for the whole week. See
attached png showing the most recent week, and pdf showing all available data.
This analysis has an implicit assumption that all antennas are
getting commanded to do the same thing. This will occasionally not
be true, for example if an antenna is removed from observing for
part of a day for some reason. So isolated data points away from
1.0, or small non-zero numbers of long-duration flags can probably
be ignored. But long-term patterns where certain antennas have
consistently high/scattered points or many long-duration flags are
meaningful, for example the "bad" ones I mentioned above.
The other situation that may confuse this analysis somewhat is
subarray observations. To help avoid this, I've excluded all
datasets that used less than 24 antennas. There may be some
residual effect on the first full-array observation following a
subarray project since the antennas will have different starting
subreflector positions. These have not been removed since they are
more difficult to automatically identify. But I think this happens
infrequently enough that it's not a big problem.
Please let me know if you have comments, suggestions, or questions
about any of this.
Cheers,
Paul
_______________________________________________
evlatests mailing list
<a class="moz-txt-link-abbreviated" href="mailto:evlatests@listmgr.nrao.edu">evlatests@listmgr.nrao.edu</a>
<a class="moz-txt-link-freetext" href="https://listmgr.nrao.edu/mailman/listinfo/evlatests">https://listmgr.nrao.edu/mailman/listinfo/evlatests</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
_______________________________________________
evlatests mailing list
<a class="moz-txt-link-abbreviated" href="mailto:evlatests@listmgr.nrao.edu">evlatests@listmgr.nrao.edu</a>
<a class="moz-txt-link-freetext" href="https://listmgr.nrao.edu/mailman/listinfo/evlatests">https://listmgr.nrao.edu/mailman/listinfo/evlatests</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
_______________________________________________
evlatests mailing list
<a class="moz-txt-link-abbreviated" href="mailto:evlatests@listmgr.nrao.edu">evlatests@listmgr.nrao.edu</a>
<a class="moz-txt-link-freetext" href="https://listmgr.nrao.edu/mailman/listinfo/evlatests">https://listmgr.nrao.edu/mailman/listinfo/evlatests</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
_______________________________________________
evlatests mailing list
<a class="moz-txt-link-abbreviated" href="mailto:evlatests@listmgr.nrao.edu">evlatests@listmgr.nrao.edu</a>
<a class="moz-txt-link-freetext" href="https://listmgr.nrao.edu/mailman/listinfo/evlatests">https://listmgr.nrao.edu/mailman/listinfo/evlatests</a>
</pre>
</blockquote>
<div class="moz-signature">-- <br>
<img src="cid:part1.B065FE45.4A9C9101@nrao.edu" border="0"></div>
</body>
</html>