[evlatests] Multiple System Failures
Rick Perley
rperley at nrao.edu
Mon Jan 2 15:27:42 EST 2012
I ran another test last night to tweak the timing for the upcoming
flux densities run. The results from this latest test display three
different failure modes, all of them new to me.
The test comprised observations of six sources, in order:
3C84, 3C147, 3C123, 3C138, Mars, and 3C273. There was an unwrap
between 3C123 and 3C138, requiring an azimuth move of 330 degrees. This
is important to remember.
For each source, the same sequence of band observations was taken:
L - XRefPnt - X - C - S - Ku - K - Q - Ka.
For the first three, an 'on source' duration was requested of the
OPT. (This is needed for the XRefPnt and X-band observations, since the
reference pointing object may, or may not, be the target source). For
the final six bands, I used a 'durations' request to the OPT. For all
the on source observations, the requested time is sufficient to provide
15 to 20 seconds of good on-source observations. Because of these rapid
changes, each source is observed for a total of less than 7 minutes.
There were three distinct failures noted in this test:
1) Nine scans were missing from the data. The operator's log stated
that 10 BDF files were missing. What is especially interesting is that
all of the missing scans were at Ku, K, Q, and Ka bands. 3C84 missed
all four of these, 3C147 missed all four, and 3C138 missed Ku band
only. In all cases, the time needed to observe the sources was present
-- i.e., in the data files, there are gaps within which the data should
have appeared.
The same failure occurred on my previous test, taken 3 days ago. I
utilized the same sequence of observations, but with different
durations. In this earlier case, there were four missing scans, all on
3C138, and once again, it was the four high frequency bands which went
missing.
2) Referenced pointing failed twice -- on 3C138 and on J1118+1234 --
the source I use for Mars. The first failure is easy to explain -- the
antennas were busy unwrapping, so of course we were not on the source.
(A added curiosity here however -- bdf2aips filled the pointing data
even though we weren't on source. For the four successful pointing
determinations, bdf2aips did not fill the data. Apparently, it is
sensitive to whether the antennas are actually on source or not.)
The second failure is more worrisome. The Mars observation also
required a modest slew -- about three minutes from 3C138 (which, it
should be noted, the antennas never reached before the allotted time was
up). Yet I am confident we reached Mars before the pointing solution
started, since I have 65 seconds of unflagged data on Mars at L-band.
(Note, however, that at L-band, there is not enough flux to actually
know if we're on the source -- I have to trust the antenna flags to
justify the statement made above). Now, the time spent on Mars at
L-band was far too long -- I requested 15 seconds of 'on source', but
got more than 65. There was then a gap of the appropriate length for an
X-band pointing determination, after which the normal sequence of
observations began again.
3) By far the most bizarre failure -- and a unique one to me --
were the pattern of 'zero records' which afflicted the observations of
3C123, 3C138, Mars, and 3C273:
I note first that all observations of 3C84 and 3C147 were complete
in the sense that all antennas for all subbands were fully and equally
filled for those scans which were not missed (see #1, above).
However, for the remaining observations (of 3C123, 3C138, Mars, and
3C273), large fractions of the data were 'pure zero' -- both the
visibilities and the weights are zero. The zero records always last the
entire scan, and are in three patterns:
Pattern A:
Antenna 1 is zeroed with all higher numbered antennas.
Antenna 2 is zeroed with all higher numbered antennas except 3 and 4
Antenna 5 is zeroed with all higher numbered antennas.
Antenna 6 is zeroed with all higher numbered antennas except 7 and 8.
Antenna 9 is zeroed with all higher numbered antennas
Antenna 10 is zeroed with all higher numbered antennas except 11 and 12
etc. This alternating pattern continues up through antenna 28.
Pattern B:
Antennas 3, 4, 7, 8, 11, 12, 15 16, 20 21, 24, 25 27 and 28 are
zeroed out with all higher numbered antennas.
Pattern C:
All antennas zeroed against all others.
But the really odd thing is that these zero patterns affect the
different subbands in different ways.
One of these patterns afflicted all observations at all bands of
3C123, 3C138, Mars, and 3C273. However ... which pattern occurred
depends on the subband, in a very curious way. I made a chart for the
C-band observations -- so far as I can tell, a similar, or perhaps even
identical, chart applies to each band:
subband 3C123 3C138 Mars 3C273
----------------------------------------------------------------------------------
1 OK B
OK OK
2 A OK
B A
3 A B
B A
4 OK OK
OK OK
5 OK OK
OK OK
6 A A
B OK
7 A B
A OK
8 OK B
OK OK
9 C OK
B OK
10 A B
B A
11 A B
B A
12 OK OK
A OK
13 OK A
OK C
14 C OK
B A
15 OK B
OK A
16 OK B
A OK
------------------------------------------------------------------------------------------
This has got to be a correlator problem. What happened?
More information about the evlatests
mailing list