[evlatests] New and bizarre DTS behavior

[Rich Moeser]

It was the on-duty operator that sent the power reset commands to the 
DTS modules. He sent the reset commands attempting to correct total 
power and parity error issues that he saw on the antenna initialization 
screen. Rob said he will contact the operators and instruct them on how 
to resolve this issue in the future.

--Rich

Mike Revnell wrote:
> The following is a rather lengthy description to justify the request I 
> make in the last two paragraphs.
>
> We are observing a new and bizarre behavior in the DTS modules.
>
> This just started happeing about 3 weeks ago. Some of the modules it has 
> happend to have been working in antennas for a couple years. We have 
> made no changes to the affected parts of the modules for a long time so 
> I have difficulty believing it is bad hardware.
>
> At seemingly random times seemingly random groups of DTS modules go 
> off-line. The state we find them in is consistent with them having 
> received a "PSCReset" command or an explicit command to turn off the 
> formatter. The formatter board is powered down but the digitizer is 
> still under power. Power to these two boards is controlled 
> independently. If module power is interrputed all module components are 
> turned off. The logic that controls digitizer power runs off the same 
> regulator as logic that controls formatter power.
>
> The PSCReset command causes the fpga that controls formatter power to be 
> reconfigured. Thus in this event the formatter gets powered down but the 
> digitizer, if it is on, will not.
>
> Last night 6 of them went down in a 10 minute period. Here is the timing 
> of events to 1 minute resolution. These times come from timestamps on 
> email messages.
>
> 10:09 PM 16 A
> 10:11 PM 21 A
> 10:12 PM 26 A
> 10:14 PM 16 C
> 10:16 PM 16 D
> 10:19 PM 23 A
>
> As far as we are able to tell, when more than one DTS goes off-line they 
> do so in groups with time resolutions similar to the above. This timing 
> is consistent with some person doing something.
>
> Given the state we find the modules in I have been able to think of only 
> a couple scenarios to explain the behavior.
>
> 1. Something or someone has started sending PSCReset commands to the 
> modules. I can think of no reason to do this because the circumstances 
> which normally require it happen only when a module loses then regains 
> its time code input. If something were automatically doing it I would 
> expect it to happen more frequently and see more links go off line. I 
> would also expect all links to go off-line within the same minute. Since 
> doing this would have required writing some new piece of software, at 
> the expense of writing something more useful, I don't think this is the 
> case.
>
> 2. Some change in a seemingly unrelated system or procedure has started 
> expoliting a hitherto unused bug in the DTS MIB code or FPGA code. This 
> could be, for example, a command to the DTS module that for some reason 
> through a sneak logic connection causes the flip-flop that controls the 
> formatter power to be reset. Another possibility is that some command to 
> the MIB is causing it to assert its chip-select line that causes the 
> FPGA to reconfigure.
>
> Again this is behavior observed, for the first time about 3 weeks ago, 
> in modules that have had no changes. Some of them have been running, 
> without problem, in antennas for over 2 years.
>
> All that to justify this.
>
> Please review any changes to software and/or procedures which have been 
> instituted in the last few weeks. It seems to me that we may be 
> exercising a hitherto unused bug somewhere through a sneak path from an 
> unrelated change. In this case, I believe coincidence does matter.
>
> If anyone might have an idea of what they or someone else was doing with 
> the system around 10:09 last night it could provide a helpful clue.
>
> Thanks.
>
> Mike Revnell
>
> _______________________________________________
> evlatests mailing list
> evlatests at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/evlatests
>