[evla-sw-discuss] Power loss at VLA

[Kevin Ryan]

I rebooted cpcc1 and it came up well and started monitoring things.  Unfortunately, it  thinks there is a major (Stage 3 - 2nd detector) smoke alarm and the operator's CPCC GUI began bonging.  This would happen if there is no power to the RPMIB in rack S002 which is where the smoke alarm system is wired in for CPCC monitoring.  I called Bob and he said the WIDAR power plant was up and running so there may be something wrong like a circuit breaker in the rack itself tripped.  

I stopped the server app in cpcc1 until we can figure out the power situation at each of the racks and figure out what is going on with cpcc2.  Until then the correlator will be non operational.


Kevin





On Dec 23, 2011, at 8:56 PM, Kevin Ryan wrote:

> With regard to 'cpcc2', is that the one on the inverter from the WIDAR Power Plant?  Is the power plant back online?
> 
> Kev
> 
> On Dec 23, 2011, at 8:45 PM, Kevin Ryan wrote:
> 
>> I think that cpcc1 is also not quite right.  I cannot log in as 'kryan' (I can log in as 'rootb') and the server app did not start.  So it is not functional at the moment.
>> 
>> Kevin
>> 
>> 
>> On Dec 23, 2011, at 7:45 PM, James Robnett wrote:
>> 
>>> 
>>> Things actually came up pretty well.  Most everything is up except
>>> for cpcc2.  It didn't come up either time.  A bit concerned it has a
>>> hardware problem.
>>> 
>>> There's a power problem on the west arm so apparently the plan is
>>> to not worry about observing for now.   I'll let Ken fill in details.
>>> 
>>> James
>>> 
>>>> Power came back on around 5:00 pm or so and most everything came up.
>>>> The CPCCs didn't and I was trying to work through them when we lost
>>>> power again.
>>>> 
>>>> This time there was virtually no UPS battery power left so everything
>>>> just lost power.   It's still down best I can tell and I suspect things
>>>> won't come up nearly so cleanly this time.
>>>> 
>>>> James
>>>> 
>>>>> VLA has lost power and the generators didn't come on.  Tom Briscoe has
>>>>> shut
>>>>> down the correlator but the UPS is starting to complain about low power.
>>>>> 
>>>>> I'm going to shut down the servers in the correlator room and the lustre
>>>>> servers so they have some hope of booting cleanly.
>>>>> 
>>>>> Rumor has it US60 is closed (believable) but Bob Broilo is reportedly
>>>>> going to try and get out there and see what's up.
>>>>> 
>>>>> James
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> evla-sw-discuss mailing list
>>>> evla-sw-discuss at listmgr.cv.nrao.edu
>>>> http://listmgr.cv.nrao.edu/mailman/listinfo/evla-sw-discuss
>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> evla-sw-discuss mailing list
>>> evla-sw-discuss at listmgr.cv.nrao.edu
>>> http://listmgr.cv.nrao.edu/mailman/listinfo/evla-sw-discuss
>> 
>> 
>> _______________________________________________
>> evla-sw-discuss mailing list
>> evla-sw-discuss at listmgr.cv.nrao.edu
>> http://listmgr.cv.nrao.edu/mailman/listinfo/evla-sw-discuss
> 
> 
> _______________________________________________
> evla-sw-discuss mailing list
> evla-sw-discuss at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/evla-sw-discuss