[evlatests] [Fwd: Problems with EVLA 14 and 16]
Rich Moeser
rmoeser at nrao.edu
Fri Oct 28 11:19:22 EDT 2005
I can try. (times below are local and mjd)
1:11am: script /home/mchost/evla/scripts/astronomy/2005/10/390ptst-000.evla
submitted to the executor, ea14 and ea16.
1:18am: request to abort job
logged message 'starting source 0319+415 at 1.5GHz' 5 times.
53671.71800 script aborted by executor (normal exit messages)
1:19am: script: /home/mchost/evla/scripts/operations/390ptst.script
submitted to the executor, ea14 and ea16.
1:22am: request to abort job
logged message 'starting source 0319+415 at 1.5GHz' six times.
53671.72200 script aborted by executor (normal exit messages)
1:23am: script /home/mchost/evla/scripts/astronomy/2005/10/390ptst-001.evla
submitted to the executor, ea14 and ea16.
1:29am: request to abort job
logged message 'starting source 0319+415 at 1.5GHz' six times.
53671.72900 script aborted by executor (normal exit messages)
1:42am: script /home/mchost/evla/scripts/astronojmy/2005/10/390ptst-002.evla
submitted to the executor, ea14 and ea16.
1:44am: request to abort job
logged message 'starting source 0319+415 at 1.5GHz' six times.
53671.74400 script aborted by executor (normal exit messages)
1:49am: script /home/mchost/evla/scripts/astronomy/2005/10/390btst-000.evla
submitted to the executor, ea14 and ea16.
2:09am: request to abort job
53671 80900 script aborted by executor
(an exception was trapped here, but I think it tells us the thread was
interrupted. Barry would might be able to provide more details.)
(the exception text was followed by normal exit messages)
2:13am: script/home/mchost/evla/scripts/astronomy/2005/10/390btst-001.evla
submitted to the executor, ea14 and ea16.
(appeared script was running normally, message 'starting source 0319+415 at
22GHz' was logged 110 times)
53671.114400 normal exit messages from the executor
05:44am script done, moved to history, and marked as 'normal completion'
Aside from the exception that occurred at the fifth attempt everything
looked pretty reasonable. When I looked at the ACU screens this morning the
commanded positions looked a bit out of whack. On ea14, az=-350, el=(not
sure and I already changed it), and the observed el was jumping between 92
and 95 degrees. On ea16, az=-499 and el=141.
I'm going to run a test script to see how the executor responds.
--Rich
>
> rich,
>
> can you provide a reader's digest condensed version of this - including
> what you think might have happened last night?
>
> -bryan
>
>
> On 10/28/05 08:04, Rich Moeser wrote:
>> For those interested, here are the executor's logs from last night...
>>
>> ExecutorLog::publish:01:11:30 10.28::Executor->Add job :'1130483490813';
>> script: /home/mchost/evla/scripts/operations/390ptst.OBS
>> getNextInSequence: path =
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst.evla
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst-000.evla
>> Run obs2script...
>> /home/mchost/evla/bin/obs2script -i
>> /home/mchost/evla/scripts/operations/390ptst.OBS -o
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst-000.evla
>> ExecutorLog::publish:01:11:30 10.28::Executor->Add Antenna 'ea14' to job
>> '1130483490813'.
>> ExecutorLog::publish:01:11:30 10.28::Executor->Add Antenna 'ea16' to job
>> '1130483490813'.
>> ExecutorLog::publish:01:11:34 10.28::Executor->Starting job
>> '1130483490813'.
>> ExecutorLog::publish:01:11:34 10.28::Executor->Job started...
>> starting source 0319+415 at 1.5GHz
>> abort the job
>> ExecutorLog::publish:01:18:29 10.28::Executor->Aborting job 1130483490813
>> Subarray is shutting down
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> 1130483490813 53671 71800 End file
>> ExecutorLog::publish:1130483490813 53671 71800 End file
>> 1130483490813 53671 71800 Exit Array
>> Datagram receive exception
>> Socket closed
>> ExecutorLog::publish:1130483490813 53671 71800 Exit Array
>> Systole halting.
>> ExecutorLog::publish:01:18:30 10.28::Executor->Move job '1130483490813'
>> to history.
>> ExecutorLog::publish:01:19:09 10.28::Executor->Add job :'1130483949000';
>> script: /home/mchost/evla/scripts/operations/390ptst.script
>> ExecutorLog::publish:01:19:09 10.28::Executor->Add Antenna 'ea14' to job
>> '1130483949000'.
>> ExecutorLog::publish:01:19:09 10.28::Executor->Add Antenna 'ea16' to job
>> '1130483949000'.
>> ExecutorLog::publish:01:19:10 10.28::Executor->Starting job
>> '1130483949000'.
>> ExecutorLog::publish:01:19:10 10.28::Executor->Job started...
>> starting source 0319+415 at 1.5GHz
>> abort the job
>> ExecutorLog::publish:01:22:40 10.28::Executor->Aborting job 1130483949000
>> Subarray is shutting down
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> 1130483949000 53671 72200 End file
>> ExecutorLog::publish:1130483949000 53671 72200 End file
>> 1130483949000 53671 72200 Exit Array
>> Datagram receive exception
>> Socket closed
>> ExecutorLog::publish:1130483949000 53671 72200 Exit Array
>> ExecutorLog::publish:01:22:40 10.28::Executor->Move job '1130483949000'
>> to history.
>> Systole halting.
>> ExecutorLog::publish:01:23:30 10.28::Executor->Add job :'1130484210500';
>> script: /home/mchost/evla/scripts/operations/390ptst.OBS
>> getNextInSequence: path =
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst.evla
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst-000.evla
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst-001.evla
>> Run obs2script...
>> /home/mchost/evla/bin/obs2script -i
>> /home/mchost/evla/scripts/operations/390ptst.OBS -o
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst-001.evla
>> ExecutorLog::publish:01:23:30 10.28::Executor->Add Antenna 'ea14' to job
>> '1130484210500'.
>> ExecutorLog::publish:01:23:30 10.28::Executor->Add Antenna 'ea16' to job
>> '1130484210500'.
>> ExecutorLog::publish:01:23:30 10.28::Executor->Starting job
>> '1130484210500'.
>> ExecutorLog::publish:01:23:30 10.28::Executor->Job started...
>> starting source 0319+415 at 1.5GHz
>> abort the job
>> ExecutorLog::publish:01:29:28 10.28::Executor->Aborting job 1130484210500
>> Subarray is shutting down
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> 1130484210500 53671 72900 End file
>> ExecutorLog::publish:1130484210500 53671 72900 End file
>> 1130484210500 53671 72900 Exit Array
>> ExecutorLog::publish:1130484210500 53671 72900 Exit Array
>> Datagram receive exception
>> Socket closed
>> Systole halting.
>> ExecutorLog::publish:01:29:31 10.28::Executor->Move job '1130484210500'
>> to history.
>> ExecutorLog::publish:01:42:10 10.28::Executor->Add job :'1130485330792';
>> script: /home/mchost/evla/scripts/operations/390ptst.OBS
>> getNextInSequence: path =
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst.evla
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst-000.evla
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst-001.evla
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst-002.evla
>> Run obs2script...
>> /home/mchost/evla/bin/obs2script -i
>> /home/mchost/evla/scripts/operations/390ptst.OBS -o
>> /home/mchost/evla/scripts/astronomy/2005/10/390ptst-002.evla
>> ExecutorLog::publish:01:42:10 10.28::Executor->Add Antenna 'ea14' to job
>> '1130485330792'.
>> ExecutorLog::publish:01:42:10 10.28::Executor->Add Antenna 'ea16' to job
>> '1130485330792'.
>> ExecutorLog::publish:01:42:12 10.28::Executor->Starting job
>> '1130485330792'.
>> ExecutorLog::publish:01:42:12 10.28::Executor->Job started...
>> starting source 0319+415 at 1.5GHz
>> abort the job
>> ExecutorLog::publish:01:44:26 10.28::Executor->Aborting job 1130485330792
>> Subarray is shutting down
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> starting source 0319+415 at 1.5GHz
>> 1130485330792 53671 74400 End file
>> ExecutorLog::publish:1130485330792 53671 74400 End file
>> 1130485330792 53671 74400 Exit Array
>> ExecutorLog::publish:1130485330792 53671 74400 Exit Array
>> Datagram receive exception
>> Socket closed
>> Systole halting.
>> ExecutorLog::publish:01:44:28 10.28::Executor->Move job '1130485330792'
>> to history.
>> ExecutorLog::publish:01:49:38 10.28::Executor->Add job :'1130485778513';
>> script: /home/mchost/evla/scripts/operations/390btst.OBS
>> getNextInSequence: path =
>> /home/mchost/evla/scripts/astronomy/2005/10/390btst.evla
>> /home/mchost/evla/scripts/astronomy/2005/10/390btst-000.evla
>> Run obs2script...
>> /home/mchost/evla/bin/obs2script -i
>> /home/mchost/evla/scripts/operations/390btst.OBS -o
>> /home/mchost/evla/scripts/astronomy/2005/10/390btst-000.evla
>> ExecutorLog::publish:01:49:38 10.28::Executor->Add Antenna 'ea14' to job
>> '1130485778513'.
>> ExecutorLog::publish:01:49:38 10.28::Executor->Add Antenna 'ea16' to job
>> '1130485778513'.
>> ExecutorLog::publish:01:49:38 10.28::Executor->Starting job
>> '1130485778513'.
>> ExecutorLog::publish:01:49:38 10.28::Executor->Job started...
>> starting source 0137+331 at 6GHz
>> abort the job
>> ExecutorLog::publish:02:09:13 10.28::Executor->Aborting job 1130485778513
>> 1130485778513 53671 80900 Java exception at python line 74
>> ExecutorLog::publish:1130485778513 53671 80900 Java exception at python
>> line 74
>> 1130485778513 53671 80900 sleep interrupted
>> ExecutorLog::publish:1130485778513 53671 80900 sleep interrupted
>> 1130485778513 53671 80900 Class java.lang.Thread in File Thread.java
>> at -2
>> ExecutorLog::publish:1130485778513 53671 80900 Class java.lang.Thread in
>> File Thread.java at -2
>> 1130485778513 53671 80900 Class edu.nrao.evla.observe.Array in File
>> Array.java at 224
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> edu.nrao.evla.observe.Array in File Array.java at 224
>> 1130485778513 53671 80900 Class sun.reflect.NativeMethodAccessorImpl in
>> File NativeMethodAccessorImpl.java at -2
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> sun.reflect.NativeMethodAccessorImpl in File
>> NativeMethodAccessorImpl.java at -2
>> 1130485778513 53671 80900 Class sun.reflect.NativeMethodAccessorImpl in
>> File NativeMethodAccessorImpl.java at 39
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> sun.reflect.NativeMethodAccessorImpl in File
>> NativeMethodAccessorImpl.java at 39
>> 1130485778513 53671 80900 Class sun.reflect.DelegatingMethodAccessorImpl
>> in File DelegatingMethodAccessorImpl.java at 25
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> sun.reflect.DelegatingMethodAccessorImpl in File
>> DelegatingMethodAccessorImpl.java at 25
>> 1130485778513 53671 80900 Class java.lang.reflect.Method in File
>> Method.java at 585
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> java.lang.reflect.Method in File Method.java at 585
>> 1130485778513 53671 80900 Class org.python.core.PyReflectedFunction in
>> File PyReflectedFunction.java at -1
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> org.python.core.PyReflectedFunction in File PyReflectedFunction.java
>> at -1
>> 1130485778513 53671 80900 Class org.python.core.PyMethod in File
>> PyMethod.java at -1
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> org.python.core.PyMethod in File PyMethod.java at -1
>> 1130485778513 53671 80900 Class org.python.core.PyObject in File
>> PyObject.java at 270
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> org.python.core.PyObject in File PyObject.java at 270
>> 1130485778513 53671 80900 Class org.python.core.PyInstance in File
>> PyInstance.java at -1
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> org.python.core.PyInstance in File PyInstance.java at -1
>> 1130485778513 53671 80900 Class org.python.pycode._pyx991 in File
>> <string> at 1
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> org.python.pycode._pyx991 in File <string> at 1
>> 1130485778513 53671 80900 Class org.python.pycode._pyx991 in File
>> <string> at -1
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> org.python.pycode._pyx991 in File <string> at -1
>> 1130485778513 53671 80900 Class org.python.core.PyTableCode in File
>> PyTableCode.java at -1
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> org.python.core.PyTableCode in File PyTableCode.java at -1
>> 1130485778513 53671 80900 Class org.python.core.PyCode in File
>> PyCode.java at -1
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> org.python.core.PyCode in File PyCode.java at -1
>> 1130485778513 53671 80900 Class org.python.core.Py in File Py.java at -1
>> ExecutorLog::publish:1130485778513 53671 80900 Class org.python.core.Py
>> in File Py.java at -1
>> 1130485778513 53671 80900 Class org.python.core.Py in File Py.java at -1
>> ExecutorLog::publish:1130485778513 53671 80900 Class org.python.core.Py
>> in File Py.java at -1
>> 1130485778513 53671 80900 Class org.python.util.PythonInterpreter in File
>> PythonInterpreter.java at -1
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> org.python.util.PythonInterpreter in File PythonInterpreter.java at -1
>> 1130485778513 53671 80900 Class edu.nrao.evla.observe.Array in File
>> Array.java at 420
>> ExecutorLog::publish:1130485778513 53671 80900 Class
>> edu.nrao.evla.observe.Array in File Array.java at 420
>> 1130485778513 53671 80900 Ending execution
>> ExecutorLog::publish:1130485778513 53671 80900 Ending execution
>> 1130485778513 53671 80900 Exit Array
>> ExecutorLog::publish:1130485778513 53671 80900 Exit Array
>> Datagram receive exception
>> Socket closed
>> Systole halting.
>> ExecutorLog::publish:02:09:16 10.28::Executor->Move job '1130485778513'
>> to history.
>> ExecutorLog::publish:02:13:55 10.28::Executor->Add job :'1130487235812';
>> script: /home/mchost/evla/scripts/operations/390btst.OBS
>> getNextInSequence: path =
>> /home/mchost/evla/scripts/astronomy/2005/10/390btst.evla
>> /home/mchost/evla/scripts/astronomy/2005/10/390btst-000.evla
>> /home/mchost/evla/scripts/astronomy/2005/10/390btst-001.evla
>> Run obs2script...
>> /home/mchost/evla/bin/obs2script -i
>> /home/mchost/evla/scripts/operations/390btst.OBS -o
>> /home/mchost/evla/scripts/astronomy/2005/10/390btst-001.evla
>> ExecutorLog::publish:02:13:55 10.28::Executor->Add Antenna 'ea14' to job
>> '1130487235812'.
>> ExecutorLog::publish:02:13:55 10.28::Executor->Add Antenna 'ea16' to job
>> '1130487235812'.
>> ExecutorLog::publish:02:13:56 10.28::Executor->Starting job
>> '1130487235812'.
>> ExecutorLog::publish:02:13:56 10.28::Executor->Job started...
>> starting source 0137+331 at 6GHz
>> starting source 0137+331 at 10GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> starting source 0319+415 at 22GHz
>> 1130487235812 53671 114400 End file
>> ExecutorLog::publish:1130487235812 53671 114400 End file
>> 1130487235812 53671 114400 Exit Array
>> ExecutorLog::publish:1130487235812 53671 114400 Exit Array
>> Datagram receive exception
>> Socket closed
>> Systole halting.
>> ExecutorLog::publish:05:44:12 10.28::Executor->Move job '1130487235812'
>> to history.
>> _______________________________________________
>> evlatests mailing list
>> evlatests at listmgr.cv.nrao.edu
>> http://listmgr.cv.nrao.edu/mailman/listinfo/evlatests
More information about the evlatests
mailing list