<div dir="ltr">I have tee'ed the output from mpirun and errormon2 :<div><br></div><div><div>oper@ww-flexbuf-01 DiFX-2.4.3 hw04> cat mpirun.log</div><div>About to run MPIInit on node ww-flexbuf-01</div><div>About to run MPIInit on node ww-flexbuf-01</div><div>About to run MPIInit on node wark167</div><div>About to run MPIInit on node wark167</div><div>NOTE: difxmessage is in use. If you are not running errormon/errormon2, you are missing all the (potentially important) info messages!</div><div>[wark167 1] INFO MPI Process 1 is running on host wark167</div><div>[wark167 3] INFO MPI Process 3 is running on host wark167</div><div>[wark167 3] INFO About to process the input file..</div><div>[wark167 1] INFO About to process the input file..</div><div>[wark167 3] INFO DIFX VERSION = DiFX-2.4.3</div><div>[wark167 1] INFO DIFX VERSION = DiFX-2.4.3</div><div>[wark167 3] DEBUG NS accumulate is 125 and max geom slip is 39.9397, maxnsslip is 0</div><div>[wark167 3] DEBUG NS accumulate is 125 and max geom slip is 25.2648, maxnsslip is 0</div><div>[wark167 1] DEBUG NS accumulate is 125 and max geom slip is 39.9397, maxnsslip is 0</div><div>[wark167 1] DEBUG NS accumulate is 125 and max geom slip is 25.2648, maxnsslip is 0</div><div>[wark167 3] INFO Receive socket opened; socket is -1</div><div>[wark167 3] WARNING Could not open command monitoring socket! Aborting message receive thread.</div><div>[wark167 1] INFO Receive socket opened; socket is -1</div><div>[wark167 1] WARNING Could not open command monitoring socket! Aborting message receive thread.</div></div><div><br></div><div><div>oper@ww-flexbuf-01 DiFX-2.4.3 hw04> cat /tmp/errormon2.log</div><div>2016-04-22 14:56:26,955 DiFXAlert INFO MPI[ 2] ww-flexbuf-01 hw04_1 MPI Process 2 is running on host ww-flexbuf-01</div><div>2016-04-22 14:56:26,955 DiFXAlert INFO MPI[ 0] ww-flexbuf-01 hw04_1 MPI Process 0 is running on host ww-flexbuf-01</div><div>2016-04-22 14:56:26,956 DiFXAlert INFO MPI[ 2] ww-flexbuf-01 hw04_1 DIFX VERSION = DiFX-2.4.3</div><div>2016-04-22 14:56:26,956 DiFXAlert INFO MPI[ 0] ww-flexbuf-01 hw04_1 DIFX VERSION = DiFX-2.4.3</div><div>2016-04-22 14:56:26,957 DiFXAlert INFO MPI[ 0] ww-flexbuf-01 hw04_1 Receive socket opened; socket is 12</div><div>2016-04-22 14:56:26,957 DiFXAlert INFO MPI[ 2] ww-flexbuf-01 hw04_1 Receive socket opened; socket is 12</div></div><div><br></div><div><div>oper@wark167:/etc/network# cat /tmp/errormon2.log</div><div>2016-04-22 14:57:56,932 DiFXAlert INFO MPI[ 2] ww-flexbuf-01 hw04_1 MPI Process 2 is running on host ww-flexbuf-01</div><div>2016-04-22 14:57:56,933 DiFXAlert INFO MPI[ 0] ww-flexbuf-01 hw04_1 MPI Process 0 is running on host ww-flexbuf-01</div><div>2016-04-22 14:57:56,933 DiFXAlert INFO MPI[ 2] ww-flexbuf-01 hw04_1 DIFX VERSION = DiFX-2.4.3</div><div>2016-04-22 14:57:56,934 DiFXAlert INFO MPI[ 0] ww-flexbuf-01 hw04_1 DIFX VERSION = DiFX-2.4.3</div><div>2016-04-22 14:57:56,941 DiFXAlert INFO MPI[ 2] ww-flexbuf-01 hw04_1 Receive socket opened; socket is 12</div><div>2016-04-22 14:57:56,941 DiFXAlert INFO MPI[ 0] ww-flexbuf-01 hw04_1 Receive socket opened; socket is 12</div></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 22, 2016 at 1:25 PM, <span dir="ltr"><<a href="mailto:Chris.Phillips@csiro.au" target="_blank">Chris.Phillips@csiro.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
Hi Stuart
<div><br>
</div>
<div>I assume that actually the message was:</div>
<div><br>
</div>
<div>"<span style="font-family:Menlo;font-size:11px">Could not open command monitoring socket! Aborting message receive thread.</span><font face="Menlo"><span style="font-size:11px">”</span></font></div>
<div><span style="font-family:Menlo;font-size:11px"><br>
</span></div>
<div>You really need to send the full output for us to have any chance of diagnosing this. When you say “nothing more” in errormon2, does ANYTHING appear there?</div>
<div><br>
</div>
<div>The message receive thread in most circumstances is not important. If DIFX messages is not working you will however not get any logging messages.</div>
<div><br>
</div>
<div>Which processes give this message and on which machines are they running? </div>
<div><br>
</div>
<div>difxmessage library does not report error unfortunately, just return if there were errors. </div>
<div><br>
</div>
<div>I would suggest making a temp change to <span style="font-family:Menlo;font-size:11px">difxmessage/multicast.c </span>and recompiling it and mpifxcorr</div>
<div><br>
</div>
<div>Add some calls to perror before all the error returns in the routine <span style="font-family:Menlo;font-size:11px">openMultiCastSocket</span>. E.g.</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<br>
</div>
</div>
<div><span>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> /* Make UDP
</span><span style="color:#ffffff;background-color:#000000">socket</span><span> */</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> sock =
</span><span style="color:#ffffff;background-color:#000000">socket</span><span>(AF_INET, SOCK_DGRAM, 0);</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> if(sock < 0) </span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> {</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> <span style="white-space:pre-wrap">
</span>perror(“Trying to create socket: ”);</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span><span style="white-space:pre-wrap"></span> return -1;</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> }</span></div>
<div><span><br>
</span></div>
</span></div>
<div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> <span style="white-space:pre-wrap">
</span>/* Allow reuse of port */</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> v = setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes));</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> if(v < 0) </span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> {</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span><span style="white-space:pre-wrap"></span>perror(“Setsockopt: “);</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> return -2;</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> }</span></div>
</div>
<div><span><br>
</span></div>
<div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> <span style="white-space:pre-wrap">
</span>/* bind to receive address */</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> v = bind(sock, (struct sockaddr *)&addr, sizeof(struct sockaddr_in));</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> if(v < 0) </span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> {</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span><span style="white-space:pre-wrap"></span>perror(“Binding to socket: ");</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> return -3;</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> }</span></div>
</div>
<div><span style="font-family:Menlo;font-size:11px"> </span></div>
<div><span>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> v = inet_aton(group, &mreq.imr_multiaddr);</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> if(!v) </span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> {</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span><span style="white-space:pre-wrap"></span>perror(“inet_aton: );</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> return -4;</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> }</span></div>
<div><span><br>
</span></div>
</span></div>
<div>I am pretty sure the problem is not the choice of multicast address - if cannot connect to multicast group the code should give a major warning.</div>
<div><br>
</div>
<div>Just to double check - do you see the following message:</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo">
<span> </span><span style="color:#ffffff;background-color:#000000">Unicast</span><span>
(XXXXX) difxMessage in use. Some functionallity may be reduced</span></div>
</div>
<div><br>
</div>
<div>If you do, thats the problem</div>
<div><br>
</div>
<div>Cheers</div>
<div>Chris</div>
<div><br>
<div>
<div>
<div>
<blockquote type="cite"><div><div class="h5">
<div>On 22 Apr 2016, at 11:02 AM, Stuart Weston <<a href="mailto:nzobservers@gmail.com" target="_blank">nzobservers@gmail.com</a>> wrote:</div>
<br>
</div></div><div><div><div class="h5">
<div dir="ltr">
<p class="MsoNormal">I have two servers, they both have 2 x CPU ( 6 cores, hyperthreaded). So potentially I have 24 cores and 48 threads.</p>
<div> <br>
</div>
<p class="MsoNormal">mpirun starts mpifxcorr on both servers, but we get the “WARNING Could not open monitoring socket ! Aborting message receive thread” on the master ? The processes seem to sit there and do nothing, nothing more in errmon2.</p>
<div> <br>
</div>
<p class="MsoNormal">If I change the machines file I can run the same correlation on each server individually to completion, so DiFX has to be good.</p>
<div> <br>
</div>
<p class="MsoNormal">ww-flexbuf-01:/raid0/etransfer/hw04# cat machines</p>
<p class="MsoNormal">ww-flexbuf-01</p>
<p class="MsoNormal">wark167</p>
<p class="MsoNormal">ww-flexbuf-01:/raid0/etransfer/hw04# cat threads</p>
<p class="MsoNormal">NUMBER OF CORES: 6</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<div> <br>
</div>
<p class="MsoNormal">Note our network we have been asked to use a different multicast address, so in DIFXHOME/setup.bash I have set:</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">DIFX_MESSAGE_GROUP=239.253.253.90<br>
</p>
<p class="MsoNormal">DIFX_BINARY_GROUP=239.253.253.90<br>
</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">Any ideas ?</p>
</div></div></div>
_______________________________________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote></div><br></div>