<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Stuart
<div class=""><br class="">
</div>
<div class="">I assume that actually the message was:</div>
<div class=""><br class="">
</div>
<div class="">"<span style="font-family: Menlo; font-size: 11px;" class="">Could not open command monitoring socket! Aborting message receive thread.</span><font face="Menlo" class=""><span style="font-size: 11px;" class="">”</span></font></div>
<div class=""><span style="font-family: Menlo; font-size: 11px;" class=""><br class="">
</span></div>
<div class="">You really need to send the full output for us to have any chance of diagnosing this. When you say “nothing more” in errormon2, does ANYTHING appear there?</div>
<div class=""><br class="">
</div>
<div class="">The message receive thread in most circumstances is not important. If DIFX messages is not working you will however not get any logging messages.</div>
<div class=""><br class="">
</div>
<div class="">Which processes give this message and on which machines are they running? </div>
<div class=""><br class="">
</div>
<div class="">difxmessage library does not report error unfortunately, just return if there were errors. </div>
<div class=""><br class="">
</div>
<div class="">I would suggest making a temp change to <span style="font-family: Menlo; font-size: 11px;" class="">difxmessage/multicast.c </span>and recompiling it and mpifxcorr</div>
<div class=""><br class="">
</div>
<div class="">Add some calls to perror before all the error returns in the routine <span style="font-family: Menlo; font-size: 11px;" class="">openMultiCastSocket</span>. E.g.</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<br class="">
</div>
</div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> /* Make UDP
</span><span style="font-variant-ligatures: no-common-ligatures; color: #ffffff; background-color: #000000" class="">socket</span><span style="font-variant-ligatures: no-common-ligatures" class=""> */</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> sock =
</span><span style="font-variant-ligatures: no-common-ligatures; color: #ffffff; background-color: #000000" class="">socket</span><span style="font-variant-ligatures: no-common-ligatures" class="">(AF_INET, SOCK_DGRAM, 0);</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> if(sock < 0) </span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> {</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> <span class="Apple-tab-span" style="white-space:pre">
</span>perror(“Trying to create socket: ”);</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"></span> return -1;</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> }</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
</span></div>
<div class="">
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> <span class="Apple-tab-span" style="white-space:pre">
</span>/* Allow reuse of port */</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> v = setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes));</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> if(v < 0) </span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> {</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"></span>perror(“Setsockopt: “);</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> return -2;</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> }</span></div>
</div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class="">
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> <span class="Apple-tab-span" style="white-space:pre">
</span>/* bind to receive address */</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> v = bind(sock, (struct sockaddr *)&addr, sizeof(struct sockaddr_in));</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> if(v < 0) </span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> {</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"></span>perror(“Binding to socket: ");</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> return -3;</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> }</span></div>
</div>
<div class=""><span style="font-family: Menlo; font-size: 11px;" class=""> </span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> v = inet_aton(group, &mreq.imr_multiaddr);</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> if(!v) </span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> {</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"></span>perror(“inet_aton: );</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> return -4;</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> }</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
</span></div>
<div class="">I am pretty sure the problem is not the choice of multicast address - if cannot connect to multicast group the code should give a major warning.</div>
<div class=""><br class="">
</div>
<div class="">Just to double check - do you see the following message:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #ffffff; background-color: #000000" class="">Unicast</span><span style="font-variant-ligatures: no-common-ligatures" class="">
(XXXXX) difxMessage in use. Some functionallity may be reduced</span></div>
</div>
<div class=""><br class="">
</div>
<div class="">If you do, thats the problem</div>
<div class=""><br class="">
</div>
<div class="">Cheers</div>
<div class="">Chris</div>
<div class=""><br class="">
<div class="">
<div class="">
<div>
<blockquote type="cite" class="">
<div class="">On 22 Apr 2016, at 11:02 AM, Stuart Weston <<a href="mailto:nzobservers@gmail.com" class="">nzobservers@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<p class="MsoNormal">I have two servers, they both have 2 x CPU ( 6 cores, hyperthreaded). So potentially I have 24 cores and 48 threads.</p>
<div class=""> <br class="webkit-block-placeholder">
</div>
<p class="MsoNormal">mpirun starts mpifxcorr on both servers, but we get the “WARNING Could not open monitoring socket ! Aborting message receive thread” on the master ? The processes seem to sit there and do nothing, nothing more in errmon2.</p>
<div class=""> <br class="webkit-block-placeholder">
</div>
<p class="MsoNormal">If I change the machines file I can run the same correlation on each server individually to completion, so DiFX has to be good.</p>
<div class=""> <br class="webkit-block-placeholder">
</div>
<p class="MsoNormal">ww-flexbuf-01:/raid0/etransfer/hw04# cat machines</p>
<p class="MsoNormal">ww-flexbuf-01</p>
<p class="MsoNormal">wark167</p>
<p class="MsoNormal">ww-flexbuf-01:/raid0/etransfer/hw04# cat threads</p>
<p class="MsoNormal">NUMBER OF CORES: 6</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<p class="MsoNormal">2</p>
<div class=""> <br class="webkit-block-placeholder">
</div>
<p class="MsoNormal">Note our network we have been asked to use a different multicast address, so in DIFXHOME/setup.bash I have set:</p>
<p class="MsoNormal"><br class="">
</p>
<p class="MsoNormal">DIFX_MESSAGE_GROUP=239.253.253.90<br class="">
</p>
<p class="MsoNormal">DIFX_BINARY_GROUP=239.253.253.90<br class="">
</p>
<p class="MsoNormal"><br class="">
</p>
<p class="MsoNormal"><br class="">
</p>
<p class="MsoNormal"><br class="">
</p>
<p class="MsoNormal">Any ideas ?</p>
</div>
_______________________________________________<br class="">
Difx-users mailing list<br class="">
<a href="mailto:Difx-users@listmgr.nrao.edu" class="">Difx-users@listmgr.nrao.edu</a><br class="">
https://listmgr.nrao.edu/mailman/listinfo/difx-users<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</body>
</html>