<div dir="ltr">Hi all,<div><br></div><div>I'm what you might call an MPI newbie, and I've been trying to run mpifxcorr on a cluster with the Slurm scheduler and running into some problems. In the cluster setup, there's an InfiniBand device that handles communication between nodes, but the setup doesn't seem to recognize/utilize it properly.</div><div><br></div><div>For the command line prompt,</div><div><span style="font-family:monospace">salloc -N 7 mpirun -np 7 mpifxcorr ${EXPER}.input</span><br></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">I get:</span></div><div><span style="font-family:monospace">WARNING: There is at least non-excluded one OpenFabrics device found, but there are no active ports detected (or Open MPI was unable to use them). This is most certainly not what you wanted. Check your cables, subnet manager configuration, etc. The openib BTL will be ignored for this job. Local host: nod50</span><br></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">This leads to a fatal failure to connect between nodes (I think):</span></div><div><div class="gmail-adn gmail-ads" style="border-left:none;padding:0px;display:flex;font-size:medium"><div class="gmail-gs" style="margin:0px;padding:0px 0px 20px;width:1632px"><div class="gmail-" style=""><div id="gmail-:zj" class="gmail-ii gmail-gt" style="font-size:0.875rem;direction:ltr;margin:8px 0px 0px;padding:0px"><div id="gmail-:zi" class="gmail-a3s gmail-aiL" style="overflow:hidden;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:small;line-height:1.5"><div style=""><p style="font-family:Arial,Helvetica,sans-serif"><code>WARNING: Open MPI failed to TCP connect to a peer MPI process. This should not happen. Your Open MPI job may now fail. Local host: nod77 PID: 4410 Message: connect() to <a href="http://192.168.5.76:1024"><font color="red"><b>MailScanner warning: numerical links are often malicious:</b></font> 192.168.5.76:1024</a> failed Error: Operation now in progress (115)</code></p><p style="font-family:Arial,Helvetica,sans-serif"><code>Notably, if I force connection through an ethernet device with the command line prompt,</code></p><p style="font-family:Arial,Helvetica,sans-serif"><code>salloc -N 7 mpirun -np 7 --mca btl_tcp_if_include eth0 mpifxcorr ${EXPER}.input<br></code></p><p style="">mpifxcorr runs with no problem, although presumably at a large loss in efficiency<font face="monospace">.</font></p><p style="font-family:Arial,Helvetica,sans-serif">This may be impossible to diagnose without knowing more about the server/cluster architecture, but I figured I'd see if anyone else has run into similar issues and found a solution. It's also entirely possible I'm missing something obvious.</p><p style="font-family:Arial,Helvetica,sans-serif"><br></p><p style="font-family:Arial,Helvetica,sans-serif">Thanks,</p><p style="font-family:Arial,Helvetica,sans-serif">Joe Skeens</p><p style="font-family:Arial,Helvetica,sans-serif"><br></p><div class="gmail-yj6qo" style="font-family:Arial,Helvetica,sans-serif"></div><div class="gmail-adL" style="font-family:Arial,Helvetica,sans-serif"></div></div><div class="gmail-adL" style="font-family:Arial,Helvetica,sans-serif"></div></div></div><div class="gmail-hi" style="font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;border-bottom-left-radius:1px;border-bottom-right-radius:1px;padding:0px;width:auto;background:rgb(242,242,242);margin:0px"></div></div></div><div class="gmail-ajx" style="font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;clear:both"></div></div><div class="gmail-gA gmail-gt gmail-acV" style="font-size:0.875rem;padding:0px;width:auto;border-bottom-left-radius:0px;border-bottom-right-radius:0px;border-top:none;margin:0px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif"><div class="gmail-gB gmail-xu" style="border-top:0px;padding:0px"><div class="gmail-ip gmail-iq" style="clear:both;margin:0px;padding:16px 0px;border-top:none"><div id="gmail-:zk"><table class="gmail-cf gmail-wS" role="presentation" style="border-collapse:collapse"><tbody><tr><td class="gmail-amq" style="padding:0px 16px;vertical-align:top;width:44px"><img id="gmail-:ll_19" name=":ll" src="https://ssl.gstatic.com/ui/v1/icons/mail/no_photo.png" class="gmail-ajn gmail-bofPge" style="display: block; width: 40px; height: 40px; border-radius: 50%;"></td><td class="gmail-amr" style="padding:0px;width:1632px"><div class="gmail-nr gmail-wR" style="box-sizing:border-box;border-radius:1px;padding:0px;border:none;margin:0px"><div class="gmail-amn" style="color:inherit;height:auto;padding:0px;display:flex;line-height:20px"><span id="gmail-:zd" role="link" tabindex="0" class="gmail-ams gmail-bkH" style="border:none;display:inline-flex;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:0.875rem;letter-spacing:0.25px;background:none rgba(32,33,36,0.06);border-radius:4px;box-sizing:border-box;color:rgb(95,99,104);height:36px;outline:none;padding:0px 16px 0px 12px;min-width:104px;margin-right:12px">Reply</span><span id="gmail-:zf" role="link" tabindex="0" class="gmail-ams gmail-bkG" style="border:none;display:inline-flex;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:0.875rem;letter-spacing:0.25px;background:none;border-radius:4px;box-sizing:border-box;color:rgb(95,99,104);height:36px;outline:none;padding:0px 16px 0px 12px;min-width:104px;margin-right:12px">Forward</span></div></div></td></tr></tbody></table></div></div></div></div></div></div>