<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">Hi Arash,</div>
<div class=""><br class="">
</div>
<div class="">A few comments which may (or not) be helpful.</div>
<div class=""><br class="">
</div>
<div class="">Your openmpi problem could be an issue with multiple interfaces, as Adam suggested. I have had this in the past.</div>
<div class=""><br class="">
</div>
<div class="">On the machine which I has this problem, every node I have the following in ~/.openmpi/mca-params.conf</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">btl_tcp_if_include=eth1</div>
<div class="">rmaps=seq </div>
<div class="">mpi_yield_when_idle=1</div>
</div>
<div class=""><br class="">
</div>
<div class="">I think only a variant of the first line would be relevant in your case - you will need to consult with the manual (you can also add these options directly to the mpirun command). You want to figure out the right command option to disable one
of the interfaces (e.g. ethernet if you want DIFX to be using infiniband for data).</div>
<div class=""><br class="">
</div>
<div class="">Wrt lack of output - have you setup the DIFXMESSAGE environment variables:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">setenv DIFX_MESSAGE_GROUP 224.2.2.1</div>
<div class="">setenv DIFX_MESSAGE_PORT 50201</div>
<div class="">setenv DIFX_BINARY_GROUP 224.2.2.1</div>
<div class="">setenv DIFX_BINARY_PORT 50202</div>
</div>
<div class=""><br class="">
</div>
<div class="">These need to be in your .bashrc, .cshrc or similar (or sourced from there).</div>
<div class=""><br class="">
</div>
<div class="">Regards</div>
<div class="">Chris</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 30 Jun 2017, at 9:23 AM, Adam Deller <<a href="mailto:adeller@astro.swin.edu.au" class="">adeller@astro.swin.edu.au</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="">Hi Arash,<br class="">
<br class="">
</div>
A few comments below:<br class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On 30 June 2017 at 04:29, Arash Roshanineshat <span dir="ltr" class="">
<<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank" class="">arash.roshanineshat@cfa.harvard.edu</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi All<br class="">
<br class="">
Thank you all for your help. I want to update you about what I did and what the result was.<br class="">
<br class="">
I tried executing "mpirun" on a target node using "--host" argument:<br class="">
<br class="">
user@fringes-difx0$ mpirun --host fringes-difx1 ...<br class="">
<br class="">
but it couldn't use more than 1 cpu. to use more than a cpu I had to pass "--oversubscribe" argument to "mpirun". Furthermore, although the node had 40 cpus (checked with "htop"), it couldn't use more than 20 cpus. I passed 20 and I also tried passing 90 threads
to "mpirun", but it didn't make any difference. only maximum of 20 of cpus were used.<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">Probably it is 20 physical CPU cores with hyperthreading (=40 virtual CPU cores), and probably MPI / IPP is not using the hyperthreading (which doesn't generally work well for this kind of parallel processing anyway, since all the registers are
being fairly efficiently used already).<br class="">
<br class="">
</div>
<div class="">What happens if you just add --oversubscribe to the mpirun command produced by startdifx? And did you try adding --report-bindings, as I suggested?<br class="">
</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="">
I installed SLURM on the cluster and nodes and done the same experiment using the following command:<br class="">
<br class="">
$ salloc -N 2 mpirun ~/difx/bin/mpispeed<br class="">
<br class="">
All 80 cpus (2 nodes) got involved in some processing. but no output, no update and no results. the output was just bunch of:<br class="">
<br class="">
About to run MPIInit on node fringes-difx0<br class="">
About to run MPIInit on node fringes-difx0<br class="">
About to run MPIInit on node fringes-difx0<br class="">
About to run MPIInit on node fringes-difx0<br class="">
About to run MPIInit on node fringes-difx0<br class="">
About to run MPIInit on node fringes-difx1<br class="">
<br class="">
and it stuck there and I had to Ctrl+C to get out.<br class="">
<br class="">
Based on Adam's suggestion and the experiences above, I decided to try other versions of "openmpi". The experiments above were done using Ubuntu 16.04, openmpi 1.10.2 and SLURM 15.08.7.<br class="">
<br class="">
I tried installing "Openmpi 2.1.1", but no success. I was getting an error about "pmi" component. Although pmi2.h was included in the package and I tried several ways to introduce it, I failed.<br class="">
<br class="">
So I purged and removed Openmpi and its libraries and decided to move to "mpich" (version 3.2) instead.<br class="">
<br class="">
"startdifx" didn't work again. Because it tries to execute a command which includes "mpirun" and "mca" and it seems "mpich" does not recognize "mca".<br class="">
<br class="">
However, the first try using difx and "mpirun" was successful and it used the whole system resources. It also worked on several nodes at the same time as well successfully. I executed a difx experiment and checked the result and it was correct.<br class="">
<br class="">
I would be grateful if you give your feedback and opinion. I also would be happy to track the problem I had with "openmpi".<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">Like Geoff said, I'm guessing we just need to change the command line parameters that startdifx passes to mpirun with later versions of openmpi, and possibly in a way that depends on the networking interface(s) that are being used. I don't have
a convenient setup to try that out on now, but I'd encourage others to try out e.g. openmpi 1.10 and see if they run into similar problems or not.<br class="">
<br class="">
</div>
<div class="">Cheers,<br class="">
</div>
<div class="">Adam<br class="">
</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="">
Best Regards<br class="">
Arash Roshanineshat<br class="">
<br class="">
<br class="">
On 06/28/2017 09:08 PM, Adam Deller wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
OK, then there is clearly a problem with the mpirun command generated by startdifx, if it is giving the same error when you are only using a single node. What is the mpirun command you have used previously which *did* work, and on which machine are you running
it?<br class="">
<br class="">
The other thing that might be an issue is mpirun simply getting confused about how many processes it should be allowed to start on the node. See
<a href="https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php" rel="noreferrer" target="_blank" class="">
https://www.open-mpi.org/doc/v<wbr class="">2.0/man1/mpirun.1.php</a> for a huge list of options (specifically the "Mapping, ranking and binding" section). What happens if you run the mpirun command manually, but add --report-bindings?<br class="">
<br class="">
Cheers,<br class="">
Adam/<br class="">
/<br class="">
<br class="">
On 29 June 2017 at 09:58, Arash Roshanineshat <<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank" class="">arash.roshanineshat@cfa.harva<wbr class="">rd.edu</a> <mailto:<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank" class="">arash.roshanineshat@cf<wbr class="">a.harvard.edu</a>>>
wrote:<br class="">
<br class="">
Hi Adam<br class="">
<br class="">
Thank you for your information.<br class="">
<br class="">
I disabled each node by putting 0 in the third column of cluster<br class="">
configuration file (C.txt) one by one and I doubled checked to see<br class="">
if the disabled node is removed from "threads" and "machines" file.<br class="">
<br class="">
However, the problem is still there with the same error message in<br class="">
both cases.<br class="">
<br class="">
Yes, the nodes in the cluster are connected to each other using 40G<br class="">
cables and the master node that I want it to run difx is connected<br class="">
to my workstation using a regular LAN (RJ-45) cable. So the master<br class="">
node has two interfaces up.<br class="">
<br class="">
<br class="">
Best Regards<br class="">
Arash Roshanineshat<br class="">
<br class="">
<br class="">
On 06/28/2017 07:42 PM, Adam Deller wrote:<br class="">
<br class="">
Hi Arash,<br class="">
<br class="">
I'm fairly sure this is an openmpi issue and not a DiFX issue,<br class="">
hence the number of threads should not be important - it is<br class="">
barfing well before the stage of trying to start processing<br class="">
threads. For some reason, openmpi thinks there are not enough<br class="">
CPUs available to bind processes on your first machine (although<br class="">
there should be, given that you're only allocating 5 processes<br class="">
to it and it has 20 cores). I know Lupin Liu posted a similar<br class="">
problem about 2 years ago, but when I Iook at that thread there<br class="">
was never a resolution - perhaps Lupin can comment? (You can<br class="">
search through the difx-users archive for "While computing<br class="">
bindings", and you'll see it).<br class="">
<br class="">
If you change C.txt to only have one machine enabled (first<br class="">
fringes-difx0, then fringes-difx1), does it work in both cases? Do you have any funny networking infrastructure like infiniband<br class="">
in parallel with ethernet? Sometimes mpi gets confused when<br class="">
multiple interfaces are present.<br class="">
<br class="">
If you can easily do so, I also suggest trying a different<br class="">
version of openmpi.<br class="">
<br class="">
Cheers,<br class="">
Adam<br class="">
<br class="">
On 29 June 2017 at 09:21, Arash Roshanineshat<br class="">
<<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.com</a> <mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.c<wbr class="">om</a>><br class="">
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.c<wbr class="">om</a><br class="">
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.c<wbr class="">om</a>>>> wrote:<br class="">
<br class="">
Thank you Chris.<br class="">
<br class="">
I reduced the number of threads to 5 in the cluster<br class="">
configuration<br class="">
file (C.txt) and the same problem is still there.<br class="">
<br class="">
As per your request, I have attached the files to this email.<br class="">
<br class="">
"errormon2" does not report anything when I execute<br class="">
startdifx. I<br class="">
mean the output of errormon2 is blank.<br class="">
<br class="">
It might be useful to say that, executing "mpispeed", as it was<br class="">
suggested in an archived mail-list email, using the following<br class="">
command and the same configuration files, works correctly.<br class="">
It uses 6<br class="">
cpus in total in both cluster nodes and returns "done" outputs.<br class="">
<br class="">
$ mpirun -np 6 --hostfile<br class="">
/home/arash/Shared_Examples/E<wbr class="">xample2/e17d05-Sm-Sr_1000.mach<wbr class="">ines<br class="">
/home/arash/difx/bin/mpispeed<br class="">
<br class="">
Regarding the large number of threads, If I have understood<br class="">
correctly from NRAO's difx tutorial, MPI should handle the<br class="">
threads(and hyperthreads) automatically. So I chose a large<br class="">
number<br class="">
to use whole system's resources and speed up the difx.<br class="">
<br class="">
Best Regards<br class="">
<br class="">
Arash Roshanineshat<br class="">
<br class="">
<br class="">
<br class="">
On 06/28/2017 07:03 PM, <a href="mailto:Chris.Phillips@csiro.au" class="">
Chris.Phillips@csiro.au</a> wrote:<br class="">
<br class="">
Hi Arash,<br class="">
<br class="">
Without the full setup, it is basically impossible to<br class="">
debug such<br class="">
problems. We would need to see the .input file and<br class="">
preferably<br class="">
the .v2d and .vex file also. Also the full output from<br class="">
DIFX for<br class="">
the entire run (use errormon or errormon2)<br class="">
<br class="">
Specifically how many antennas are being correlated?<br class="">
Maybe you<br class="">
just have too few DIFX processes.<br class="">
<br class="">
Given you are running all the DataStream processes in a<br class="">
single<br class="">
node, I would suggest you probably have too many<br class="">
threads running<br class="">
per core. I doubt that is the problem you are seeing<br class="">
though.<br class="">
<br class="">
Actually maybe it is - stardifx maybe is being clever<br class="">
(I don't<br class="">
use it) and not being willing to allocate more than 20<br class="">
processes<br class="">
(ignore hyperthreads, they tend to be useless for<br class="">
DIFX). Try<br class="">
changing # threads to, say, 5.<br class="">
<br class="">
Cheers<br class="">
Chris<br class="">
<br class="">
_____________________________<wbr class="">___________<br class="">
From: Difx-users <<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank" class="">difx-users-bounces@listmgr.nr<wbr class="">ao.edu</a><br class="">
<mailto:<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank" class="">difx-users-bounces@lis<wbr class="">tmgr.nrao.edu</a>><br class="">
<mailto:<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank" class="">difx-users-bounces@li<wbr class="">stmgr.nrao.edu</a><br class="">
<mailto:<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank" class="">difx-users-bounces@lis<wbr class="">tmgr.nrao.edu</a>>>> on behalf of Arash<br class="">
Roshanineshat <<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.com</a><br class="">
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.c<wbr class="">om</a>><br class="">
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.<wbr class="">com</a><br class="">
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.c<wbr class="">om</a>>>><br class="">
Sent: Thursday, 29 June 2017 7:34 AM<br class="">
To: <a href="mailto:difx-users@listmgr.nrao.edu" target="_blank" class="">
difx-users@listmgr.nrao.edu</a><br class="">
<mailto:<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank" class="">difx-users@listmgr.nra<wbr class="">o.edu</a>><br class="">
<mailto:<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank" class="">difx-users@listmgr.nra<wbr class="">o.edu</a><br class="">
<mailto:<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank" class="">difx-users@listmgr.nra<wbr class="">o.edu</a>>><br class="">
Cc: <a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank" class="">
arash.roshanineshat@cfa.harvar<wbr class="">d.edu</a><br class="">
<mailto:<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank" class="">arash.roshanineshat@cf<wbr class="">a.harvard.edu</a>><br class="">
<mailto:<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank" class="">arash.roshanineshat@c<wbr class="">fa.harvard.edu</a><br class="">
<mailto:<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank" class="">arash.roshanineshat@cf<wbr class="">a.harvard.edu</a>>><br class="">
<br class="">
Subject: [Difx-users] mpicorrdifx cannot be loaded<br class="">
correctly on<br class="">
more than a single node<br class="">
<br class="">
Hi,<br class="">
<br class="">
I could install difx but it can only be run on a single<br class="">
node<br class="">
cluster.<br class="">
<br class="">
The *.machines and *.threads files are attached to this<br class="">
email.<br class="">
<br class="">
Openmpi is installed on all nodes and difx folder and<br class="">
data folder is<br class="">
shared among the clusters using NFS filesystem. Difx<br class="">
works perfectly<br class="">
with correct output on single machines.<br class="">
<br class="">
executing "startdifx -v -f e17d05-Sm-Sr_1000.input"<br class="">
returns the<br class="">
following error:<br class="">
<br class="">
DIFX_MACHINES -> /home/arash/Shared_Examples/Ex<wbr class="">ample2/C.txt<br class="">
Found modules:<br class="">
Executing: mpirun -np 6 --hostfile<br class="">
/home/arash/Shared_Examples/Ex<wbr class="">ample2/e17d05-Sm-Sr_1000.machi<wbr class="">nes<br class="">
--mca<br class="">
mpi_yield_when_idle 1 --mca rmaps seq runmpifxcorr.DiFX-2.5<br class="">
/home/arash/Shared_Examples/Ex<wbr class="">ample2/e17d05-Sm-Sr_1000.input<br class="">
------------------------------<wbr class="">------------------------------<wbr class="">--------------<br class="">
While computing bindings, we found no available cpus on<br class="">
the following node:<br class="">
<br class="">
Node: fringes-difx0<br class="">
<br class="">
Please check your allocation.<br class="">
------------------------------<wbr class="">------------------------------<wbr class="">--------------<br class="">
Elapsed time (s) = 0.50417590141<br class="">
<br class="">
and executing<br class="">
<br class="">
$ mpirun -np 6 --hostfile<br class="">
/home/arash/Shared_Examples/Ex<wbr class="">ample2/e17d05-Sm-Sr_1000.machi<wbr class="">nes<br class="">
/home/arash/difx/bin/<wbr class="">mpifxcorr<br class="">
/home/arash/Shared_Examples/Ex<wbr class="">ample2/e17d05-Sm-Sr_1000.input<br class="">
<br class="">
seems to be working but by observing the cpu usage, I<br class="">
see only 6<br class="">
cpus<br class="">
involving "5 in fringes-difx0 and 1 in fringes-difx1".<br class="">
I was<br class="">
expecting<br class="">
it to use the number of cpus equal to the number in<br class="">
"*.threads"<br class="">
file.<br class="">
How can I solve this issue?<br class="">
<br class="">
the specification of the cluster is Socket=2, Core per<br class="">
Socket=10 and<br class="">
Threads per core=2.<br class="">
<br class="">
Best Regards<br class="">
<br class="">
Arash Roshanineshat<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
_____________________________<wbr class="">__________________<br class="">
Difx-users mailing list<br class="">
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank" class="">Difx-users@listmgr.nrao.edu</a> <mailto:<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank" class="">Difx-users@listmgr.nra<wbr class="">o.edu</a>><br class="">
<mailto:<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank" class="">Difx-users@listmgr.nra<wbr class="">o.edu</a><br class="">
<mailto:<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank" class="">Difx-users@listmgr.nra<wbr class="">o.edu</a>>><br class="">
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank" class="">
https://listmgr.nrao.edu/mailm<wbr class="">an/listinfo/difx-users</a><br class="">
<<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank" class="">https://listmgr.nrao.edu/mail<wbr class="">man/listinfo/difx-users</a>><br class="">
<<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank" class="">https://listmgr.nrao.edu/mai<wbr class="">lman/listinfo/difx-users</a><br class="">
<<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank" class="">https://listmgr.nrao.edu/mail<wbr class="">man/listinfo/difx-users</a>>><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
-- !============================<wbr class="">==============================<wbr class="">===!<br class="">
Dr. Adam Deller<br class="">
ARC Future Fellow, Senior Lecturer<br class="">
Centre for Astrophysics & Supercomputing<br class="">
Swinburne University of Technology<br class="">
John St, Hawthorn VIC 3122 Australia<br class="">
phone: <a href="tel:%2B61%203%209214%205307" value="+61392145307" target="_blank" class="">
+61 3 9214 5307</a> <<a href="tel:%2B61%203%209214%205307" class="">tel:%2B61%203%209214%205307</a>><br class="">
fax: <a href="tel:%2B61%203%209214%208797" value="+61392148797" target="_blank" class="">
+61 3 9214 8797</a> <<a href="tel:%2B61%203%209214%208797" class="">tel:%2B61%203%209214%208797</a>><br class="">
<br class="">
office days (usually): Mon-Thu<br class="">
!=============================<wbr class="">==============================<wbr class="">==!<br class="">
<br class="">
<br class="">
<br class="">
<span class="HOEnZb"><font color="#888888" class=""><br class="">
-- <br class="">
!=============================<wbr class="">==============================<wbr class="">==!<br class="">
Dr. Adam Deller<br class="">
ARC Future Fellow, Senior Lecturer<br class="">
Centre for Astrophysics & Supercomputing<br class="">
Swinburne University of Technology<br class="">
John St, Hawthorn VIC 3122 Australia<br class="">
phone: <a href="tel:%2B61%203%209214%205307" value="+61392145307" target="_blank" class="">
+61 3 9214 5307</a><br class="">
fax: <a href="tel:%2B61%203%209214%208797" value="+61392148797" target="_blank" class="">
+61 3 9214 8797</a><br class="">
<br class="">
office days (usually): Mon-Thu<br class="">
!=============================<wbr class="">==============================<wbr class="">==!<br class="">
</font></span></blockquote>
</blockquote>
</div>
<br class="">
<br clear="all" class="">
<br class="">
-- <br class="">
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div dir="ltr" style="font-size:12.8000001907349px" class="">
<div dir="ltr" style="font-size:12.8px" class="">
<div dir="ltr" style="font-size:12.8px" class="">
<div dir="ltr" style="font-size:12.8px" class="">
<div dir="ltr" style="font-size:12.8px" class="">!=============================================================!<br class="">
Dr. Adam Deller </div>
<div dir="ltr" style="font-size:12.8px" class="">ARC Future Fellow, Senior Lecturer</div>
<div style="font-size:12.8px" class="">Centre for Astrophysics & Supercomputing </div>
<div dir="ltr" style="font-size:12.8px" class="">Swinburne University of Technology <br class="">
John St, Hawthorn VIC 3122 Australia</div>
<div style="font-size:12.8px" class="">phone: +61 3 9214 5307</div>
<div style="font-size:12.8px" class="">fax: +61 3 9214 8797</div>
<div style="font-size:12.8px" class=""><br class="">
</div>
<div style="font-size:12.8px" class="">office days (usually): Mon-Thu<br class="">
!=============================================================!</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>