<div dir="ltr"><div>Hi Arash,<br><br></div>A few comments below:<br><div class="gmail_extra"><br><div class="gmail_quote">On 30 June 2017 at 04:29, Arash Roshanineshat <span dir="ltr"><<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank">arash.roshanineshat@cfa.harvard.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi All<br>
<br>
Thank you all for your help. I want to update you about what I did and what the result was.<br>
<br>
I tried executing "mpirun" on a target node using "--host" argument:<br>
<br>
user@fringes-difx0$ mpirun --host fringes-difx1 ...<br>
<br>
but it couldn't use more than 1 cpu. to use more than a cpu I had to pass "--oversubscribe" argument to "mpirun". Furthermore, although the node had 40 cpus (checked with "htop"), it couldn't use more than 20 cpus. I passed 20 and I also tried passing 90 threads to "mpirun", but it didn't make any difference. only maximum of 20 of cpus were used.<br></blockquote><div><br></div><div>Probably it is 20 physical CPU cores with hyperthreading (=40 virtual CPU cores), and probably MPI / IPP is not using the hyperthreading (which doesn't generally work well for this kind of parallel processing anyway, since all the registers are being fairly efficiently used already).<br><br></div><div>What happens if you just add --oversubscribe to the mpirun command produced by startdifx? And did you try adding --report-bindings, as I suggested?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I installed SLURM on the cluster and nodes and done the same experiment using the following command:<br>
<br>
$ salloc -N 2 mpirun ~/difx/bin/mpispeed<br>
<br>
All 80 cpus (2 nodes) got involved in some processing. but no output, no update and no results. the output was just bunch of:<br>
<br>
About to run MPIInit on node fringes-difx0<br>
About to run MPIInit on node fringes-difx0<br>
About to run MPIInit on node fringes-difx0<br>
About to run MPIInit on node fringes-difx0<br>
About to run MPIInit on node fringes-difx0<br>
About to run MPIInit on node fringes-difx1<br>
<br>
and it stuck there and I had to Ctrl+C to get out.<br>
<br>
Based on Adam's suggestion and the experiences above, I decided to try other versions of "openmpi". The experiments above were done using Ubuntu 16.04, openmpi 1.10.2 and SLURM 15.08.7.<br>
<br>
I tried installing "Openmpi 2.1.1", but no success. I was getting an error about "pmi" component. Although pmi2.h was included in the package and I tried several ways to introduce it, I failed.<br>
<br>
So I purged and removed Openmpi and its libraries and decided to move to "mpich" (version 3.2) instead.<br>
<br>
"startdifx" didn't work again. Because it tries to execute a command which includes "mpirun" and "mca" and it seems "mpich" does not recognize "mca".<br>
<br>
However, the first try using difx and "mpirun" was successful and it used the whole system resources. It also worked on several nodes at the same time as well successfully. I executed a difx experiment and checked the result and it was correct.<br>
<br>
I would be grateful if you give your feedback and opinion. I also would be happy to track the problem I had with "openmpi".<br></blockquote><div><br></div><div>Like Geoff said, I'm guessing we just need to change the command line parameters that startdifx passes to mpirun with later versions of openmpi, and possibly in a way that depends on the networking interface(s) that are being used. I don't have a convenient setup to try that out on now, but I'd encourage others to try out e.g. openmpi 1.10 and see if they run into similar problems or not.<br><br></div><div>Cheers,<br></div><div>Adam<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Best Regards<br>
Arash Roshanineshat<br>
<br>
<br>
On 06/28/2017 09:08 PM, Adam Deller wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
OK, then there is clearly a problem with the mpirun command generated by startdifx, if it is giving the same error when you are only using a single node. What is the mpirun command you have used previously which *did* work, and on which machine are you running it?<br>
<br>
The other thing that might be an issue is mpirun simply getting confused about how many processes it should be allowed to start on the node. See <a href="https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php" rel="noreferrer" target="_blank">https://www.open-mpi.org/doc/v<wbr>2.0/man1/mpirun.1.php</a> for a huge list of options (specifically the "Mapping, ranking and binding" section). What happens if you run the mpirun command manually, but add --report-bindings?<br>
<br>
Cheers,<br>
Adam/<br>
/<br>
<br>
On 29 June 2017 at 09:58, Arash Roshanineshat <<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank">arash.roshanineshat@cfa.harva<wbr>rd.edu</a> <mailto:<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank">arash.roshanineshat@cf<wbr>a.harvard.edu</a>>> wrote:<br>
<br>
Hi Adam<br>
<br>
Thank you for your information.<br>
<br>
I disabled each node by putting 0 in the third column of cluster<br>
configuration file (C.txt) one by one and I doubled checked to see<br>
if the disabled node is removed from "threads" and "machines" file.<br>
<br>
However, the problem is still there with the same error message in<br>
both cases.<br>
<br>
Yes, the nodes in the cluster are connected to each other using 40G<br>
cables and the master node that I want it to run difx is connected<br>
to my workstation using a regular LAN (RJ-45) cable. So the master<br>
node has two interfaces up.<br>
<br>
<br>
Best Regards<br>
Arash Roshanineshat<br>
<br>
<br>
On 06/28/2017 07:42 PM, Adam Deller wrote:<br>
<br>
Hi Arash,<br>
<br>
I'm fairly sure this is an openmpi issue and not a DiFX issue,<br>
hence the number of threads should not be important - it is<br>
barfing well before the stage of trying to start processing<br>
threads. For some reason, openmpi thinks there are not enough<br>
CPUs available to bind processes on your first machine (although<br>
there should be, given that you're only allocating 5 processes<br>
to it and it has 20 cores). I know Lupin Liu posted a similar<br>
problem about 2 years ago, but when I Iook at that thread there<br>
was never a resolution - perhaps Lupin can comment? (You can<br>
search through the difx-users archive for "While computing<br>
bindings", and you'll see it).<br>
<br>
If you change C.txt to only have one machine enabled (first<br>
fringes-difx0, then fringes-difx1), does it work in both cases? Do you have any funny networking infrastructure like infiniband<br>
in parallel with ethernet? Sometimes mpi gets confused when<br>
multiple interfaces are present.<br>
<br>
If you can easily do so, I also suggest trying a different<br>
version of openmpi.<br>
<br>
Cheers,<br>
Adam<br>
<br>
On 29 June 2017 at 09:21, Arash Roshanineshat<br>
<<a href="mailto:arashroshani92@gmail.com" target="_blank">arashroshani92@gmail.com</a> <mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank">arashroshani92@gmail.c<wbr>om</a>><br>
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank">arashroshani92@gmail.c<wbr>om</a><br>
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank">arashroshani92@gmail.c<wbr>om</a>>>> wrote:<br>
<br>
Thank you Chris.<br>
<br>
I reduced the number of threads to 5 in the cluster<br>
configuration<br>
file (C.txt) and the same problem is still there.<br>
<br>
As per your request, I have attached the files to this email.<br>
<br>
"errormon2" does not report anything when I execute<br>
startdifx. I<br>
mean the output of errormon2 is blank.<br>
<br>
It might be useful to say that, executing "mpispeed", as it was<br>
suggested in an archived mail-list email, using the following<br>
command and the same configuration files, works correctly.<br>
It uses 6<br>
cpus in total in both cluster nodes and returns "done" outputs.<br>
<br>
$ mpirun -np 6 --hostfile<br>
/home/arash/Shared_Examples/E<wbr>xample2/e17d05-Sm-Sr_1000.mach<wbr>ines<br>
/home/arash/difx/bin/mpispeed<br>
<br>
Regarding the large number of threads, If I have understood<br>
correctly from NRAO's difx tutorial, MPI should handle the<br>
threads(and hyperthreads) automatically. So I chose a large<br>
number<br>
to use whole system's resources and speed up the difx.<br>
<br>
Best Regards<br>
<br>
Arash Roshanineshat<br>
<br>
<br>
<br>
On 06/28/2017 07:03 PM, Chris.Phillips@csiro.au wrote:<br>
<br>
Hi Arash,<br>
<br>
Without the full setup, it is basically impossible to<br>
debug such<br>
problems. We would need to see the .input file and<br>
preferably<br>
the .v2d and .vex file also. Also the full output from<br>
DIFX for<br>
the entire run (use errormon or errormon2)<br>
<br>
Specifically how many antennas are being correlated?<br>
Maybe you<br>
just have too few DIFX processes.<br>
<br>
Given you are running all the DataStream processes in a<br>
single<br>
node, I would suggest you probably have too many<br>
threads running<br>
per core. I doubt that is the problem you are seeing<br>
though.<br>
<br>
Actually maybe it is - stardifx maybe is being clever<br>
(I don't<br>
use it) and not being willing to allocate more than 20<br>
processes<br>
(ignore hyperthreads, they tend to be useless for<br>
DIFX). Try<br>
changing # threads to, say, 5.<br>
<br>
Cheers<br>
Chris<br>
<br>
_____________________________<wbr>___________<br>
From: Difx-users <<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank">difx-users-bounces@listmgr.nr<wbr>ao.edu</a><br>
<mailto:<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank">difx-users-bounces@lis<wbr>tmgr.nrao.edu</a>><br>
<mailto:<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank">difx-users-bounces@li<wbr>stmgr.nrao.edu</a><br>
<mailto:<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank">difx-users-bounces@lis<wbr>tmgr.nrao.edu</a>>>> on behalf of Arash<br>
Roshanineshat <<a href="mailto:arashroshani92@gmail.com" target="_blank">arashroshani92@gmail.com</a><br>
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank">arashroshani92@gmail.c<wbr>om</a>><br>
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank">arashroshani92@gmail.<wbr>com</a><br>
<mailto:<a href="mailto:arashroshani92@gmail.com" target="_blank">arashroshani92@gmail.c<wbr>om</a>>>><br>
Sent: Thursday, 29 June 2017 7:34 AM<br>
To: <a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nrao.edu</a><br>
<mailto:<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nra<wbr>o.edu</a>><br>
<mailto:<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nra<wbr>o.edu</a><br>
<mailto:<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nra<wbr>o.edu</a>>><br>
Cc: <a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank">arash.roshanineshat@cfa.harvar<wbr>d.edu</a><br>
<mailto:<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank">arash.roshanineshat@cf<wbr>a.harvard.edu</a>><br>
<mailto:<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank">arash.roshanineshat@c<wbr>fa.harvard.edu</a><br>
<mailto:<a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank">arash.roshanineshat@cf<wbr>a.harvard.edu</a>>><br>
<br>
Subject: [Difx-users] mpicorrdifx cannot be loaded<br>
correctly on<br>
more than a single node<br>
<br>
Hi,<br>
<br>
I could install difx but it can only be run on a single<br>
node<br>
cluster.<br>
<br>
The *.machines and *.threads files are attached to this<br>
email.<br>
<br>
Openmpi is installed on all nodes and difx folder and<br>
data folder is<br>
shared among the clusters using NFS filesystem. Difx<br>
works perfectly<br>
with correct output on single machines.<br>
<br>
executing "startdifx -v -f e17d05-Sm-Sr_1000.input"<br>
returns the<br>
following error:<br>
<br>
DIFX_MACHINES -> /home/arash/Shared_Examples/Ex<wbr>ample2/C.txt<br>
Found modules:<br>
Executing: mpirun -np 6 --hostfile<br>
/home/arash/Shared_Examples/Ex<wbr>ample2/e17d05-Sm-Sr_1000.machi<wbr>nes<br>
--mca<br>
mpi_yield_when_idle 1 --mca rmaps seq runmpifxcorr.DiFX-2.5<br>
/home/arash/Shared_Examples/Ex<wbr>ample2/e17d05-Sm-Sr_1000.input<br>
------------------------------<wbr>------------------------------<wbr>--------------<br>
While computing bindings, we found no available cpus on<br>
the following node:<br>
<br>
Node: fringes-difx0<br>
<br>
Please check your allocation.<br>
------------------------------<wbr>------------------------------<wbr>--------------<br>
Elapsed time (s) = 0.50417590141<br>
<br>
and executing<br>
<br>
$ mpirun -np 6 --hostfile<br>
/home/arash/Shared_Examples/Ex<wbr>ample2/e17d05-Sm-Sr_1000.machi<wbr>nes<br>
/home/arash/difx/bin/<wbr>mpifxcorr<br>
/home/arash/Shared_Examples/Ex<wbr>ample2/e17d05-Sm-Sr_1000.input<br>
<br>
seems to be working but by observing the cpu usage, I<br>
see only 6<br>
cpus<br>
involving "5 in fringes-difx0 and 1 in fringes-difx1".<br>
I was<br>
expecting<br>
it to use the number of cpus equal to the number in<br>
"*.threads"<br>
file.<br>
How can I solve this issue?<br>
<br>
the specification of the cluster is Socket=2, Core per<br>
Socket=10 and<br>
Threads per core=2.<br>
<br>
Best Regards<br>
<br>
Arash Roshanineshat<br>
<br>
<br>
<br>
<br>
<br>
<br>
_____________________________<wbr>__________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a> <mailto:<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nra<wbr>o.edu</a>><br>
<mailto:<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nra<wbr>o.edu</a><br>
<mailto:<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nra<wbr>o.edu</a>>><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/difx-users</a><br>
<<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mail<wbr>man/listinfo/difx-users</a>><br>
<<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mai<wbr>lman/listinfo/difx-users</a><br>
<<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mail<wbr>man/listinfo/difx-users</a>>><br>
<br>
<br>
<br>
<br>
-- !============================<wbr>==============================<wbr>===!<br>
Dr. Adam Deller<br>
ARC Future Fellow, Senior Lecturer<br>
Centre for Astrophysics & Supercomputing<br>
Swinburne University of Technology<br>
John St, Hawthorn VIC 3122 Australia<br>
phone: <a href="tel:%2B61%203%209214%205307" value="+61392145307" target="_blank">+61 3 9214 5307</a> <tel:%2B61%203%209214%205307><br>
fax: <a href="tel:%2B61%203%209214%208797" value="+61392148797" target="_blank">+61 3 9214 8797</a> <tel:%2B61%203%209214%208797><br>
<br>
office days (usually): Mon-Thu<br>
!=============================<wbr>==============================<wbr>==!<br>
<br>
<br>
<br><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
!=============================<wbr>==============================<wbr>==!<br>
Dr. Adam Deller<br>
ARC Future Fellow, Senior Lecturer<br>
Centre for Astrophysics & Supercomputing<br>
Swinburne University of Technology<br>
John St, Hawthorn VIC 3122 Australia<br>
phone: <a href="tel:%2B61%203%209214%205307" value="+61392145307" target="_blank">+61 3 9214 5307</a><br>
fax: <a href="tel:%2B61%203%209214%208797" value="+61392148797" target="_blank">+61 3 9214 8797</a><br>
<br>
office days (usually): Mon-Thu<br>
!=============================<wbr>==============================<wbr>==!<br>
</font></span></blockquote>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8000001907349px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px">!=============================================================!<br>Dr. Adam Deller </div><div dir="ltr" style="font-size:12.8px">ARC Future Fellow, Senior Lecturer</div><div style="font-size:12.8px">Centre for Astrophysics & Supercomputing </div><div dir="ltr" style="font-size:12.8px">Swinburne University of Technology <br>John St, Hawthorn VIC 3122 Australia</div><div style="font-size:12.8px">phone: +61 3 9214 5307</div><div style="font-size:12.8px">fax: +61 3 9214 8797</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">office days (usually): Mon-Thu<br>!=============================================================!</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div></div>