<html><head><meta http-equiv="Content-Type" content="text/html charset=big5"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Dear Adam,</div><div class=""><br class=""></div><div class="">I forgot whether I really reported the problem in using startdifx two years ago, but I indeed post the problem about startdifx on Nov. 4 of 2016 and Apr. 10 of 2017.</div><div class="">I noted these problems because we were installing the DiFX (both trunk and ver 2.4) in the new computers; and one is for Ubuntu 16.04 while the second time is for Scientific Linux 6.8.</div><div class="">(Please note the error messages obtained from two posts are not the same.)</div><div class="">But I think my problem is not similar to what Arash met.</div><div class=""><br class=""></div><div class="">In reality, I did not fully solve my problems in startdifx, but just find other way to replace the usage on “startdifx”.</div><div class="">Now we use "mpirun --machinefile xxxx.machines -np xx mpifxcorr xxxx.input” in correlation to replace the original "startdifx -v -v -n -f xxxx.input”.</div><div class="">In the previous post, I followed Walter’s suggestion to examine the mpispeed problem, but I can successfully pass the test.</div><div class="">I also checked the installation of different openmpi versions, but I still cannot solve the problem in using “startdifx”.</div><div class="">Since my purpose is to correlate the data, I have not been seriously tied by the problem of "startdifx” …….. </div><div class="">I also wish to solve this problem, but I have no idea on other tests even I checked some documents related to openmpi. </div><div class="">(But in reality, I only have basic experiences in using openmpi as well …)</div><div class=""><br class=""></div><div class=""> </div><div class="">Sincerely,</div><div class="">
<div class=""><div style="font-family: 新細明體, sans-serif; font-size: 14px;" class=""><span style="font-size: 12px;" class="">--</span></div><div style="font-family: 新細明體, sans-serif; font-size: 14px;" class=""><span style="font-size: 12px;" class="">Lupin Chun-Che Lin</span></div><div style="font-family: 新細明體, sans-serif; font-size: 14px;" class=""><span style="font-size: 12px;" class="">Supporting Scientist of GLT (GreenLand Telescope) project </span></div><div style="font-family: 新細明體, sans-serif; font-size: 14px;" class=""><span style="font-size: 12px;" class="">in Institute of Astronomy and Astrophysics, Academia Sinica,</span></div><div style="font-family: 新細明體, sans-serif; font-size: 14px;" class=""><span style="font-size: 12px;" class="">14F of Astronomy-Mathematics Building (Rm: 1405), </span></div><div style="font-family: 新細明體, sans-serif; font-size: 14px;" class=""><span style="font-size: 12px;" class="">National Taiwan University. </span></div><div style="font-family: 新細明體, sans-serif; font-size: 14px;" class=""><span style="font-size: 12px;" class="">No.1, Sec. 4, Roosevelt Rd, Taipei 10617, Taiwan.</span></div><div style="font-family: 新細明體, sans-serif; font-size: 14px;" class=""><span style="font-size: 12px;" class="">Tel: +886-2-2366-5464</span></div><div style="font-family: 新細明體, sans-serif; font-size: 14px;" class=""><span style="font-size: 12px;" class="">Fax: +886-2-2367-7849</span></div></div><div class=""><br class=""></div><br class="Apple-interchange-newline">
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">Adam Deller <<a href="mailto:adeller@astro.swin.edu.au" class="">adeller@astro.swin.edu.au</a>> 於 2017年6月29日 上午7:42 寫道：</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">Hi Arash,<br class=""><br class=""></div><div class="">I'm fairly sure this is an openmpi issue and not a DiFX issue, hence the number of threads should not be important - it is barfing well before the stage of trying to start processing threads. For some reason, openmpi thinks there are not enough CPUs available to bind processes on your first machine (although there should be, given that you're only allocating 5 processes to it and it has 20 cores). I know Lupin Liu posted a similar problem about 2 years ago, but when I Iook at that thread there was never a resolution - perhaps Lupin can comment? (You can search through the difx-users archive for "While computing bindings", and you'll see it). <br class=""><br class=""></div><div class="">If you change C.txt to only have one machine enabled (first fringes-difx0, then fringes-difx1), does it work in both cases? Do you have any funny networking infrastructure like infiniband in parallel with ethernet? Sometimes mpi gets confused when multiple interfaces are present.<br class=""></div><div class=""><br class=""></div><div class="">If you can easily do so, I also suggest trying a different version of openmpi. <br class=""><br class=""></div><div class="">Cheers,<br class=""></div><div class="">Adam<br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 29 June 2017 at 09:21, Arash Roshanineshat <span dir="ltr" class=""><<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thank you Chris.<br class="">
<br class="">
I reduced the number of threads to 5 in the cluster configuration file (C.txt) and the same problem is still there.<br class="">
<br class="">
As per your request, I have attached the files to this email.<br class="">
<br class="">
"errormon2" does not report anything when I execute startdifx. I mean the output of errormon2 is blank.<br class="">
<br class="">
It might be useful to say that, executing "mpispeed", as it was suggested in an archived mail-list email, using the following command and the same configuration files, works correctly. It uses 6 cpus in total in both cluster nodes and returns "done" outputs.<br class="">
<br class="">
$ mpirun -np 6 --hostfile /home/arash/Shared_Examples/Ex<wbr class="">ample2/e17d05-Sm-Sr_1000.machi<wbr class="">nes /home/arash/difx/bin/mpispeed<br class="">
<br class="">
Regarding the large number of threads, If I have understood correctly from NRAO's difx tutorial, MPI should handle the threads(and hyperthreads) automatically. So I chose a large number to use whole system's resources and speed up the difx.<br class="">
<br class="">
Best Regards<span class="HOEnZb"><font color="#888888" class=""><br class="">
<br class="">
Arash Roshanineshat</font></span><div class="HOEnZb"><div class="h5"><br class="">
<br class="">
<br class="">
On 06/28/2017 07:03 PM, <a href="mailto:Chris.Phillips@csiro.au" class="">Chris.Phillips@csiro.au</a> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Arash,<br class="">
<br class="">
Without the full setup, it is basically impossible to debug such problems. We would need to see the .input file and preferably the .v2d and .vex file also. Also the full output from DIFX for the entire run (use errormon or errormon2)<br class="">
<br class="">
Specifically how many antennas are being correlated? Maybe you just have too few DIFX processes.<br class="">
<br class="">
Given you are running all the DataStream processes in a single node, I would suggest you probably have too many threads running per core. I doubt that is the problem you are seeing though.<br class="">
<br class="">
Actually maybe it is - stardifx maybe is being clever (I don't use it) and not being willing to allocate more than 20 processes (ignore hyperthreads, they tend to be useless for DIFX). Try changing # threads to, say, 5.<br class="">
<br class="">
Cheers<br class="">
Chris<br class="">
<br class="">
______________________________<wbr class="">__________<br class="">
From: Difx-users <<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank" class="">difx-users-bounces@listmgr.nr<wbr class="">ao.edu</a>> on behalf of Arash Roshanineshat <<a href="mailto:arashroshani92@gmail.com" target="_blank" class="">arashroshani92@gmail.com</a>><br class="">
Sent: Thursday, 29 June 2017 7:34 AM<br class="">
To: <a href="mailto:difx-users@listmgr.nrao.edu" target="_blank" class="">difx-users@listmgr.nrao.edu</a><br class="">
Cc: <a href="mailto:arash.roshanineshat@cfa.harvard.edu" target="_blank" class="">arash.roshanineshat@cfa.harvar<wbr class="">d.edu</a><br class="">
Subject: [Difx-users] mpicorrdifx cannot be loaded correctly on more than a single node<br class="">
<br class="">
Hi,<br class="">
<br class="">
I could install difx but it can only be run on a single node cluster.<br class="">
<br class="">
The *.machines and *.threads files are attached to this email.<br class="">
<br class="">
Openmpi is installed on all nodes and difx folder and data folder is<br class="">
shared among the clusters using NFS filesystem. Difx works perfectly<br class="">
with correct output on single machines.<br class="">
<br class="">
executing "startdifx -v -f e17d05-Sm-Sr_1000.input" returns the<br class="">
following error:<br class="">
<br class="">
DIFX_MACHINES -> /home/arash/Shared_Examples/Ex<wbr class="">ample2/C.txt<br class="">
Found modules:<br class="">
Executing: mpirun -np 6 --hostfile<br class="">
/home/arash/Shared_Examples/Ex<wbr class="">ample2/e17d05-Sm-Sr_1000.machi<wbr class="">nes --mca<br class="">
mpi_yield_when_idle 1 --mca rmaps seq runmpifxcorr.DiFX-2.5<br class="">
/home/arash/Shared_Examples/Ex<wbr class="">ample2/e17d05-Sm-Sr_1000.input<br class="">
------------------------------<wbr class="">------------------------------<wbr class="">--------------<br class="">
While computing bindings, we found no available cpus on<br class="">
the following node:<br class="">
<br class="">
Node: fringes-difx0<br class="">
<br class="">
Please check your allocation.<br class="">
------------------------------<wbr class="">------------------------------<wbr class="">--------------<br class="">
Elapsed time (s) = 0.50417590141<br class="">
<br class="">
and executing<br class="">
<br class="">
$ mpirun -np 6 --hostfile<br class="">
/home/arash/Shared_Examples/Ex<wbr class="">ample2/e17d05-Sm-Sr_1000.machi<wbr class="">nes<br class="">
/home/arash/difx/bin/mpifxcorr<br class="">
/home/arash/Shared_Examples/Ex<wbr class="">ample2/e17d05-Sm-Sr_1000.input<br class="">
<br class="">
seems to be working but by observing the cpu usage, I see only 6 cpus<br class="">
involving "5 in fringes-difx0 and 1 in fringes-difx1". I was expecting<br class="">
it to use the number of cpus equal to the number in "*.threads" file.<br class="">
How can I solve this issue?<br class="">
<br class="">
the specification of the cluster is Socket=2, Core per Socket=10 and<br class="">
Threads per core=2.<br class="">
<br class="">
Best Regards<br class="">
<br class="">
Arash Roshanineshat<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</blockquote>
<br class="">
</div></div><br class="">______________________________<wbr class="">_________________<br class="">
Difx-users mailing list<br class="">
<a href="mailto:Difx-users@listmgr.nrao.edu" class="">Difx-users@listmgr.nrao.edu</a><br class="">
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank" class="">https://listmgr.nrao.edu/<wbr class="">mailman/listinfo/difx-users</a><br class="">
<br class=""></blockquote></div><br class=""><br clear="all" class=""><br class="">-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" style="font-size:12.8000001907349px" class=""><div dir="ltr" style="font-size:12.8px" class=""><div dir="ltr" style="font-size:12.8px" class=""><div dir="ltr" style="font-size:12.8px" class=""><div dir="ltr" style="font-size:12.8px" class="">!=============================================================!<br class="">Dr. Adam Deller </div><div dir="ltr" style="font-size:12.8px" class="">ARC Future Fellow, Senior Lecturer</div><div style="font-size:12.8px" class="">Centre for Astrophysics & Supercomputing </div><div dir="ltr" style="font-size:12.8px" class="">Swinburne University of Technology <br class="">John St, Hawthorn VIC 3122 Australia</div><div style="font-size:12.8px" class="">phone: +61 3 9214 5307</div><div style="font-size:12.8px" class="">fax: +61 3 9214 8797</div><div style="font-size:12.8px" class=""><br class=""></div><div style="font-size:12.8px" class="">office days (usually): Mon-Thu<br class="">!=============================================================!</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>
_______________________________________________<br class="">Difx-users mailing list<br class=""><a href="mailto:Difx-users@listmgr.nrao.edu" class="">Difx-users@listmgr.nrao.edu</a><br class="">https://listmgr.nrao.edu/mailman/listinfo/difx-users<br class=""></div></blockquote></div><br class=""></body></html>