<div dir="ltr">Echoing Walter: this is an openmpi issue, not a DiFX issue per se. It may well be related to your specific openmpi version, see <a href="http://users.open-mpi.narkive.com/XhLEHyPF/ompi-users-openmpi-1-8-rmaps-seq-doesn-t-work">http://users.open-mpi.narkive.com/XhLEHyPF/ompi-users-openmpi-1-8-rmaps-seq-doesn-t-work</a>. Try upgrading (or downgrading) openmpi.</div><div class="gmail_extra"><br><div class="gmail_quote">On 10 April 2017 at 23:25, Walter Brisken <span dir="ltr"><<a href="mailto:wbrisken@lbo.us" target="_blank">wbrisken@lbo.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
There is a program called mpispeed which I find useful when diagnosing mpi-related problems. It comes with the mpifxcorr package. To run it, select an even number of hosts (I usually just use 2) and run:<br>
<br>
mpirun <options> mpispeed<br>
<br>
where <options> are whatever options (including either a .machines file or explicit list of hosts) you want to run. Its often easier to diagnose the issues without the extra complexity of the correlator. If successful, it will do 256 sends from the first host to the second.<br>
<br>
When there are mpi problems, the issue may be beyond the scope of this mailing list and you might do well using google or documentation at <a href="http://openmpi.org" rel="noreferrer" target="_blank">openmpi.org</a> to find the fix.<br>
<br>
-Walter<div><div class="h5"><br>
<br>
On Mon, 10 Apr 2017, Lupin Lin wrote:<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Dear Adam,<br>
<br>
Thanks for your response.<br>
I checked the verbose output, and obtained the following messages:<br>
<br>
Executing: difxlog g3_7038 /data/lupin/g3-7038/g3_7038.di<wbr>fxlog 4 44843 &<br>
Executing: mpirun -np 5 --hostfile /data/lupin/g3-7038/<a href="http://g3_7038.ma">g3_7038.ma</a><wbr>chines --mca mpi_yield_when_idle 1 --mca rmaps seq runmpifxcorr.trunk /data/lupin/g3-7038/<a href="http://g3_7038.in">g3_7038.in</a><wbr>put<br>
------------------------------<wbr>------------------------------<wbr>--------------<br>
Your job failed to map. Either no mapper was available, or none<br>
of the available mappers was able to perform the requested<br>
mapping operation. This can happen if you request a map type<br>
(e.g., loadbalance) and the corresponding mapper was not built.<br>
------------------------------<wbr>------------------------------<wbr>--------------<br>
Elapsed time (s) = 0.351403951645<br>
<br>
Then I compared with the mpirun command manually, and I found that the following command/description can be successfully performed.<br>
------------------<br>
mpirun -np 5 --hostfile /data/lupin/g3-7038/<a href="http://g3_7038.ma">g3_7038.ma</a><wbr>chines --mca mpi_yield_when_idle 1 mpifxcorr /data/lupin/g3-7038/<a href="http://g3_7038.in">g3_7038.in</a><wbr>put<br>
------------------<br>
<br>
However, I will meet the problem if I increase the parameters of “--mac rmaps" and "seq runmpifxcorr.trunk”.<br>
Should I avoid to use mapping operation?<br>
<br>
Sincerely,<br>
--<br>
Lupin Chun-Che Lin<br>
Supporting Scientist of GLT (GreenLand Telescope) project<br>
in Institute of Astronomy and Astrophysics, Academia Sinica,<br>
14F of Astronomy-Mathematics Building (Rm: 1405),<br>
National Taiwan University.<br>
No.1, Sec. 4, Roosevelt Rd, Taipei 10617, Taiwan.<br>
Tel: <a href="tel:%2B886-2-2366-5464" value="+886223665464" target="_blank">+886-2-2366-5464</a><br>
Fax: <a href="tel:%2B886-2-2367-7849" value="+886223677849" target="_blank">+886-2-2367-7849</a><br>
<br>
<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Adam Deller <<a href="mailto:adeller@astro.swin.edu.au" target="_blank">adeller@astro.swin.edu.au</a>> 於 2017年4月10日 下午6:45 寫道：<br>
<br>
Hi Lupin,<br>
<br>
Try startdifx -v -v to get verbose output, and compare the mpirun command to the one you're using manually.<br>
<br>
Cheers,<br>
Adam<br>
<br></div></div><span class="">
On 10 April 2017 at 19:00, Lupin Lin <<a href="mailto:lupin@asiaa.sinica.edu.tw" target="_blank">lupin@asiaa.sinica.edu.tw</a> <mailto:<a href="mailto:lupin@asiaa.sinica.edu.tw" target="_blank">lupin@asiaa.sinica.edu<wbr>.tw</a>>> wrote:<br>
To the experienced user of DiFX,<br>
<br>
I have re-installed the DiFX on the OS of "Scientific Linux release 6.8 (Carbon)”.<br>
However, when I performed the “startdifx”, I obtained the following error messages.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
</blockquote></blockquote></blockquote>
------------------------------<wbr>------------------------------<wbr>--------------<br>
Your job failed to map. Either no mapper was available, or none<br>
of the available mappers was able to perform the requested<br>
mapping operation. This can happen if you request a map type<br>
(e.g., loadbalance) and the corresponding mapper was not built.<br>
------------------------------<wbr>------------------------------<wbr>--------------<br>
<br>
But when I used the mpifxcorr, I did not get any error message and I can successfully obtain the result.<br>
For example, I can use the following command to replace the startdifx:<br>
mpirun --machinefile g3_7038.machines -np 11 mpifxcorr g3_7038.input<br>
<br>
So it seems the problem is not due to the connection to the mpi.<br>
<br>
Does anyone know the possible origin to cause this problem? And please give an indication to solve it.<br>
<br>
Thanks,<br>
--<br>
Lupin Chun-Che Lin<br>
Supporting Scientist of GLT (GreenLand Telescope) project<br>
in Institute of Astronomy and Astrophysics, Academia Sinica,<br>
14F of Astronomy-Mathematics Building (Rm: 1405),<br>
National Taiwan University.<br>
No.1, Sec. 4, Roosevelt Rd, Taipei 10617, Taiwan.<br></span>
Tel: <a href="tel:%2B886-2-2366-5464" value="+886223665464" target="_blank">+886-2-2366-5464</a> <tel:+886%202%202366%205464><br>
Fax: <a href="tel:%2B886-2-2367-7849" value="+886223677849" target="_blank">+886-2-2367-7849</a> <tel:+886%202%202367%207849><br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a> <mailto:<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nra<wbr>o.edu</a>><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/difx-users</a> <<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mail<wbr>man/listinfo/difx-users</a>><span class=""><br>
<br>
<br>
<br>
<br>
--<br>
!=============================<wbr>==============================<wbr>==!<br>
Dr. Adam Deller<br>
ARC Future Fellow, Senior Lecturer<br>
Centre for Astrophysics & Supercomputing<br>
Swinburne University of Technology<br>
John St, Hawthorn VIC 3122 Australia<br>
phone: <a href="tel:%2B61%203%209214%205307" value="+61392145307" target="_blank">+61 3 9214 5307</a><br>
fax: <a href="tel:%2B61%203%209214%208797" value="+61392148797" target="_blank">+61 3 9214 8797</a><br>
<br>
office days (usually): Mon-Thu<br>
!=============================<wbr>==============================<wbr>==!<br>
</span></blockquote>
<br>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
-------------------------<br>
Walter Brisken<br>
Director<br>
Long Baseline Observatory<br>
<a href="tel:%28505%29-234-5912" value="+15052345912" target="_blank">(505)-234-5912</a></font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8000001907349px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px">!=============================================================!<br>Dr. Adam Deller </div><div dir="ltr" style="font-size:12.8px">ARC Future Fellow, Senior Lecturer</div><div style="font-size:12.8px">Centre for Astrophysics & Supercomputing </div><div dir="ltr" style="font-size:12.8px">Swinburne University of Technology <br>John St, Hawthorn VIC 3122 Australia</div><div style="font-size:12.8px">phone: +61 3 9214 5307</div><div style="font-size:12.8px">fax: +61 3 9214 8797</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">office days (usually): Mon-Thu<br>!=============================================================!</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>