<div dir="ltr">Hi Stuart,<div><br></div><div>So I'm assuming that job2 does actually start and something gets written to the difxlog, then it pauses until job1 finishes, and then it fires up and runs to completion? If that is the case, can you post the job2 difxlog as it stands during the "pause" phase? That might give a clue as to what it is waiting for.</div><div><br></div><div>Also is this file-based correlation?</div><div><br></div><div>Cheers,</div><div>Adam</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 20 April 2017 at 12:37, Stuart Weston <span dir="ltr"><<a href="mailto:stuart.weston@aut.ac.nz" target="_blank">stuart.weston@aut.ac.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-NZ" link="#0563C1" vlink="#954F72">
<div class="m_9157603122673682319WordSection1">
<p class="MsoNormal">I have two head nodes, each head node has 6 workers.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I split the job up into two groups of files, the idea being Head-1 does scans/files 1-6 and Head-2 does scans/files 7-11.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I create two separate input files with different file lists etc. Also two separate thread and machine files appropriate to the two different groups of ip addresses.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">head-1, worker-1-1, worker-1-2 … worker-1-6<u></u><u></u></p>
<p class="MsoNormal">head-2, worker-2-1 ….. worker-2-6<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">So set two jobs running in parallel<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Head-1 > mpirun -machinefile machines-1 -np 5 mpifxcorr hw03_1.input<u></u><u></u></p>
<p class="MsoNormal">Head-2 > mpirun -machinefile machines-2 -np 5 mpifxcorr hw03_2.input<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Now all machines are in the same subnet. I am guessing some communication is going on as Head-2 seem’s to wait while Head-1 processes files 1-6, once Head -1 has finished Head-2 gets busy doing files 7-11.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Is there any way to have Head-1 and Head-2 running at the same time ? ie Head-2 doesn’t wait for Head-1 to finish !<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Stuart Weston Bsc (Hons), MPhil (Hons), MInstP</span><span style="color:black"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Mobile: 021 713062</span><span style="color:black"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Skype: stuart.d.weston<br>
Email: <a href="mailto:stuart.weston@aut.ac.nz" target="_blank"><span style="color:blue">stuart.weston@aut.ac.nz</span></a></span><span style="color:black"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><a href="http://www.atnf.csiro.au/people/Stuart.Weston/index.html" target="_blank"><span style="color:blue">http://www.atnf.csiro.au/<wbr>people/Stuart.Weston/index.<wbr>html</span></a><br>
<br>
Software Engineer<br>
Institute for Radio Astronomy & Space Research (IRASR)<br>
School of Computing & Mathematical Sciences<br>
Faculty of Creative Technologies<br>
Auckland University of Technology, New Zealand.</span><span style="color:black"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><a href="http://www.irasr.aut.ac.nz/" target="_blank"><span style="color:blue">http://www.irasr.aut.ac.nz/</span></a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:black"><img border="0" width="622" height="245" id="m_9157603122673682319Picture_x0020_1" src="cid:image003.jpg@01D2B9E3.B0F92B40" alt="NewIRASRLogo"></span><span><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/<wbr>mailman/listinfo/difx-users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8000001907349px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px">!=============================================================!<br>Dr. Adam Deller </div><div dir="ltr" style="font-size:12.8px">ARC Future Fellow, Senior Lecturer</div><div style="font-size:12.8px">Centre for Astrophysics & Supercomputing </div><div dir="ltr" style="font-size:12.8px">Swinburne University of Technology <br>John St, Hawthorn VIC 3122 Australia</div><div style="font-size:12.8px">phone: +61 3 9214 5307</div><div style="font-size:12.8px">fax: +61 3 9214 8797</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">office days (usually): Mon-Thu<br>!=============================================================!</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>