<div dir="ltr">Does <div><br></div><div>% ulimit -n 4096</div><div><br></div><div> help with this?</div><div> R</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 25, 2017 at 6:56 PM, 江悟 <span dir="ltr"><<a href="mailto:jiangwu@shao.ac.cn" target="_blank">jiangwu@shao.ac.cn</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>Hi all,<br><br>I think Bill Chen found the problem. Many difxlog threads are opened and kept there when running the correlation. So if the number of jobs is large, it is easy to reach the maximum of thread number limited by the operating system. <br>Is there any 'key' or setting to turn off the difxlog threads timely? <br><br>Cheers,<br>Wu<br><br><blockquote name="replyContent" style="padding-left:5px;margin-left:5px;border-left:#b6b6b6 2px solid;margin-right:0">-----原始邮件-----<br>
<b>发件人:</b> "Chen Bill" <<a href="mailto:billchen001@gmail.com" target="_blank">billchen001@gmail.com</a>><br>
<b>发送时间:</b> 2017年5月24日 星期三<br>
<b>收件人:</b> "江悟" <<a href="mailto:jiangwu@shao.ac.cn" target="_blank">jiangwu@shao.ac.cn</a>><br>
<b>抄送:</b> "Adam Deller" <<a href="mailto:adeller@astro.swin.edu.au" target="_blank">adeller@astro.swin.edu.au</a>>, difxusers <<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nrao.edu</a>><br>
<b>主题:</b> Re: [Difx-users] FxManager: Error in launching writethread!!<div><div class="h5"><br><br><div dir="ltr">Hi All,<div><br></div><div>I checked this issue, I think the issue is about too many of difxlog processes, in this case there are around 800 scans need to be process, and there will be 800 difxlog process running until all work done. The default Linux kernel only support 1024 process for one user.</div><div><br></div><div>I just wonder is it possible to close that difxlog process when one scan finished.</div><div>for this issue, we can increase the kernel parameter "noproc" to a big number, but I think the good way is to enhance the code to reduce the number of difxlog process.</div><div><br></div><div>Jiangwu, please correct me if I have mistake.</div><div><br></div><div> </div><div><br></div><div>Thanks,</div><div>Bill Chen</div><div><a href="http://www.simplehpc.com" target="_blank">www.simplehpc.com</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 23, 2017 at 1:27 PM, 江悟 <span dir="ltr"><<a href="mailto:jiangwu@shao.ac.cn" target="_blank">jiangwu@shao.ac.cn</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>Hi Chris and Adam,<br><br>Attached are the .v2d, .input, .vex files I used. I was using errormon2 when the error turned out, please check the last line of the errormon2.log. Unfortunately, when I used errormon this morning for re-correlating the same scans, no error reported. <br><br>Regards,<br>Wu<br><br><blockquote name="replyContent" style="padding-left:5px;margin-left:5px;border-left:#b6b6b6 2px solid;margin-right:0">-----原始邮件-----<br>
<b>发件人:</b> "Adam Deller" <<a href="mailto:adeller@astro.swin.edu.au" target="_blank">adeller@astro.swin.edu.au</a>><br>
<b>发送时间:</b> 2017年5月23日 星期二<br>
<b>收件人:</b> "江悟" <<a href="mailto:jiangwu@shao.ac.cn" target="_blank">jiangwu@shao.ac.cn</a>><br>
<b>抄送:</b> difxusers <<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nrao.edu</a>><br>
<b>主题:</b> Re: FxManager: Error in launching writethread!!<br><br><div dir="ltr">I don't recall seeing this before. Might it be an error with pthreads generally? Does it still occur if you're running a much smaller correlation?<div><br></div><div>Cheers,<br>Adam</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 23 May 2017 at 12:03, 江悟 <span dir="ltr"><<a href="mailto:jiangwu@shao.ac.cn" target="_blank">jiangwu@shao.ac.cn</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>Hi all,<br><br>Recently I ran difx (2.4.1 version) and came cross the error as following:<br>FxManager: Error in launching writethread!!<br><br>I was using 100 cores and 4 threads each, 1 seperate header node as set in the v2d file. And the visbufferlength was set to 80. The number of stations was 3, the raw data was put in a RAID with parallel file system, while the visbility output was collected and recorded on the local disk of the header noder. Other correlation parameters was,<br> tInt =<a href="tel:13%2010%2072" value="+61131072" target="_blank">0.131072</a><br> subintNS = 8192000<br> nChan = 512<br><br>I also checked the memory of the header node, the maximum occupied memory is less than 7%. So I don't know the reason of this error. Have you ever met this error before and could you please help to identify it?<br><br>Thanks a lot.<br><br>Best regards,<br>Wu Jiang<br><br><span></span><br><br><br></blockquote></div><br><br clear="all"><span class="m_-4562297447392207684HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_-4562297447392207684m_-8610413220803424249gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8000001907349px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px">!=============================<wbr>==============================<wbr>==!<br>Dr. Adam Deller </div><div dir="ltr" style="font-size:12.8px">ARC Future Fellow, Senior Lecturer</div><div style="font-size:12.8px">Centre for Astrophysics & Supercomputing </div><div dir="ltr" style="font-size:12.8px">Swinburne University of Technology <br>John St, Hawthorn VIC 3122 Australia</div><div style="font-size:12.8px">phone: <a href="tel:+61%203%209214%205307" value="+61392145307" target="_blank">+61 3 9214 5307</a></div><div style="font-size:12.8px">fax: <a href="tel:+61%203%209214%208797" value="+61392148797" target="_blank">+61 3 9214 8797</a></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">office days (usually): Mon-Thu<br>!=============================<wbr>==============================<wbr>==!</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</font></span></div>
</blockquote><br><span></span><br><br><br><br>______________________________<wbr>_________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/difx-users</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote><br><span></span><br><br><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">-------------------------<br>Dr Richard Dodson,<br>International Centre for Radio Astronomy Research<br>University of Western Australia<br>P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org" target="_blank">richard.dodson@icrar.org</a></div>
</div>