<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-AU;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-AU" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Apologies if this comes off as “vendor advertising”, but Intel has been working on this exact topic.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText" style="margin-left:36.0pt"><span lang="EN-US">We have published a solution brief here.
<a href="https://www.intel.com/content/www/us/en/architecture-and-technology/ethernet/hpn-with-ethernet-brief.html">
https://www.intel.com/content/www/us/en/architecture-and-technology/ethernet/hpn-with-ethernet-brief.html</a><o:p></o:p></span></p>
<p class="MsoPlainText" style="margin-left:36.0pt"><span lang="EN-US">And the software and tools stack here.
<a href="https://www.intel.com/content/www/us/en/collections/products/networking/high-performance-networking.html?s=Newest">
https://www.intel.com/content/www/us/en/collections/products/networking/high-performance-networking.html?s=Newest</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">If you would like more information, please contact me “off list”.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Regards, Peter Kerney.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Difx-users <difx-users-bounces@listmgr.nrao.edu>
<b>On Behalf Of </b>Adam Deller via Difx-users<br>
<b>Sent:</b> Tuesday, September 21, 2021 8:04 AM<br>
<b>To:</b> Walter Brisken <wbrisken@nrao.edu><br>
<b>Cc:</b> difxusers <difx-users@nrao.edu><br>
<b>Subject:</b> Re: [Difx-users] HPC using Ethernet vs Infiniband<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I've spoken to people about RoCE, but not sure if any of them have gone ahead and taken the plunge on it yet. I'll ask around to update myself.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Adam<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, 21 Sept 2021 at 05:28, Walter Brisken via Difx-users <<a href="mailto:difx-users@listmgr.nrao.edu">difx-users@listmgr.nrao.edu</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><br>
Hi DiFX Users,<br>
<br>
In the not so distant future we at VLBA may be be in the position to upgrade <br>
the network backbone of the VLBA correlator. Currently we have a 40 Gbps <br>
Infiniband system dating back about 10 years. At the time we installed that <br>
system, Infiniband showed clear advantages, likely driven by RDMA capability <br>
which offloads a significant amount of work from the CPU. Now it seems <br>
Ethernet has RoCE (RDMA over Converged Ethernet) which aims to do the same <br>
thing.<br>
<br>
1. Does anyone have experience with RoCE? If so, is this as easy to configure <br>
as the OpenMPI page suggests? Any drawbacks of using it?<br>
<br>
2. Has anyone else gone through this decision process recently? If so, any <br>
thoughts or advice?<br>
<br>
3. Has anyone run DiFX on an RoCE-based network?<br>
<br>
-Walter<br>
<br>
-------------------------<br>
Walter Brisken<br>
NRAO<br>
Deputy Assistant Director for VLBA Development<br>
(505)-234-5912 (cell)<br>
(575)-835-7133 (office; not useful during COVID times)<br>
<br>
_______________________________________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">!=============================================================!<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">A/Prof. Adam Deller <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">ARC Future Fellow<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Centre for Astrophysics & Supercomputing <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Swinburne University of Technology <br>
John St, Hawthorn VIC 3122 Australia<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">phone: +61 3 9214 5307<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">fax: +61 3 9214 8797<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">office days (usually): Mon-Thu<br>
!=============================================================!<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>