[Difx-users] HPC using Ethernet vs Infiniband

[Kerney, Peter]

Hi All,

Apologies if this comes off as “vendor advertising”, but Intel has been working on this exact topic.


We have published a solution brief here. https://www.intel.com/content/www/us/en/architecture-and-technology/ethernet/hpn-with-ethernet-brief.html

And the software and tools stack here. https://www.intel.com/content/www/us/en/collections/products/networking/high-performance-networking.html?s=Newest

If you would like more information, please contact me “off list”.

Regards, Peter Kerney.



From: Difx-users <difx-users-bounces at listmgr.nrao.edu> On Behalf Of Adam Deller via Difx-users
Sent: Tuesday, September 21, 2021 8:04 AM
To: Walter Brisken <wbrisken at nrao.edu>
Cc: difxusers <difx-users at nrao.edu>
Subject: Re: [Difx-users] HPC using Ethernet vs Infiniband

I've spoken to people about RoCE, but not sure if any of them have gone ahead and taken the plunge on it yet.  I'll ask around to update myself.

Cheers,
Adam



On Tue, 21 Sept 2021 at 05:28, Walter Brisken via Difx-users <difx-users at listmgr.nrao.edu<mailto:difx-users at listmgr.nrao.edu>> wrote:

Hi DiFX Users,

In the not so distant future we at VLBA may be be in the position to upgrade
the network backbone of the VLBA correlator.  Currently we have a 40 Gbps
Infiniband system dating back about 10 years.  At the time we installed that
system, Infiniband showed clear advantages, likely driven by RDMA capability
which offloads a significant amount of work from the CPU.  Now it seems
Ethernet has RoCE (RDMA over Converged Ethernet) which aims to do the same
thing.

1. Does anyone have experience with RoCE?  If so, is this as easy to configure
as the OpenMPI page suggests?  Any drawbacks of using it?

2. Has anyone else gone through this decision process recently?  If so, any
thoughts or advice?

3. Has anyone run DiFX on an RoCE-based network?

        -Walter

-------------------------
Walter Brisken
NRAO
Deputy Assistant Director for VLBA Development
(505)-234-5912 (cell)
(575)-835-7133 (office; not useful during COVID times)

_______________________________________________
Difx-users mailing list
Difx-users at listmgr.nrao.edu<mailto:Difx-users at listmgr.nrao.edu>
https://listmgr.nrao.edu/mailman/listinfo/difx-users


--
!=============================================================!
A/Prof. Adam Deller
ARC Future Fellow
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797

office days (usually): Mon-Thu
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20210921/6918ade3/attachment-0001.html>