[Difx-users] HPC using Ethernet vs Infiniband

[Walter Brisken]

Hi DiFX Users,

In the not so distant future we at VLBA may be be in the position to upgrade 
the network backbone of the VLBA correlator.  Currently we have a 40 Gbps 
Infiniband system dating back about 10 years.  At the time we installed that 
system, Infiniband showed clear advantages, likely driven by RDMA capability 
which offloads a significant amount of work from the CPU.  Now it seems 
Ethernet has RoCE (RDMA over Converged Ethernet) which aims to do the same 
thing.

1. Does anyone have experience with RoCE?  If so, is this as easy to configure 
as the OpenMPI page suggests?  Any drawbacks of using it?

2. Has anyone else gone through this decision process recently?  If so, any 
thoughts or advice?

3. Has anyone run DiFX on an RoCE-based network?

 	-Walter

-------------------------
Walter Brisken
NRAO
Deputy Assistant Director for VLBA Development
(505)-234-5912 (cell)
(575)-835-7133 (office; not useful during COVID times)