[Difx-users] Some recent benchmarks

[Walter Brisken]

Hi DiFX Users,

Over the past few days I've had the chance to benchmark a few different CPU 
types.  The test was 5 seconds of VLBA data: 10 stations, Mark5B format, 2048 
Mbps.  All machines being tested had separate datastream nodes running in 
"fake" mode, with essentially unlimited throughput. 40 Gbps Infiniband fabric 
was used.  The top two boxes are new and were on loan for the DiFX tests 
(thanks Casey Law!).  The third is one of our new Mark6 units attached to our 
correlator.  The last three entries represent the compute nodes we have on the 
DiFX cluster in Socorro.  With one exception, each box tested was a dual-CPU 
setup.

I give two performance numbers for each tested CPU:

1 core: the bit-rate per VLBA antenna (Mbps) that one core of the stated CPU 
type could digest.  2048 divided by this number is the number of cores, 
assuming perfect scalability, required to process VLBA data in real-time.

1 box: the bit-rate per VLBA antenna (Mbps) that one server box can digest.  In 
each cases one "DiFX thread" was spawned for each physical core in the box.

And the results are...

CPU		1 core	1 box	Notes
---		------	-----	-----
2x Gold 6126	19.0	367	Excellent scalability w/ cores
2x Gold 5115	13.5	242	Probably limited by TDP
1x E5-2650v3	20.4	165	Unexpectedly good single-core performance
2x E5-2670v2	19.5	272	Fastest in DiFX cluster; terrible scalability
2x X5650	13.0	110	Medium fastest nodes in DiFX cluster
2x E5520	10.7	64.6	Slowest CPUs in DiFX cluster

The main conclusion I can draw from this is that the Thermal Design Power (TDP) 
is a very good predictor of whole-CPU performance for a given process (e.g., 
14nm or 22nm).  Above a certain clock*cores level, TDP per dollar may be the 
correct metric to use when selecting between CPU options for DiFX.  I'm not 
sure where the best value is.  I'd love to try something in the "Skylake-W" CPU 
line, especially Xeon W-2145.  That may be a good test of the TDP theory.

Some details on each mentioned CPU:

CPU		Cores	Clock	Process	TDP	Release
---		-----	-----	-------	---	-------
Gold 6126	12	2.6GHz	14nm	125W	Sep 2017
Gold 5115	10	2.4GHz	14nm	85W	Sep 2017
E5-2650v3	10	2.3GHz	22nm	105W	Sep 2014
E5-2670v2	10	2.5GHz	22nm	115W	Sep 2013
X5650		6	2.67GHz	32nm	95W	Mar 2010
E5520		4	2.27GHz	45nm	80W	Mar 2009

Xeon W-2145	8	3.7GHz	14nm	140W	Aug 2017

 	-Walter