[Difx-users] VDIF Benchmarking {External}

[Walter Brisken]

Hi Paul,

In my experience, a good fraction of MPI issues (and I think that is what 
we are looking at here) are related to routing.  I would make sure that 
each computer in your cluster can send TCP/UDP packets to each other on 
the correct interface.  After that there are two useful tests that can 
be easily performed:

1. replace all host names with "localhost" and make sure you can proceed 
running multiple processes all on one machine.  If this fails, then it is 
unlikely a routing issue and you can proceed to look at the DiFX-specific 
issues.

2. use the "mpispeed" program that comes with mpifxcorr.  Run it on an 
even number of hosts.  This will measure the (RDMA) transfer speed from 
the first to second host, from the third to the fourth, ... .   If this 
succeeds, then routing is almost certainly working as expected.  If it 
runs but you see lower than expected transfer speeds, then it is likely 
routing over an incorrect path.

I hope this helps,

Walter

-------------------------
Walter Brisken
NRAO
Deputy Assistant Director for VLBA Development
(505)-234-5912 (cell)
(575)-835-7133 (office; not useful during COVID times)

On Wed, 1 Dec 2021, Paul Harrison via Difx-users wrote:

> Hi,
> Some more progress on this - it was true that jumbo frames were not enabled everywhere however, even after
> correcting this, mpifxcorr was still just sitting there - so I decided to look with the debugger and then
> realised that the code was actually waiting in a TCP reading loop - so then I added UDP_MTU lines to my .vsd
> file, but have struggled to get a correct value that will not result in the Datastream complaining that is
> receiving the wrong amount of data - I noticed this line (90) in mk5.cpp that looks suspiciously wrong on
> several points
> 
>     udpsize = abs(tcpwindowsizebytes/1024)-20-2*4-sizeof(long long); // IP header is 20 bytes, UDP is 4x2
> bytes
> 
> I even tried gaming the TCP widow size to make this expression equal the UDP MTU that I know is being sent,
> but there must be another mistake elsewhere that at least partially compensates for the above as I am left
> with 
> 
> /share/nas/pah/difx_test/net/vdiftest.inputDataStream 7: Expected 40928 bytes, got 8224bytes. Quiting
> 
> So I am rather stuck as I think that I have explored the full range of plausible values that the MTU could be
> set to.
> 
> In looking at the code, I noticed that there are a whole range of DataStream subclasses, and perhaps the
> MK5DataStream that I seem to be using at the moment is not the best for eVLBI - is another one of these that
> is recommended, and how do you select different ones in the .vsd?
> 
> Cheers,
> Paul.
>
>       On 2021-11 -30, at 01:06, Phillips, Chris (S&A, Marsfield) via Difx-users
>       <difx-users at listmgr.nrao.edu> wrote:
> 
> Hi Paul,
>  
> I would *highly* recommend you initially test vlbi_fake is producing the VDIF data you expect. Try
> running it independently with some sort of VDIF (or generic) UDP receiver to test. I have some of my own
> code you can use if you don?t have anything yourself.
>  
> A few other suggestions/comments:
>  
>  *  Avoid 5000 byte VDIF frames. That is some hang up from Mark5B 10K frames I think and has no purpose.
>     Something closer to 8k (e.g. 8000 pr 8192 + header) makes more sense. Try ?9000?.
>  *  Vlbi_fake chooses a frame size which makes sense based on the data rate AND networking parameters,
>     the 5000 probably include VDIF header and maybe network headers (I forget). You do need to setup.
>     DIFX to match the frame size vlbi_fake uses, so some experimentation may ne needed
>  *  Are you sure ?Jumbo frames? are enabled (ie 9000 byte MTU enabled everywhere)
>  *  Make sure the ?threads? usage matches what you expect
>  
> Cheers
> Chris
> 
> 
> 
>