[Difx-users] Generic FFT in DiFX / DiFX on RPI

[Adam Deller]

Hi everyone,

As a fun little holiday project I decided to get DiFX compiled and running
on my raspberry pi.  That was mostly pretty easy, except that it appears
there was a bug in the allocation of the buffers for the generic (FFTW) FFT
routines in mpifxcorr (it created an FFT which was twice as long as it
should have been).  IPP wasn't an option, for obvious reasons.  The bug was
a little hard to track down, because it turns out valgrind is known to give
spurious errors about memcpy on the raspberry pi, which threw me off the
track because it was indeed the memcpy in the generic FFT routine where the
segfault occurred.  I have checked in an update to trunk mpifxcorr.
Perhaps one of either Richard or Chris (who are the people who did the
majority of the generic implementation, if I remember right) might like to
check to see that they agree.  I left the 2D FFT untouched since I don't
know about its usage, but I suspect that might need to be changed in the
same way in architecture.h.in as my change for the 1D FFTs.

There remains a problem causing a segfault somewhere in the pcal handling,
but unlike the FFT problem this doesn't give any valgrind errors on x86_64
architectures, so I haven't been able to track that down yet.  And I could
bypass it by disabling pcal, so I did.

For those who are interested, an old raspberry PI B+ can correlate an
aggregate input bandwidth of 0.05 MHz in real time (while it is also doing
all the FxManager and DataStream tasks), which is 600x slower than a single
Intel Xeon E5-2620 core @ 2.40GHz.  That's a worst-case comparison, since
that single Intel core wasn't doing anything else.  Even burdened so, the
raspberry Pi is still faster than the very first VLBI correlator, which ran
on an IBM 360 costing upwards of 2 million of todays dollars. You'd only
need something approaching 100,000 raspberry pis for a system capable of
correlating the VLBA @ 2 Gbps (plus, I suppose, a pretty expensive switch).

For anyone who wishes to also run very slow correlations on $20 hardware,
here is the recipe for getting DiFX compiled on a vanilla raspbian install:

svn co https://svn.atnf.csiro.au/difx/master_tags/DiFX-2.4 --username
YourUsername
# Edit DiFX-2.4/setup.bash appropriately, add it to ~/.bashrc, and source it
# Create and chmod runmpifxcorr.DiFX-2.4.1, and added the directory
containing it to $PATH
sudo apt-get install subversion
sudo apt-get install openmpi-bin
sudo apt-get install openmpi-dev
sudo apt-get install autotools
sudo apt-get install autotools-dev
sudo apt-get install autoconf
sudo apt-get install libtool
sudo apt-get install gfortran
sudo apt-get install expat
sudo apt-get install libexpat1-dev
sudo apt-get install  fftw3-dev
sudo apt-get install bison
sudo apt-get install flex
cd DiFX-2.4 ; ./install-difx --noipp
sudo rpcbind
sudo $DIFXROOT/bin/startCalcServer
# I used a cutdown version of the rdv70 dataset to test, I can send the v2d
file on request
vex2difx pi.v2d
calcif2 pi_1.calc
# Manually generated pi_1.threads and pi_1.machines, which I can also send
on request
startdifx -n pi_1.input

Cheers,
Adam

-- 
!=============================================================!
Dr. Adam Deller
Ph  +31 521595785 / Fax +31 521595101
Staff Astronomer, Astronomy Group
ASTRON, Oude Hoogeveensedijk 4
7991 PD Dwingeloo, The Netherlands
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20151230/5a02e81b/attachment.html>