[Difx-users] Generic FFT in DiFX / DiFX on RPI

[Chris.Phillips at csiro.au]

Hi Adam

I think your problem with Pcal and Generic mpifxcorr is that FFTW is not thread safe for anything other than plan execution. I had the same problem with the IPP9 branch with more than 1 Core thread. I have added some mutexes and it runs fine now.

I will merge my changes soon (end of week?) and it would be good to see if you can run it then.

Note I have a Raspberry Pi2 - it would be interesting to see how much faster it is.

Cheers
Chris



On 31 Dec 2015, at 8:14 am, Adam Deller <deller at astron.nl<mailto:deller at astron.nl>> wrote:

Hi everyone,

As a fun little holiday project I decided to get DiFX compiled and running on my raspberry pi.  That was mostly pretty easy, except that it appears there was a bug in the allocation of the buffers for the generic (FFTW) FFT routines in mpifxcorr (it created an FFT which was twice as long as it should have been).  IPP wasn't an option, for obvious reasons.  The bug was a little hard to track down, because it turns out valgrind is known to give spurious errors about memcpy on the raspberry pi, which threw me off the track because it was indeed the memcpy in the generic FFT routine where the segfault occurred.  I have checked in an update to trunk mpifxcorr.  Perhaps one of either Richard or Chris (who are the people who did the majority of the generic implementation, if I remember right) might like to check to see that they agree.  I left the 2D FFT untouched since I don't know about its usage, but I suspect that might need to be changed in the same way in architecture.h.in<http://architecture.h.in/> as my change for the 1D FFTs.

There remains a problem causing a segfault somewhere in the pcal handling, but unlike the FFT problem this doesn't give any valgrind errors on x86_64 architectures, so I haven't been able to track that down yet.  And I could bypass it by disabling pcal, so I did.

For those who are interested, an old raspberry PI B+ can correlate an aggregate input bandwidth of 0.05 MHz in real time (while it is also doing all the FxManager and DataStream tasks), which is 600x slower than a single Intel Xeon E5-2620 core @ 2.40GHz.  That's a worst-case comparison, since that single Intel core wasn't doing anything else.  Even burdened so, the raspberry Pi is still faster than the very first VLBI correlator, which ran on an IBM 360 costing upwards of 2 million of todays dollars. You'd only need something approaching 100,000 raspberry pis for a system capable of correlating the VLBA @ 2 Gbps (plus, I suppose, a pretty expensive switch).

For anyone who wishes to also run very slow correlations on $20 hardware, here is the recipe for getting DiFX compiled on a vanilla raspbian install:

svn co https://svn.atnf.csiro.au/difx/master_tags/DiFX-2.4 --username YourUsername
# Edit DiFX-2.4/setup.bash appropriately, add it to ~/.bashrc, and source it
# Create and chmod runmpifxcorr.DiFX-2.4.1, and added the directory containing it to $PATH
sudo apt-get install subversion
sudo apt-get install openmpi-bin
sudo apt-get install openmpi-dev
sudo apt-get install autotools
sudo apt-get install autotools-dev
sudo apt-get install autoconf
sudo apt-get install libtool
sudo apt-get install gfortran
sudo apt-get install expat
sudo apt-get install libexpat1-dev
sudo apt-get install  fftw3-dev
sudo apt-get install bison
sudo apt-get install flex
cd DiFX-2.4 ; ./install-difx --noipp
sudo rpcbind
sudo $DIFXROOT/bin/startCalcServer
# I used a cutdown version of the rdv70 dataset to test, I can send the v2d file on request
vex2difx pi.v2d
calcif2 pi_1.calc
# Manually generated pi_1.threads and pi_1.machines, which I can also send on request
startdifx -n pi_1.input

Cheers,
Adam

--
!=============================================================!
Dr. Adam Deller
Ph  +31 521595785 / Fax +31 521595101
Staff Astronomer, Astronomy Group
ASTRON, Oude Hoogeveensedijk 4
7991 PD Dwingeloo, The Netherlands
!=============================================================!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20160106/73165ee7/attachment.html>