[Difx-users] Generic FFT in DiFX / DiFX on RPI

[Richard Dodson]

Nice!

   You might recall that I did similar tests with a ARM cluster using these
generic routines. That was a few 100s watts system with 48 cores (4coresx12
cards). That could  be a low power correlator somewhere (MRO site?).

  I'll check those routines when I am back in Perth. We are finishing up
another winter trip to KASI, to escape the Australian heat :)

      Richard

On 31 Dec 2015 06:15, "Adam Deller" <deller at astron.nl> wrote:
>
> Hi everyone,
>
> As a fun little holiday project I decided to get DiFX compiled and
running on my raspberry pi.  That was mostly pretty easy, except that it
appears there was a bug in the allocation of the buffers for the generic
(FFTW) FFT routines in mpifxcorr (it created an FFT which was twice as long
as it should have been).  IPP wasn't an option, for obvious reasons.  The
bug was a little hard to track down, because it turns out valgrind is known
to give spurious errors about memcpy on the raspberry pi, which threw me
off the track because it was indeed the memcpy in the generic FFT routine
where the segfault occurred.  I have checked in an update to trunk
mpifxcorr.  Perhaps one of either Richard or Chris (who are the people who
did the majority of the generic implementation, if I remember right) might
like to check to see that they agree.  I left the 2D FFT untouched since I
don't know about its usage, but I suspect that might need to be changed in
the same way in architecture.h.in as my change for the 1D FFTs.
>
> There remains a problem causing a segfault somewhere in the pcal
handling, but unlike the FFT problem this doesn't give any valgrind errors
on x86_64 architectures, so I haven't been able to track that down yet.
And I could bypass it by disabling pcal, so I did.
>
> For those who are interested, an old raspberry PI B+ can correlate an
aggregate input bandwidth of 0.05 MHz in real time (while it is also doing
all the FxManager and DataStream tasks), which is 600x slower than a single
Intel Xeon E5-2620 core @ 2.40GHz.  That's a worst-case comparison, since
that single Intel core wasn't doing anything else.  Even burdened so, the
raspberry Pi is still faster than the very first VLBI correlator, which ran
on an IBM 360 costing upwards of 2 million of todays dollars. You'd only
need something approaching 100,000 raspberry pis for a system capable of
correlating the VLBA @ 2 Gbps (plus, I suppose, a pretty expensive switch).
>
> For anyone who wishes to also run very slow correlations on $20 hardware,
here is the recipe for getting DiFX compiled on a vanilla raspbian install:
>
> svn co https://svn.atnf.csiro.au/difx/master_tags/DiFX-2.4 --username
YourUsername
> # Edit DiFX-2.4/setup.bash appropriately, add it to ~/.bashrc, and source
it
> # Create and chmod runmpifxcorr.DiFX-2.4.1, and added the directory
containing it to $PATH
> sudo apt-get install subversion
> sudo apt-get install openmpi-bin
> sudo apt-get install openmpi-dev
> sudo apt-get install autotools
> sudo apt-get install autotools-dev
> sudo apt-get install autoconf
> sudo apt-get install libtool
> sudo apt-get install gfortran
> sudo apt-get install expat
> sudo apt-get install libexpat1-dev
> sudo apt-get install  fftw3-dev
> sudo apt-get install bison
> sudo apt-get install flex
> cd DiFX-2.4 ; ./install-difx --noipp
> sudo rpcbind
> sudo $DIFXROOT/bin/startCalcServer
> # I used a cutdown version of the rdv70 dataset to test, I can send the
v2d file on request
> vex2difx pi.v2d
> calcif2 pi_1.calc
> # Manually generated pi_1.threads and pi_1.machines, which I can also
send on request
> startdifx -n pi_1.input
>
> Cheers,
> Adam
>
> --
> !=============================================================!
> Dr. Adam Deller
> Ph  +31 521595785 / Fax +31 521595101
> Staff Astronomer, Astronomy Group
> ASTRON, Oude Hoogeveensedijk 4
> 7991 PD Dwingeloo, The Netherlands
> !=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20160101/b7194856/attachment.html>