[Difx-users] Generic FFT in DiFX / DiFX on RPI

[Adam Deller]

Hi Chris,

That's funny, I was using just one core thread and it was still segfaulting
in pcal (but only on the RPI, not x86_64).

Let me know when you merge and I will test it.

With diffDiFX, the difference is usually at the 1e-5-6 level between x86_64
architectures (so I think I usually have the threshold at a few times
1e-5), but I have seen it be more the 1e-4 level for one specific Intel
chipset vs the rest.  A few times 1e-4 is getting pretty large.  As long as
it is zero mean though, it shouldn't really matter.

Cheers,
Adam

On Wed, Jan 6, 2016 at 5:43 AM, <Chris.Phillips at csiro.au> wrote:

> Hi Adam
>
> I think your problem with Pcal and Generic mpifxcorr is that FFTW is not
> thread safe for anything other than plan execution. I had the same problem
> with the IPP9 branch with more than 1 Core thread. I have added some
> mutexes and it runs fine now.
>
> I will merge my changes soon (end of week?) and it would be good to see if
> you can run it then.
>
> Note I have a Raspberry Pi2 - it would be interesting to see how much
> faster it is.
>
> Cheers
> Chris
>
>
>
> On 31 Dec 2015, at 8:14 am, Adam Deller <deller at astron.nl> wrote:
>
> Hi everyone,
>
> As a fun little holiday project I decided to get DiFX compiled and running
> on my raspberry pi.  That was mostly pretty easy, except that it appears
> there was a bug in the allocation of the buffers for the generic (FFTW) FFT
> routines in mpifxcorr (it created an FFT which was twice as long as it
> should have been).  IPP wasn't an option, for obvious reasons.  The bug was
> a little hard to track down, because it turns out valgrind is known to give
> spurious errors about memcpy on the raspberry pi, which threw me off the
> track because it was indeed the memcpy in the generic FFT routine where the
> segfault occurred.  I have checked in an update to trunk mpifxcorr.
> Perhaps one of either Richard or Chris (who are the people who did the
> majority of the generic implementation, if I remember right) might like to
> check to see that they agree.  I left the 2D FFT untouched since I don't
> know about its usage, but I suspect that might need to be changed in the
> same way in architecture.h.in as my change for the 1D FFTs.
>
> There remains a problem causing a segfault somewhere in the pcal handling,
> but unlike the FFT problem this doesn't give any valgrind errors on x86_64
> architectures, so I haven't been able to track that down yet.  And I could
> bypass it by disabling pcal, so I did.
>
> For those who are interested, an old raspberry PI B+ can correlate an
> aggregate input bandwidth of 0.05 MHz in real time (while it is also doing
> all the FxManager and DataStream tasks), which is 600x slower than a single
> Intel Xeon E5-2620 core @ 2.40GHz.  That's a worst-case comparison, since
> that single Intel core wasn't doing anything else.  Even burdened so, the
> raspberry Pi is still faster than the very first VLBI correlator, which ran
> on an IBM 360 costing upwards of 2 million of todays dollars. You'd only
> need something approaching 100,000 raspberry pis for a system capable of
> correlating the VLBA @ 2 Gbps (plus, I suppose, a pretty expensive switch).
>
> For anyone who wishes to also run very slow correlations on $20 hardware,
> here is the recipe for getting DiFX compiled on a vanilla raspbian install:
>
> svn co https://svn.atnf.csiro.au/difx/master_tags/DiFX-2.4 --username
> YourUsername
> # Edit DiFX-2.4/setup.bash appropriately, add it to ~/.bashrc, and source
> it
> # Create and chmod runmpifxcorr.DiFX-2.4.1, and added the directory
> containing it to $PATH
> sudo apt-get install subversion
> sudo apt-get install openmpi-bin
> sudo apt-get install openmpi-dev
> sudo apt-get install autotools
> sudo apt-get install autotools-dev
> sudo apt-get install autoconf
> sudo apt-get install libtool
> sudo apt-get install gfortran
> sudo apt-get install expat
> sudo apt-get install libexpat1-dev
> sudo apt-get install  fftw3-dev
> sudo apt-get install bison
> sudo apt-get install flex
> cd DiFX-2.4 ; ./install-difx --noipp
> sudo rpcbind
> sudo $DIFXROOT/bin/startCalcServer
> # I used a cutdown version of the rdv70 dataset to test, I can send the
> v2d file on request
> vex2difx pi.v2d
> calcif2 pi_1.calc
> # Manually generated pi_1.threads and pi_1.machines, which I can also send
> on request
> startdifx -n pi_1.input
>
> Cheers,
> Adam
>
> --
> !=============================================================!
> Dr. Adam Deller
> Ph  +31 521595785 / Fax +31 521595101
> Staff Astronomer, Astronomy Group
> ASTRON, Oude Hoogeveensedijk 4
> 7991 PD Dwingeloo, The Netherlands
> !=============================================================!
>
>
>


-- 
!=============================================================!
Dr. Adam Deller
Ph  +31 521595785 / Fax +31 521595101
Staff Astronomer, Astronomy Group
ASTRON, Oude Hoogeveensedijk 4
7991 PD Dwingeloo, The Netherlands
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20160106/5aa75042/attachment.html>