[Difx-users] DiFX in the Cloud - Status

[Stuart Weston]

Dear DiFX users,

Since my remote presentation at the Users & Developers Meeting earlier this month we have progressed to having a working install of DiFX in a commercial cloud environment with the generous and kind assistance of Catalyst NZ Ltd.

I have started with a 2 antenna experiment we did last year as a fringe test between Wa and Hh in C-Band. 16x16MHz bands, 2bit Mk5B recording so 1Gbps record rate.

On our dedicated DiFX physical server (wark167 - 2 x Core i7 6 Cores Hyperthreaded, 64GB RAM, RAID0 for data) this takes ~ 38 mins to correlate with np=6 (see I hope attached fourfit plot file : wark167_no0006.pdf).

I was able to use our "tsunami" and "jive5ab" tools to copy data into the cloud storage. REANNZ our Research Network provider have provided a 1Gbps connection between the themselves and Catalyst. They are in the process of planning a 10Gbps link into Catalyst for other researchers who want to make use of Cloud Computing.

In the cloud VM (c1.c8r4 - 8 CPU's and 4GB RAM). I have split things into three separate volumes to try and get the best I/O:

/difx-data/ : This has the vex files, filelists etc, also the output goes here
/vlbi-data/hhwa03/hh  - Hart raw data files
/vlbi-data02/hhwa03/wa - Wark raw data files

Note it is easy to share these volumes with other instances.

Using the same data files and settings (it was a straight copy from the wark167 server, just changing paths to datafiles etc) with np=6 it takes ~ 51mins to correlate (see I hope attached fourfit plot file : lwcc_no0006.pdf). Granted this VM is not comparable to our server above but this was what Catalyst gave me to get started (again we don't want to upset their paying customers)

Over the coming weeks I will be working with Catalyst to try and improve those correlation times. We will also benchmark the same LBA session I presented our initial work with at the meeting and IVTW2016. They just have to do some housekeeping to give me > 10TB of volume storage and not upset their paying customers.

I am not sure if the fourfit plots attached will come through (perhaps some one can suggest a better way to share them). But I note slight differences between the two environments, for example:

Item        wark167              Cloud
SNR        253.1                   253.2
Int time   297.886               297.720
Amp        10.726                10.734
Phase       -88.4                  -87.5
......

Interesting the start times are slightly different ?

Start       133601.57            133602.10
End        134059.90            134059.90

The Phase Amplitude plots look the same

Catalyst are aware of their data storage charges being high, they are open to the idea of hosting 3rd party eqpt so we could put a large storage array in for ourselves. But they are also looking at bringing those charges down. They have some nice scripting tools that mean the head node could parse the VEX file and spin up as many worker instances as required (quickly !), something I will explore with them in due course. We can also explore having several head nodes and running several across sub-sets of the data files to get better turn around.

I hope this is of interest to the DiFX community.

Stuart Weston Bsc (Hons), MPhil (Hons), MInstP

Mobile: 021 713062

Skype: stuart.d.weston
Email:  stuart.weston at aut.ac.nz

http://www.atnf.csiro.au/people/Stuart.Weston/index.html

Software Engineer
Institute for Radio Astronomy & Space Research (IRASR)
School of Engineering, Computer and Mathematical Sciences
Faculty of Design and Creative Technologies
Auckland University of Technology, New Zealand.

http://www.irasr.aut.ac.nz/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20161123/3051740f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wark167_no00006.pdf
Type: application/pdf
Size: 103364 bytes
Desc: wark167_no00006.pdf
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20161123/3051740f/attachment-0002.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lwcc_no0006.pdf
Type: application/pdf
Size: 104632 bytes
Desc: lwcc_no0006.pdf
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20161123/3051740f/attachment-0003.pdf>