[Difx-users] DiFX File Storage - Object Storage

[Stuart Weston]

Hi All,

I am still working away on DiFX in the Cloud.

I have tried ext3/4 and RAID0 in cloud block storage and it seems cloud I/O is a (as expected) limiting factor on correlation speedup.

I have run a LBA session (6 antennas, 15 baselines) in the cloud and got upto 0.8 speedup using block storage in a RAID0 (one RAID0 for each antenna on a different Hypervisor).

I have no first hand experience with object storage, but I am being told by the Cloud Provider (Catalyst IT) here that they are faster than block storage. They tell me that Amazon Web Services (AWS) is using Object Storage, it has API's (but only in Python) and is S3 compliant.

https://www.catalyst.net.nz/catalyst-cloud/iaas/object-storage

Catalyst are asking me how easy it would be to build Object Storage addressing into DiFX ? I am thinking this is very much a non-trivial task and would need changes to many file access code programs for each data format.

We are also looking at using "Ansible" (https://www.ansible.com/cloud) to break a job out and create a head and worker nodes on each hypervisor, so each exp.input gets processed in a different group where a group consists off Head + N*Workers. "Ansible" will dynamically build the VM's (Head + N*Workers groups, one group per hypervisior) submit the mpirun job on each head (see attached for the tree structure of VM's). Once finished break it all down so you only have the VM's up and running for how long the job takes to run.

We haven't used "Ansible" in anger yet on an actual correlation, so far just to test provisioning VM's taking the input file and the expresso cluster definition file. We still have physical limits ie 15 hypervisors so we can create entries in the cluster definition file taking that limit into consideration. We then provision the VM's required on-demand. I hope by the end of this week to have used "Ansible" in anger.
Stuart Weston
Mobile: 021 713062
Skype: stuart.d.weston
Email:  stuart.weston at aut.ac.nz<mailto:stuart.weston at aut.ac.nz>
http://www.atnf.csiro.au/people/Stuart.Weston/index.html

Software Engineer
Institute for Radio Astronomy & Space Research (IRASR)
School of Computing & Mathematical Sciences
Faculty of Creative Technologies
Auckland University of Technology, New Zealand.
http://www.irasr.aut.ac.nz/

[NewIRASRLogo]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20170111/25a14b20/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 11334 bytes
Desc: image002.jpg
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20170111/25a14b20/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: master-head-worker.pdf
Type: application/pdf
Size: 22111 bytes
Desc: master-head-worker.pdf
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20170111/25a14b20/attachment-0001.pdf>