[Difx-users] running the example datasets

Wed Aug 5 19:16:38 EDT 2020

Hi Greg,

If you’re interested in trying to learn DIFX and contribute a little to the collaboration, there is a small task you could do.

I’m experimenting with some synthetic data with the idea people could generate their own data in-situ and then run DIFX on that data set. This is not ready for testing, but in a week or two if you were willing I could send you some instructions on what is needed and see if we can get it running. Any feedback on the process from you would help me decide if it was a useful approach.

Regards
Chris

On 6 Aug 2020, at 09:00, Adam Deller via Difx-users <difx-users at listmgr.nrao.edu<mailto:difx-users at listmgr.nrao.edu>> wrote:

Hi Greg,

Thanks for getting involved! This is very timely as the ADACS team at Swinburne will also be putting together some automated testing in the coming months, so this is the ideal opportunity to get this (long desired) goal sorted. I'd be very happy to work with you to help document your efforts.

diffDiFX simply steps through the visibility records one at a time, subtracts the test record from the reference record, and checks the result (both the raw difference, and the difference normalised by the magnitude of the reference record.)  So it can only work for correlations that are set up identically - same number of telescopes, exact same start time, same delay model, etc. It prints out any records that differ by too much, and it reports some statistics at the end, as you saw.  So it is really just a very blunt tool; good to reporting a "pass" but not much use for digging in to a failure.

Due to numerical precision effects, differences in the vector processing library, etc, low-level variations are expected and acceptable. Testing across different architectures across the years showed that a mean absolute percentage difference (which is < |difference| / |reference|>, so the average of the absolute value of the fractional difference across all records expressed as a percentage) of around 10^-5-10^-6 was normal, which is pretty reasonable when we consider that 32 bit floats are being used to house the visibilities.

One "gotcha" is that there was a bug fixed late last year which affected the calculation of data weights when FFT buffering is used.  (If you don't know what that is, it's not super important - it is a performance tuning parameter in DiFX).  The description is at this difx-developers thread: https://listmgr.nrao.edu/mailman/private/difx-developers/2019-August/002270.html.  The upshot is that the reference dataset was produced before this bug was fixed, and hence contains a small error.  We really need to regenerate these reference datasets to remove this source of difference.  It is on my to-do list as part of the ADACS work.

You should be able to run the tests on a cluster size of 1 machine.  You just need to run all the different processes (manager, datastream, core) on the same node.  If you're having trouble setting that up, let me know.

Cheers,
Adam

On Thu, 6 Aug 2020 at 03:07, Greg Lindahl via Difx-users <difx-users at listmgr.nrao.edu<mailto:difx-users at listmgr.nrao.edu>> wrote:
I'm new to DiFX, and recently took a whack at running the example datasets (which Adam recently mentioned.) My goal was to prove that I'd built DiFX correctly, and was getting correct answers. I'd love to do this as an automated test on my correlation cluster.

It wasn't too hard building DiFX and following the instructions in the rdv70 tarball. After running diffDiFX, there didn't seem to be any clues for what a good answer was.

This output from diffDiFX looked a bit alarming:

> the mean percentage absolute difference is 22.00246523

After consulting with CK Chan, he suggested that the rdv70 example output in the tarball was generated using Calc9, and I was using Calc 11. The "running DiFX" instructions on the wiki were good enough that I was able to figure out how to use the .im file in the tarball (which, indeed, was CALC VERSION 1 and not CALC VERSION 11) and the "mean percentage absolute difference" dropped to 0.0015. Which I still don't know is correct in an absolute sense.

Another thing that surprised me was the minimum cluster size needed to run this example.

This is all probably in the "we already knew the documentation was a bit sparse" category. I'd be happy to volunteer to do some of the writing/test the instructions, but I'll need some clues about diffDiFX's output.

--
Greg Lindahl
Software Architect, Event Horizon Telescope
Smithsonian Astrophysical Observatory
60 Garden Street | MS 66 | Cambridge, MA 02138
_______________________________________________
Difx-users mailing list
Difx-users at listmgr.nrao.edu<mailto:Difx-users at listmgr.nrao.edu>
https://listmgr.nrao.edu/mailman/listinfo/difx-users

--
!=============================================================!
A/Prof. Adam Deller
ARC Future Fellow
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797

office days (usually): Mon-Thu
!=============================================================!
_______________________________________________
Difx-users mailing list
Difx-users at listmgr.nrao.edu<mailto:Difx-users at listmgr.nrao.edu>
https://listmgr.nrao.edu/mailman/listinfo/difx-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20200805/f3d9b88e/attachment-0001.html>