[Difx-users] [Difx-developers] INQUIRY: Processing correlation environment {External}
西 健宏
nishi-t96pc at mlit.go.jp
Wed Aug 19 01:27:29 EDT 2026
Dear Adam,
Sorry for late reply and thank you for your patience.
We would like to share the results of investigation.
We measured the processing time of DiFX to process 10 scans of the IS and IT baselines in a VGOS-INT-C session, changing parameters such as subintNS to change the block size. The processing time was obtained as the average time of five runs.
As a result, the processing speed of DiFX was improved with a smaller block size (0.1 GB) than our current setting (0.4 GB) or larger block size (tested up to 2 GB), surprisingly contrary to your suggestion.
For your reference, I have attached the file. We will keep the smaller block size for further investigation.
We also would like to ask about following three topics.
1. Server compatibility
2. Optimization of the number of processes and threads
3. Well-optimized processing time
1. Server compatibility
In this investigation, we used six servers for distributed processing.
However, when an additional server was added, the processing speed decreased despite the increase of usable core numbers for the processing. We suspect that this trouble is caused by the differences between servers but are not sure about the details.
If you have any insights into this issue, we would appreciate your comments.
Regarding the specifications of the six servers and added server, please refer to the attached file.
2. Optimization of the number of processes and threads
We determined the optimal number of processes and threads by running DiFX with different numbers of processes and threads.
Are there any other methods for finding the optimal settings?
In addition, we would like to know if there are any known general trends regarding the optimal settings of the number of servers, the number of CPU cores or the amount of memory. We plan to investigate process and thread numbers as the next thing to optimize.
3. Well-optimized processing time
If DiFX setting is well-optimized, how much time will it take to process 1-sec data?
It would be good indicator for us to determine how well our difx system is optimized.
Sorry for asking a lot of questions at once.
Thank you for your continued support and advice.
Best regards,
Takehiro
GSI
---------------------------------------------------------------
Takehiro Nishi
Space Geodesy Division
Geodetic Department
Geospatial Information Authority of Japan (GSI)
E-mail: nishi-t96pc at mlit.go.jp
---------------------------------------------------------------
-----Original Message-----
From: Adam Deller <adeller at astro.swin.edu.au>
Sent: Tuesday, July 14, 2026 4:47 PM
To: 西 健宏 <nishi-t96pc at mlit.go.jp>
Cc: 古居 晴菜 <furui-h96pd at mlit.go.jp>; 原 哲也 <hara-t96xs at mlit.go.jp>
Subject: 【警告：フリーメールからの発信です。】 Re: Re: [Difx-developers] INQUIRY: Processing correlation environment {External}
Hi Takehiro,
Thanks for the detailed info. I will make a quick response here but I also encourage you to post to the difx-users list where you'll have more chance of additional feedback: the DiFX read size is set both the combination of subintegration length, recording rate, and the parameters "dataBufferFactor" and "nDataSegments" in the v2d file. The size of one read is given by (subintegration length x recording rate) x (dataBufferFactor / nDataSegments). So if the recording rate was 1 Gbps and the subintegration duration was 0.08s, one subintegration would be 80 Mb / 8 = 10 MB. If dataBufferFactor is 32 the ring buffer for the datastream is 32 subintegrations long, but it is divided into nDatasegments chunks, so if nDataSegments is 8, then 32/8 = 4 subintegrations worth of data are read at a time, i.e 40MB. So check the subintNS value in the v2d file (or, if that is not being explicitly specified, look at the .input file and see what the subintegration time has been assigned to), and then you'll be able to calculate the read sizes. A quick check would then be to double or quadruple the subintNS and see if your read rate increases. In the other systems that I've typically seen/used it hasn't been necessary to increase the read sizes to >>10s of MB to get the best disk performance, but they were probably much smaller RAID systems, so you might benefit more from big reads than is typical.
Cheers,
Adam
On Tue, 14 Jul 2026 at 11:03, 西 健宏 <nishi-t96pc at mlit.go.jp <mailto:nishi-t96pc at mlit.go.jp> > wrote:
Dear Adam,
Thank you for your patients.
We checked with our system engineer and gathered information as below.
I hope this information will help us to find out the cause of our issue.
+ Correlator topology and network connectivity
- Star topology (One network switch was connected to manager node, datastream node, and core nodes)
- Datastream node is connected to the network switch by 100GbE.
- Manager node and core nodes are connected to the network switch by 10GbE.
- The amount of the traffic between manager node and core nodes are 1-2 Gbps per each.
- There are more rooms in CPU and memory use rate for each server.
+ Is this reading from commodity disk, Mark5/Mark6, or...?
- The server contains 3 RAID cards which each of them has RAID6 volumes made from 32 of 16TB NL-SAS HDDs.
- 3 RAID6 volumes are combined by LVM Stripe, and mounted on OS as 1.5 PB volume.
+ Hardware the datastream nodes are running on
- CPU is XeonSilver4510*2(Total:24Core)
- Memory is 64GB
+ The disk reading speed of datastream node
107374182400 bytes (107 GB, 100 GiB) copied, 152.365 s, 705 MB/s [root at gero ~]# dd if=/Extend/test of=/dev/null bs=64M iflag=direct
107374182400 bytes (107 GB, 100 GiB) copied, 53.5203 s, 2.0 GB/s [root at gero ~]# dd if=/Extend/test of=/dev/null bs=128M iflag=direct
107374182400 bytes (107 GB, 100 GiB) copied, 52.2909 s, 2.1 GB/s [root at gero ~]# dd if=/Extend/test of=/dev/null bs=256M iflag=direct
107374182400 bytes (107 GB, 100 GiB) copied, 34.0534 s, 3.2 GB/s [root at gero ~]# dd if=/Extend/test of=/dev/null bs=512M iflag=direct
107374182400 bytes (107 GB, 100 GiB) copied, 23.9807 s, 4.5 GB/s [root at gero Extend]# dd if=/Extend/test of=/dev/null bs=1G iflag=direct
107374182400 bytes (107 GB, 100 GiB) copied, 18.5585 s, 5.8 GB/s
- It highly depends on the size of blocks to read.
- We would like to know the size of blocks DiFX datastream processes reads since it remains unclear in our side.
+ The read rate being achieved
- per-antenna datarate: 8 Gbps for VGOS sessions and 2 Gbps for S/X sessions
- correlation speed-up factor: about 4 (please find the attachment for details)
Please let us know if there is any other information necessary to investigate the issue.
Thank you for your continued support.
Best regards,
Takehiro
GSI
---------------------------------------------------------------
Takehiro Nishi
Space Geodesy Division
Geodetic Department
Geospatial Information Authority of Japan (GSI)
E-mail: nishi-t96pc at mlit.go.jp <mailto:nishi-t96pc at mlit.go.jp>
---------------------------------------------------------------
-----Original Message-----
From: 西 健宏
Sent: Friday, July 10, 2026 11:02 AM
To: 'Adam Deller' <adeller at astro.swin.edu.au <mailto:adeller at astro.swin.edu.au> >
Cc: difx-users at listmgr.nrao.edu <mailto:difx-users at listmgr.nrao.edu> ; difx-developers at nrao.edu <mailto:difx-developers at nrao.edu> ; 古居 晴菜 <furui-h96pd at mlit.go.jp <mailto:furui-h96pd at mlit.go.jp> >; 原 哲也 <hara-t96xs at mlit.go.jp <mailto:hara-t96xs at mlit.go.jp> >
Subject: RE: 【警告：フリーメールからの発信です。】 Re: [Difx-developers] INQUIRY: Processing correlation environment {External}
Dear Adam,
Thank you for your kind reply.
Your comments and insights are very helpful to us.
Regarding your questions, we are currently checking with our system engineer, and it may take us a little more time to gather the necessary information.
We would appreciate your patience, and we will get back to you once we are ready to respond all the items on your check list.
Thank you for your continued support.
Best regards,
Takehiro
GSI
---------------------------------------------------------------
Takehiro Nishi
Space Geodesy Division
Geodetic Department
Geospatial Information Authority of Japan (GSI)
E-mail: nishi-t96pc at mlit.go.jp <mailto:nishi-t96pc at mlit.go.jp>
---------------------------------------------------------------
-----Original Message-----
From: Adam Deller <adeller at astro.swin.edu.au <mailto:adeller at astro.swin.edu.au> >
Sent: Monday, July 6, 2026 7:56 PM
To: 西 健宏 <nishi-t96pc at mlit.go.jp <mailto:nishi-t96pc at mlit.go.jp> >
Cc: difx-users at listmgr.nrao.edu <mailto:difx-users at listmgr.nrao.edu> ; difx-developers at nrao.edu <mailto:difx-developers at nrao.edu> ; 古居 晴菜 <furui-h96pd at mlit.go.jp <mailto:furui-h96pd at mlit.go.jp> >; 原 哲也 <hara-t96xs at mlit.go.jp <mailto:hara-t96xs at mlit.go.jp> >
Subject: 【警告：フリーメールからの発信です。】 Re: [Difx-developers] INQUIRY: Processing correlation environment {External}
Hi Takehiro,
Generally speaking, DiFX is pretty robust to having servers with different capabilities, since the time division multiplexing that is used will happily allocate fewer subintegrations to the slower nodes and more to the faster ones. If the throughput does not go up when you add more processing resources, you are not compute bound, but must instead be either:
1) input voltage read bound;
2) output visibility write bound; or
3) network I/O bound (either a) in the voltage send from the datastream reading nodes, or b) in the visibility subintegration assembly at the manager node)
It would be fairly rare for case 2) to be the problem, unless you have short integration times, many baselines, and many channels. If you have a decent network, and you are not doing very short subintegrations with lots of channels, it would also be unusual for case 3b) to be the problem. So that leaves the voltage data read or send - either getting it off disk in the first place, or away from the read node to the processing nodes.
Can you let us know what your correlator topology is? Is this reading from commodity disk, Mark5/Mark6, or...? What hardware are the datastream nodes running on, and what network connectivity do they have? And what is the read rate that is being achieved (i.e., what is the per-antenna datarate, and the correlation speed-up factor [correlate time divided by obs time for a given job]?)
Cheers,
Adam
On Mon, 6 Jul 2026 at 20:47, 西 健宏 via Difx-developers <difx-developers at listmgr.nrao.edu <mailto:difx-developers at listmgr.nrao.edu> <mailto:difx-developers at listmgr.nrao.edu <mailto:difx-developers at listmgr.nrao.edu> > > wrote:
Dear colleagues,
Hello.
My name is Takehiro Nishi, in the Geospatial Information Authority of Japan (GSI).
GSI is responsible for correlating some of the geodetic VLBI sessions of IVS.
I'm writing to inquire about processing correlation environment.
Currently, GSI is trying to improve the processing speed of "DiFX", since the processing time didn’t improve when we added three new servers for distributed processing.
We need to optimize our system, but we are not sure where the bottle neck is.
Therefore, I would appreciate it if you could inform us of your processing environment to help our investigation as below:
・the number of processes written in .machines files :
・the number of threads written in .threads files :
・the capacity of memory for each server :
・the total number of cores :
・the total number of processing servers :
・Are the number of cores and memory on the servers uniform, or are they varied?
:
Thank you for your cooperation.
Best regards,
Takehiro
GSI
---------------------------------------------------------------
Takehiro Nishi
Space Geodesy Division
Geodetic Department
Geospatial Information Authority of Japan (GSI)
E-mail: nishi-t96pc at mlit.go.jp <mailto:nishi-t96pc at mlit.go.jp> <mailto:nishi-t96pc at mlit.go.jp <mailto:nishi-t96pc at mlit.go.jp> >
---------------------------------------------------------------
_______________________________________________
Difx-developers mailing list
Difx-developers at listmgr.nrao.edu <mailto:Difx-developers at listmgr.nrao.edu> <mailto:Difx-developers at listmgr.nrao.edu <mailto:Difx-developers at listmgr.nrao.edu> >
https://listmgr.nrao.edu/mailman/listinfo/difx-developers
--
!=============================================================!
Prof. Adam Deller
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797
!=============================================================!
--
!=============================================================!
Prof. Adam Deller
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797
!=============================================================!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: the result of DiFX test.pdf
Type: application/pdf
Size: 174954 bytes
Desc: the result of DiFX test.pdf
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20260819/5a1d898a/attachment-0002.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Specifications of used servers.pdf
Type: application/pdf
Size: 34672 bytes
Desc: Specifications of used servers.pdf
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20260819/5a1d898a/attachment-0003.pdf>
More information about the Difx-users
mailing list