[daip] MSORT problem in 31DEC16 AIPS

[James M. Anderson]

Hello Eric,

On Tue, 2016-07-05T01:49:15CEST, Eric Greisen <egreisen at nrao.edu> wrote:
> MSORT was written long ago when UVSRT was slow in many cases.  It is
> a quirky program which I could try debugging if you make your data
> available on some anonymous ftp site (it is fairly small although
> big for e-mail).  But I would suggest switching to UVSRT which I
> believe is just as fast these days and much more reliable.  Tell me
> if I am wrong about speed but I have been under the belief that
> UVSRT has more or less caught up in modern hardware.
> 
> Eric Greisen
> 

Thanks for the tip.  I have run a comparison test of time for MSORT
and UVSRT on my machine (purchased this year, with two separate disk
systems for AIPS to use).  For my relatively small dataset, performing
100 runs to get a reasonable average performance time, MSORT and UVSRT
have the same wall-clock time, with UVSRT just slightly faster.

However, for my larger dataset (~ 4 GB) of geodetic observations, UVSRT
requires more than 9 times as long as MSORT to sort the data.  The
tests were repeated several times to eliminate any system-caching
influences.  All tests were run as in my batch processing with
SORT='TB' and standard defaults for everything else.

Admittedly, the UVSRT run time in the larger dataset is only 93
seconds versus 10 seconds for MSORT, so I could live with that since I
don't need to sort the larger dataset too often, if you would prefer
that users migrate to UVSRT.  :)

I will see where I can make my small dataset available for you in case
you fell like testing.

Thanks for the feedback, and have a good day,


James M Anderson



Testing results --- small dataset (~ 65 MB)

MSORT1: Read UV data for the sort keys
MSORT1: Keys read - sort them
MSORT1: Keys sorted, order established, now resort
MSORT1: Keys now ready to use
MSORT1:      14241 unsorted starting at         1 avg sep     2391
MSORT1: IMSORT: Begin first (neighbor) sort pass
MSORT1:       6451 unsorted starting at         1 avg sep     1942
MSORT1: IMSORT: Begin second (brute-force) sort pass
MSORT1:       3425 unsorted starting at      5292 avg sep      677
MSORT1:       2864 unsorted starting at      8456 avg sep      402
MSORT1:        200 unsorted starting at     10236 avg sep       33
MSORT1:          2 unsorted starting at     13593 avg sep        0
MSORT1: Appears to have ended successfully
MSORT1: kg122 31DEC16 TST: Cpu=      0.4  Real=      1  IO=       305

Elapsed time=38.22 s, 0.3822 s per iteration



UVSRT1: kg122 31DEC16 TST: Cpu=      0.3  Real=      1  IO=       186

Elapsed time=36.70 s, 0.3670 s per iteration





Testing results --- larger dataset (~ 4 GB)

MSORT1: Read UV data for the sort keys
MSORT1: Keys read - sort them
MSORT1: Keys sorted, order established, now resort
MSORT1: Keys now ready to use
MSORT1:     936181 unsorted starting at         1 avg sep        4
MSORT1: IMSORT: Begin first (neighbor) sort pass
MSORT1: Appears to have ended successfully
MSORT1: kg122 31DEC16 TST: Cpu=      4.6  Real=     10  IO=     18179


UVSRT1: kg122 31DEC16 TST: Cpu=     24.8  Real=     93  IO=     72343




-- 
James M Anderson, PhD
Department 1: Geodesy
Section 1.1:  Space Geodetic Techniques
Tel.:         +49 (0)331/288-28948
Fax:          +49 (0)331/288-1111
E-Mail:       anderson at gfz-potsdam.de
___________________________________

Helmholtz-Zentrum Potsdam
Deutsches GeoForschungsZentrum GFZ
Stiftung des öff. Rechts Land Brandenburg
Telegrafenberg A6
14473 Potsdam
Germany