[daip] MSORT problem in 31DEC16 AIPS

[Eric Greisen]

On 07/04/2016 09:12 AM, James M. Anderson wrote:
> Hello,
>
> I am having a problem with MSORT.  In one UV dataset that I am trying
> to sort, MSORT does not properly sort one visibility record.  Instead,
> the whole record (time, integration time, baseline, ..., visibility
> data) is filled with zeros.  I have come across this problem because
> after runnign MSORT I run INDXR, and INDXR complains that the dataset
> MSORT produced is not sorted.
>
> This is the first time that that has happened in my batch processing
> system.  It does not happen with other datasets, but this one
> consistently causes a failure.
>
> The dataset has a 64 MB FITS file (20 MB compressed).  That is
> probably too big to send to you by e-mail.
>
> The input dataset has 14241 visibilities, and in the new, sorted
> dataset, the *last* visibility of the input dataset is missing (no
> corresponding time and baseline in the new dataset), but there is one
> visibility record at the location in the new dataset where the
> corresponding visibility record should be, looking at the times and
> baselines of the surrounding records, but the record is filled with
> zero bits.
>
> That happens when I run MSORT with a different output dataset from the
> input dataset.  When I run MSORT with the output dataset set to the
> input dataset, INDXR does not complain at all, and I find no records
> with a zero for the timestamp.
>
> The MSORT and INDXR output is provided below.  Interestingly, the
> record of zero bits appears in visibility record 13593, the initial
> visibility recortd that needs to be sorted in the last brute-force
> pass where there are only 2 records to be sorted.
>
>
>
>
> MSORT1: Task MSORT  (release of 31DEC16) begins
> MSORT1: Create S_0748+126  .UVDATA.   1 (UV)  on disk  1  cno   43
> MSORT1: Copied SU file from vol/cno/vers  1   45   1 to  1   43   1
> MSORT1: Copied FQ file from vol/cno/vers  1   45   1 to  1   43   1
> MSORT1: Copied AN file from vol/cno/vers  1   45   1 to  1   43   1
> MSORT1: Copied AN file from vol/cno/vers  1   45   2 to  1   43   2
> MSORT1: Copied AN file from vol/cno/vers  1   45   3 to  1   43   3
> MSORT1: Copied AN file from vol/cno/vers  1   45   4 to  1   43   4
> MSORT1: Copied AN file from vol/cno/vers  1   45   5 to  1   43   5
> MSORT1: Read UV data for the sort keys
> MSORT1: Keys read - sort them
> MSORT1: Keys sorted, order established, now resort
> MSORT1: Keys now ready to use
> MSORT1:      14241 unsorted starting at         1 avg sep     2391
> MSORT1: IMSORT: Begin first (neighbor) sort pass
> MSORT1:       6451 unsorted starting at         1 avg sep     1942
> MSORT1: IMSORT: Begin second (brute-force) sort pass
> MSORT1:       3425 unsorted starting at      5292 avg sep      677
> MSORT1:       2864 unsorted starting at      8456 avg sep      402
> MSORT1:        200 unsorted starting at     10236 avg sep       33
> MSORT1:          2 unsorted starting at     13593 avg sep        0
> MSORT1: Appears to have ended successfully
> MSORT1: kg122 31DEC16 TST: Cpu=      0.4  Real=      0  IO=       305
>
> INDXR1: Task INDXR  (release of 31DEC16) begins
> INDXR1: Creating CL table with version number 1
> INDXR1: QUITTING: FILE CONTAINS DATA WHICH IS OUT OF TIME ORDER
> INDXR1: UV FILE HEADER SORT ORDER IS INCORRECT
> INDXR1: Use UVSRT or MSORT to sort the file into  time order
> INDXR1: and try again.
> INDXR1: Purports to die of UNNATURAL causes
> INDXR1: kg122 31DEC16 TST: Cpu=      0.0  Real=      0  IO=        33


MSORT was written long ago when UVSRT was slow in many cases.  It is a 
quirky program which I could try debugging if you make your data 
available on some anonymous ftp site (it is fairly small although big 
for e-mail).  But I would suggest switching to UVSRT which I believe is 
just as fast these days and much more reliable.  Tell me if I am wrong 
about speed but I have been under the belief that UVSRT has more or less 
caught up in modern hardware.

Eric Greisen