[daip] Re: CLIPM and zero-UV points

[Eric Greisen]

Shami Chatterjee writes:

 > I've been fiddling with CLIPM, and while writing up another problem
 > report, I realised that AIPS might be incapable of dropping channels that
 > have been flagged in multi-src files...? 

         Bloody unlikely - something is going wrong here but AIPSA is
not incapable of FLAGGING channels in a multi-source file.
Multi-source files are themselves not written upon, so the sample is
not dropped as it can be with CLIP on a single-source file.

 > 
 > But CLIP, at least, can do this for single source files, so I'm not sure
 > what's going on. 

        CLIP deletes data, CLIPM writes a FLAG table which you must
choose to apply.

 > 
 > A blow-by-blow account of what I'm doing follows: in brief, I uvcop a tiny
 > data fragment (1 IF, 16 chans), then UVFND the zero-amp points, CLIPM it,
 > UVFND and see those same zero-amp points, SPLAT/UVCOP the data to other
 > files, and those zero points still persist.
 > 
          These zero-amplitude samples are interesting to me.  They
are probably the result of using channel-dependent flagging on a
compressed data set.  In that case, the fact that a channel has been
deleted WHEN THE COMPRESSED DATA ARE WRITTEN causes the values to be
lost.  Zeros replace the data when they are read - but the weight
should then be negative => flagged.

     The question then becomes what have you done to the original data
set when that you then used UVCOP on below.
 > 
 > ------
 > UVCOP flagver -1 // generate a tiny dataset without flgging.
 > ella  > UVCOP1: Copied     1878. vis records
 > ella  > UVCOP1: Dropped       0. flagged vis records
 > 
 > AIPS 1: Extension file type FG version   1 deleted
 > 
 > UVPLT: peak 2.4 Jy, floor of zero points
 > ella  > UVPLT1: PLTUV:      23104 Points plotted
 > 
 > UVFND: aparm 10, 0, 0.001, 0; opcode 'clip'
 > 
 >  TEST        .UVCOP .   1  Vol= 4  User= 1270  Chans=   1 to    2  IF=  1
 >  Weights have been multiplied by     10.0000
 >  Applying calibration
 >  Applying flagging table
 >   Vis #     IAT      Ant   U(klam)  V(klam)   Amp Phas Wt   Amp Phas Wt
 >       3  0/13:30:00  5- 7   -6006.   -2203.  0.263 180  7  0.000   0  7
 >       9  0/13:30:00  7- 9    6895.    2800.  0.000   0 10  0.259 -90 10
 >      15  0/13:30:00  2- 4    2219.    -893.  0.000   0 10  0.278 180 10
 >      16  0/13:30:00  1- 2      67.    7970.  0.000   0 10  0.245 180 10
 >      21  0/13:30:00  3- 7    7307.     217.  0.000   0 10  0.322 180 10
 >  <etc>
 > // I particularly want to discard these zero points.
 > 
 > CLIPM: aparm 1.5, 0, 0.005, 0; flagver 1
 > ella  > CLIPM1: Flagged      2150 correlators
 > 
 > UVFND: as before, with flagver 1
 >  Vis #     IAT      Ant   U(klam)  V(klam)   Amp Phas Wt   Amp Phas Wt
 >       3  0/13:30:00  5- 7   -6006.   -2203.  0.263 180  7  0.000   0  7
 >       9  0/13:30:00  7- 9    6895.    2800.  0.000   0 10  0.259 -90 10
 >      15  0/13:30:00  2- 4    2219.    -893.  0.000   0 10  0.278 180 10
 >      16  0/13:30:00  1- 2      67.    7970.  0.000   0 10  0.245 180 10
 >      21  0/13:30:00  3- 7    7307.     217.  0.000   0 10  0.322 180 10
 > 
 > //So the flag table is not being read? Or incompletely read?
 > //(Not even the high flags are read, according to UVPLT)
 > 
 > PRTAB: inext 'fg'; box 4 5 7 8 9 
 >      ROW      ANTS      TIME RANG      CHANS      PFLAGS      REASON
 >   NUMBER                DAYS
 >        1        2       5.625E-01        16        1011       CLIPM
 >        1        5       5.625E-01        16
 >        2        3       5.625E-01         1        1011       CLIPM
 >        2        7       5.625E-01         1
 >        3        3       5.625E-01         6        1011       CLIPM
 >        3        7       5.625E-01         6
 >        4        3       5.625E-01        13        1011       CLIPM
 >        4        7       5.625E-01        13
 > 

         The 1011 PFLAG =>that it did not find LL to be bad, only RR.

 > //So the flags exist, at least.
 > //Okay, force the flags with SPLAT or with UVCOP: (try both)
 > 
 > SPLAT flagver 1; aparm 0, 4.980736, 0
 > ella  > SPLAT1:            Previously flagged  flagged by gain kept
 > ella  > SPLAT1: Partially               0                  0    0
 > ella  > SPLAT1: Fully                   0                  0 1444

         These numbers may not reflect loss of a single channel here
and there - they are for full records, e.g ALL channels.

 > 
 > UVFND: as before  
 > Vis #     IAT      Ant   U(klam)  V(klam)   Amp Phas Wt   Amp Phas Wt
 >       3  0/13:30:00  5- 7   -6006.   -2203.  0.263 180  7  0.000   0  7
 >       9  0/13:30:00  7- 9    6895.    2800.  0.000   0 10  0.259 -90 10
 >      15  0/13:30:00  2- 4    2219.    -893.  0.000   0 10  0.278 180 10
 >      16  0/13:30:00  1- 2      67.    7970.  0.000   0 10  0.245 180 10
 >      21  0/13:30:00  3- 7    7307.     217.  0.000   0 10  0.322 180 10
 > 
 > //So these points are still here.
 > 
 > UVCOP flagver 1
 > ella  > UVCOP1: Copied     1878. vis records
 > ella  > UVCOP1: Dropped       0. flagged vis records
 > 
 > //Its almost as if I can't drop the flagged parts of a partially flagged
 > record!
 > 
 > UVFND: as before  
 >   Vis #     IAT      Ant   U(klam)  V(klam)   Amp Phas Wt   Amp Phas Wt
 >       3  0/13:30:00  5- 7   -6006.   -2203.  0.263 180  7  0.000   0  7
 >       9  0/13:30:00  7- 9    6895.    2800.  0.000   0 10  0.259 -90 10
 >      15  0/13:30:00  2- 4    2219.    -893.  0.000   0 10  0.278 180 10
 >      16  0/13:30:00  1- 2      67.    7970.  0.000   0 10  0.245 180 10
 >      21  0/13:30:00  3- 7    7307.     217.  0.000   0 10  0.322 180 10
 > 
 > //Finally, split out and clip first source (CLIP aparm 1.5, 0, 0.005, 0)
 > UVFND, UVPLT: all zero points (and high points) are gone, as expected.
 > ----

Why not use UVPRT - order it to display just what you want.  UVFND is
complex.  Also, use PRTUV if the flags should have been applied.

I will run some tests but this is very confusing.  Leonia, the time
range here is plenty.

ERic