[daip] A problem with FITLD in Aips while DOUVCOMP=1

[Eric Greisen]

Subhashis Roy wrote:
> Hi Eric,
> 
> I have come across situations with the GMRT data while loading with 
> FITLD (both 31DEC08 and 31DEC09 version) and 'DOUVCOMP=1' that most of 
> the UV visibility amplitudes for certain very weak source is 0.0000 
> (there should be noise) for most of the channels. However, this problem 
> do not occur
> when 'DOUVCOMP=-1'.
> 
> I tried to identify the problem.
> 
> As I digged deepper it turns out that the 1st Frequency channel in the 
> data set has visibility amplitudes of ~kilo-Jansky (mostly spurious DC 
> values appear there from the FX correlator) for that weak source.
> 
> While it is known that for DOUVCOMP=1, the weights for all channels, 
> Stokes and IFs for a particular baseline and time are the same, it is 
> not obvious from the FITLD documentation that why large amplitudes in 
> one channel should cause visibility values to change in other channels.
> 
> I got a hint from the FITAB documentation that for compressed data the 
> dynamic range for each visibility (I am taking it in the sense that it 
> is true for data across the frequency channels) is ~32,700. If I assume 
> the same for FITLD with 'DOUVCOMP=1' it could explain the result. (The 
> typical noise values are ~0.05, while the channel 1 has amplitudes
> ~5000).
> 
> Therefore, we can see when there are very high values in the data set 
> either due to instrumental problem or RFI, there could be problems with 
> loading that data set with 'DOUVCOMP=1'. This needs to be made clear in 
> the Doc. of FITLD.
> 
> A request:
> ------------
> I believe that many people likes to load the original data with 
> 'DOUVCOMP=1', as this data set could be Big in size, and a factor of 3 
> compression is very useful to reduce Disk usage.
> Therefore, if my above guess on dynamic range of data is right, I would
> suggest an extra step before compressing the data.
> In this step, it should identify the median value of amplitudes for all
> the data across channels, IF and Stokes and set the No. of digits for
> amplitudes scaled properly by this median value.  In this way, one could
> corrupt the high values badly, but would preserve most of the good data.
> 
> For your reference, I am attaching a PRTUV output for a particular
> baseline for 2 different channels with 'DOUVCOMP=1' and 'DOUVCOMP=-1'.
> 
> I am also keeping a FITS file in case you want to test.
> 
> This file is available by 'ftp' once you login as 'anonymous' to
> 'wm.ncra.tifr.res.in' and 'cd' to 'roy'.
> The filename is 'GSB.32M.1609+266.L.NPOLE.FIT'.

My experience with compressed data storage is that the cpu time needed 
to do the compress and uncompress operations all the time frequently 
outweighs the time saved by reading a smaller file.  This would balloon 
out of control if we did a median which is a VERY expensive operation.
It would also destroy one's maser lines which are perfectly good data.
I will add to UVLOD and FITLD help files more warnings against using
compression but you must stop GMRT from writing destructive garbage.

Eric Greisen