[daip] calib solution intervals

[Eric Greisen]

It is very tempting to me to simply delete CALIB from the system (and 
all other tasks that attempt to average users' data).  There is no right 
way to do it and CALIB works very hard at it.

The 07 version used regular intervals - a scan-dependent value of SOLINT 
which in your case is 60.4167 sec.  That meant that the first average 
started at 9:10.0, the next at 9:10.16 etc.  In this way, you got the 
desirable result that most averages included 13 samples and two 
contained 14 and all S/N was reasonable.  Note that is the data set did 
not have an index table then 60.0 seconds would have been used and you 
would have had an undesirable result with a dangling sample.

Now consider other people's data sets - say one with an integration 
every 45 seconds.  This would produce 1 sample in a regular grid of 
times and 2 samples in some intervals.

   The new scheme starts each interval with a data point.  In your case 
that means the first starts at 9:10, the next at 10:15, etc which makes 
12 averages of 13 samples with 2 left dangling.  Not desirable, but the 
other fellow gets 2 samples per interval for every interval - desirable.

I am sure that I could come up with numerous egregious examples no 
matter which way one attempts to average.  The task does not know what 
is coming except that CALIB does know the start and stop times of the 
scan but not what times if any occur in between.  I suppose that we 
could read the data twice, once to make a list of the times that occur 
and then attempt to balance the integrations well with some sort of list 
and then again to do the actual averages.  In your data, that algorithm 
would take negligible time.  On an EVLA Widar data set I am testing 
(only 4 antennas and 10 minutes data), a re-read would take 15 minutes 
or more.

I don't know the answer - one proposal was to attach an integration time 
to each sample and use it to determine when an integration is done (only 
FILLM does this at present).  Then if you asked for 60 sec you would get 
12 samples,  and 14 solutions the last with only 2 samples, but each 
integration would be the 60 seconds you asked for.

I do have a thought - one could change the format of the NX table 
(serious work to keep NX using programs from croaking on old format 
files) so that each scan had a list of the number of times in the scan 
with times with 0.01 sec of each other counted as the same time.  Then 
CALIB could adopt a scheme that said include e.g. 13 records in most 
integration times but distribute up to e.g. 13/2 samples into the other 
integrations rather than one at the end.  Of course if the times due to 
flagging and averaging are all messed up this may still make a big mess 
since CALIB will have to assume that the times are more or less regular 
in the counts in the scan.

Note that with single precision floats - your time of 3 days may not 
measure time to that accuracy.  It is really best to keep it down to 
about 1 day, but that is not the source of your problem although we did 
have an issue of trying to read past the end of the NX table because of 
this.  Now that attempt is still done but in a non-fatal way.

This is probably a longer answer than you expected but this is a problem 
that has deviled us for 29.5 years and every solution we have tried has 
made trouble for some vocal set of users.

Eric Greisen