[daip] AIPS flag entry limitations

[Eric Greisen]

Danielle Fenech wrote:
> Hello,
> 
> I'm currently working on flagging software to work on eMERLIN data via 
> parseltongue, which writes out flag tables to be loaded into AIPS via 
> TBIN. However, I have run into some problems with limitations of various 
> tasks in handling a large number of flag entries.
> TBIN itself declares a limitation of 1 million entries and won't write 
> anything larger. Tasks such as SPFLG seem to have an even lower limitation 
> of around 120,000 making assessing the flag entries difficult.
> 
> The datasets themsleves are reasonably large, up to 500G and possibly 
> larger and typically the flag tables written by the flagging software are 
> consistently a few to 15-20 million entries in size (sometimes more).
> 
> Is there any way to get the working limit for flag entries raised (or 
> even removed) and would it be possible to have a consistent limitation as 
> opposed to task-based?
> 
> I don't currently know where these limitations are set within the AIPS 
> structure, but if you can point me in the right direction, I could have a 
> look at making some changes locally. However, given the large volumes of 
> data and the intended general use of such flagging software, ideally a 
> more global fix for the problem would be better.

The limitation in AIPS at present is 30000 flags that apply to a single 
time.  That applies to all but a handful of tasks which allow larger 
tables (UVCOP primarily).  The reason there is a limited number is 
Fortran which requires allocated RAM unless one is quite clever.  The 
cleverness is straightforward for single arrays of e.g. REALs for 
computing images and AIPS does that a lot now.  But the flag table has

        REAL      TMFLST, FLGTST(MAXFLG), FLGTND(MAXFLG)
       LOGICAL   FLGPOL(4,MAXFLG), TIMORD
       INTEGER   IFGRNO, NUMFLG, FGKOLS(MAXFGC), FGNUMV(MAXFGC),
      *   KNCOR, KNCF, KNCIF, KNCS, KCOR0, FGBUFF(512),
      *   FLGSOU(MAXFLG), FLGANT(MAXFLG), FLGBAS(MAXFLG), FLGSUB(MAXFLG),
      *   FLGBIF(MAXFLG), FLGEIF(MAXFLG), FLGBCH(MAXFLG), FLGECH(MAXFLG),
      *   FLGFQD(MAXFLG)

and really no way of knowing in advance how many flags will apply to a 
single time.  Even with RFLAG which generates millions of flags, we do 
not hit this limit much at 30000 - but it does require some intelligence 
in the flag generation software to try to avoid excessive flag commands
(which take time to apply as well as space in RAM).

I can and probably will raise this limit some in 31DEC13 but there is no 
upper limit that is at all reasonable and every task with FLAGVER has to 
have this memory space.  Note that this limitation is the number of 
flags that apply to a single time, not the total number of flags in the
table.

There is a task REFLG that attempts to compress FG tables - perhaps you 
should try that.  I will look at TBIN which should probably not have a 
low limit....

Eric Greisen