[daip] gluing data together

[Eric Greisen]

Rob Ivison wrote:
> hi Eric -
> 
> i'm using a lnx64 system and the raw uvfits file is 18GB in size. the 
> 3.5TB disk i'm using for the reduction is 34% full and contains a few 
> dozen datasets of this size/type, so the one i'm imaging can't be 
> anything close to 2TB.
> 
> for the UVGLUed file, sizefile gives:
> AIPS 2: Main file size  7596.729 Megabytes
> 
> i've been through my procedure - uvflg, tvflg, clip, (self-)calib, 
> clcal, bpass, calib, clcal, splat, reway, uvglu, imagr - about a dozen 
> times. it has worked perfectly 3-4 times but IMAGR has given this kind 
> of error in the majority of cases. sometimes it works for a particular 
> dataset, but if i change the processing steps even a little bit, it 
> fails at the point that i run IMAGR. i haven't been able to track down 
> the step that causes the problem. i can't help thinking it must be 
> related to the format of the files going into UVGLU.
> 
> sometimes the SPLATed files have the following in the header:
> 
> AIPS 2: Keyword = 'MAXABSU '  value = -1.000000E+00
> AIPS 2: Keyword = 'SOURNAM1'  value = '163650  '
> AIPS 2: Keyword = 'SOURNAM2'  value = '        '
> 
> and sometimes just
> AIPS 2: Keyword = 'MAXABSU '  value = -1.000000E+00
> 
> i've not seen that before and don't know what it means. probably unrelated.
> 
> cheers,
> 
> Rob
> 
> 
> 
>> Date: Mon, 17 May 2010 13:01:07 -0600
>> From: Eric Greisen <egreisen at nrao.edu>
>> To: Rob Ivison <rji at roe.ac.uk>
>> Cc: Frazer Owen <fowen at aoc.nrao.edu>
>> Subject: Re: gluing data together
>>
>> Rob Ivison wrote:
>>
>>> broom > IMAGR1: Task IMAGR  (release of 31DEC10) begins
>>> broom > IMAGR1: Doing no flagging this time
>>> broom > IMAGR1: Create 163650      .IMAGR .   1 (UV)  on disk  4  cno 1
>>> broom > IMAGR1: Beginning channel   22 through   61 with  1 IFs
>>> broom > IMAGR1: ZMI2: REQUEST FOR BYTES -624192511 THRU = -623793896 
>>> BEYOND EOF
>>> broom >         = -624187392
>>> broom > IMAGR1: ZMIO: OPER=READ LUN=25 BLKNO=     7779046 
>>> 8-BIT-BYTES=398616
>>> broom > IMAGR1: ZERROR: IN ZMI2 ERRNO = 22 (Invalid argument)
>>> broom > IMAGR1: ZWAIT: ZWAI2 RETURNS ERROR 3 FOR LUN = 25 BUFFER = 2
>>> broom > IMAGR1: ZERROR: IN ZWAI2  ERRNO = 22 (Invalid argument)
>>> broom > IMAGR1: DATGET: ERROR  3 READING UV DATA
>>> broom > IMAGR1: UVREAD: ERROR   3 READING UV DATA
>>> broom > IMAGR1: UVREAD: ERROR READING Input UVdata
>>> broom > IMAGR1: IMACPY: ERROR COPYING Input UVdata
>>
>> How big is the UV data file?  I note that all 3 numbers here have 
>> overflowed their formats and so are negative.  This suggests that the 
>> data set is such that a 32-bit integer cannot count its size in 
>> kilobytes.  That would then require a data set > 2 Terabytes.
>>
>> ERic Greisen
>>


I find this very baffling - these numbers clearly look like a number 
overflow (how can the file size be negative???) and yet the code at the 
bottom of ZMI2 uses a number form that should be 64 bits in length.  You 
are using the binary version are you not or do you compile locally?

I have a meeting now unfortunately...

Eric Greisen