[daip] [!5346]: aips - VLA14B-292 Computer Processing Time question

[Emmanuel Momjian]

Emmanuel Momjian updated #5346
------------------------------

VLA14B-292 Computer Processing Time question
--------------------------------------------

           Ticket ID: 5346
                 URL: https://help.nrao.edu/staff/index.php?/Tickets/Ticket/View/5346
           Full Name: Robert H Gray
               Email: roberthansengray at gmail.com
             Creator: User
          Department: AIPS Data Processing
       Staff (Owner): Emmanuel Momjian
                Type: Issue
              Status: Response Overdue
            Priority: Default
                 SLA: NRAO E2E
      Template Group: Default
             Created: 11 August 2014 09:33 PM
             Updated: 15 August 2014 04:33 PM
      Resolution Due: 21 August 2014 09:41 PM (6d 5h 8m)



I will give you some numbers based on my data sets. I hope these will be helpful for you to find out how much time your data reduction may take.

The particular data set my example is based on is 4 hours in total (~80% on target) with 15 spectral windows, each 32 MHz wide with 2048 channels. So total number of channels is 30720 (or 61440 channels if you count RR and LL separately). The raw data volume is ~300 GB. I reduce the data with a pipeline in CASA that does calibration and flagging, including RFI flagging. The pipeline, end-to-end, including the loading of the data from SDM-BDF to a CASA measurement set, takes 35 hours on the cluster here in Socorro (note however that no parallel processing is involved in the CASA pipeline).

After the pipeline run I manually check the calibration and flagging quality. This could be done in a day or two depending on the problems I see in the data (note that my data have lots of RFI. Also the measurement set after calibration becomes double the size of the raw data, i.e., ~600 GB). Afterwards I add extra flags and push the data through another pipeline that only does calibration (and no more flagging). The latter takes about half the time of the 1st pipeline run.

Imaging such a data set can take a while. For instance it takes me ~2.5 hours to make an image cube using 1000 channels from this data set. The imaging did not involve cleaning.

I expect AIPS to take a comparable amount of time during calibration, flagging and imaging (if not shorter). For the imaging in AIPS, definitely use LINIMAGE that Eric noted instead of IMAGR.

Hope the above are helpful for you assessment.



Emmanuel Momjian
Scientist-Astronomer
NRAO Science Operations Center


------------------------------------------------------
Staff CP:  https://help.nrao.edu/staff