[daip] NSF review

[Eric Greisen]

I need your comments on this section to go into the "white paper"
asap.

Eric

4.5   AIPS

4.5.1  Overview and resources

In 1990, the NRAO decided to replace the AIPS package with more modern
software.  The AIPS Group was asked to carry on supporting the users
of the VLA and the VLBA until such time as the new software replaced
the functionality of AIPS.  Although new development in AIPS was
discouraged for a time, the current AIPS package is vastly better and,
in many ways, more modern than it was in 1990.  Supporting current
users of current instruments is still the main role of the group.
This includes assisting with installation of the system, answering
queries about the capabilities of AIPS, correcting bugs, and fixing
minor (and not so minor) inconveniences in the system.  Where
possible, the group does add new capabilities to improve calibration
and imaging techniques or to correct newly discovered instrumental
problems.

The official AIPS Group consists of some fraction of four individuals:
   1.  EG - Scientist 0.75 FTE, management and all software areas
   2.  AM - Asst. Scientist 0.5 FTE, VLBI expert, some software
   3.  LK - Scientist 0.5 FTE, mathematician, VLBI software
   4.  WY - Software Engineer 0.15 FTE, system support
with an additional person who works with the Group:
   5.  LS - Assoc. Scientist  0.5 FTE, pipeline processing
The total is 2.4 FTE/yr, where the effective times have been
estimated.  The official times differ from these numbers but the sum
is about the same



4.5.2  Status and details

The 31DEC04 version of AIPS was developed through 2004 and then frozen
in late December.  It was available for download (and update via the
"midnight job" or MNJ) during development and the frozen version may
now be downloaded.  The new test version, 31DEC05, was started in
December 2004 and is available for download and update.  We have
developed tools to count downloads of full AIPS versions and to count
access to the main "cvs" site.  The latter reflects both initial
installation and updating of the development version; frozen versions
do not generate cvs accesses.  Counting each unique IP address as a
"site", there were cvs contacts from 797 sites in 2004 of which 231
appear to have run the MNJ at least occasionally.  The frozen 31DEC03
version was downloaded by 196 sites and the 31DEC04 version, while
under development, was downloaded by 808 sites.  A total of 1276
unique IP addresses downloaded a copy of AIPS and/or accessed the cvs
site.  At this writing (7 September 2005), the frozen 31DEC04 version
has been downloaded by 213 sites and the development 31DEC05 version has
been downloaded by 629 sites.  The total number of IP addresses is
already 1136.  41 sites have downloaded 31DEC04 in the binary form,
while 193 sites have downloaded 31DEC05 in binary.

We have found that the Fortran compiler developed by IBM for MacIntosh
systems generates code that is 50% faster than that produced by the
GNU compilers.  Unfortunately, the IBM compiler is moderately
expensive.  Therefore, we have made available a binary distribution of
AIPS.  This binary form is available both for the frozen 31DEC04
release and the development 31DEC05 version, including periodic
updates (daily are possible) via the MNJ.  Similarly, the SUNWspro
compiler produces faster binaries on Solaris machines and the Intel
ifort compiler produces 35% faster binaries on Linux Pentium IVs.
We have therefore made binary installations available also for Solaris
and Linux systems.  The Linux binaries are produced in a manner that
allows them to run on most types of CPU with particularly good
performance on Pentium IVs.

Steps are being taken to support greater use of pipeline and other
procedures in AIPS.  A large task FLAGR was written to use the
internal statistics in a data set to flag that data set.  The new task
FINDR was written to determine some of the same statistics, returning
values to the AIPS user.  Studies are now underway to determine how
these tasks may be used in AIPS' VLA and VLBA data reduction
pipelines, particularly in flagging calibration sources.  Several
other tasks were modified or created to allow information to be
computed and passed back to AIPS for use by procedures.

Models for the primary flux calibration sources are now provided with
AIPS.  There are four sources, 3C48, 3C138, 3C147, and 3C286, at the
three highest VLA frequency bands, K, Q, and U.  These models were
provided by Claire Chandler.  X-band (3.6-cm) and C-band (6-cm) models
for 3C48 and 3C286 have been provided so far by Amy Mioduszewski.  The
L-band (21-cm) model for 3C48 has been released recently and the 3C286
L-band model is being prepared.  We expect, in the long run, to
provide models for all four sources at all VLA frequencies.  The
pipeline procedures for the VLA are being revised to use these models.
The verb CALDIR to list available models and the task CALRD to read in
models were added to AIPS.

The calibration and bandpass tasks, CALIB and BPASS, were revised to
offer "robust" gain solution methods.  Such methods progressively
refine solutions by ignoring significantly discrepant data at each
iteration.  This change in the fitting routines provides data to allow
CALIB to flag visibilities with significant closure error.  BPASS was
changed to handle channel-dependent flagging correctly.

A new task, ATMCA, was written to refine the calibration during
phase-referencing observations through the use of additional
calibration sources.  The direction-dependency of phase error is
fit from the multiple calibration observations and used to refine the
gain solution on the target source.  AIPS Memos on ATMCA and the
earlier DELZN astrometric-level calibration tasks were released.
Note that both these tasks require observations to be scheduled to
provide the additional calibration information required to model the
atmospheric direction dependency of phase.

The AIPS CookBook was kept up to date as always.  In addition, it was
developed so that, in addition to the usual PostScript version, both
html and pdf versions exist.  The latter provide full cross-reference
capability including using the web browser to examine cross-linked
help files.  This XHELP facility was corrected and greatly enhanced
during 2004.

In the past, when the Midnight Job detected changes in certain
system-like files, all the AIPS Managers were alerted and instructed
to perform a variety of manual operations.  The MNJ procedures were
changed to do almost all of these automatically, greatly simplifying
the Managers' job.  The installation from CDrom was tested and
corrected.  It is actually pretty slick.

The new task FIXBX converts Clean windows from one set of cell size
and facet locations to another.  New tasks to renumber frequency IDs
and sources were written.  New verbs to draw Clean boxes on the TV, to
return random numbers, and to provide direct access to the operating
system were added.  The last has interesting implications for
pipelines and can even run an AIPS beneath the current one.

Recently, a path was developed by scientists in Green Bank with the
assistance of the AIPS group to bring GBT data to AIPS for its
single-dish imaging.

Task upgrades included a number of changes to IMAGR to enable the
Steer-Dewdney-Ito Clean algorithm to be used efficiently.  The complex
Clean task and procedure (CXCLN and CXPOLN) were corrected.
On-the-fly imaging is still done at the 12-m telescope, formerly
operated by the NRAO, and AIPS can now handle the new, changed data
format as well as the older one.  All plot tasks and verbs that used
the Tektronix-emulation window now offer a TV-window equivalent.  The
VLA data filling task FILLM was changed to read archive data from disk
and, in numerous small ways, to handle various error conditions, to
deal with multiple data sets from the archive, and to precess
coordinates in a manner consistent with the rest of AIPS.



4.5.3  Plan

Any plan for the AIPS project begins with the assumption that
personnel in the group will not increase and the hope that management,
health concerns, aging (retirement), and career advancement will not
cause it to decrease.  Furthermore, the group's main role will remain
the support of the existing software and user base.  That task alone
will occupy most of the official AIPS Group's resources.

At present, there is a project to enhance the existing VLA calibration
and imaging pipeline and to employ it is a demonstration project on a
selected portion of the VLA archive.  This project is currently
limited to observations of total intensity in the continuum.
Polarization and spectral-line observations will be addressed in
coming years.  The test version of the pipeline will consist of a
script that fetches data from the archive, loads it into AIPS, does
preliminary flagging, writes an AIPS input RUN file, runs that file,
and then stores the results in the processed-data archive.  This
script invokes a pipeline procedure which will be made available to
all users of AIPS.  This procedure edits the data automatically,
applies model images for calibration, carries out more standard
calibrations, selects imaging parameters, images the data Cleaning to
three time theoretical noise, corrects for the primary beam, and
writes out the results as calibration tables, calibrated UV data, and
Cleaned images in FITS format.  Multi-facet imaging and
self-calibration will become optional portions of this pipeline in
time.

This pipeline project is well underway, but there is much to do.
After the structure is completed, the effectiveness of the editing and
imaging will need to be evaluated and almost certainly improved.
Adding additional capabilities will also take time.  Much of the work
will be done by the adjunct member of the group mentioned above.
Testing and any needed modifications to the normal AIPS code will be
done by the other members of the group.

The VLBA has received a variety of algorithmic support.  The
astrometric tasks DELZN and ATMCA allow special calibration
observations to enhance the accuracy of the calibration applied to the
target sources.  Recently, the group has worked on a correction to the
data to account for improved knowledge of Earth orientation
parameters.  When the VLBI data are correlated these parameters may
not be fully determined and predictions of their values must be used.
At a later date, better values become known and the data may be
corrected substantially.  We expect that other astrometric
capabilities will be needed over the coming years.  In particular,
there are needs to add the "calc" astrometric package to AIPS for
recomputing whole phase solutions.

The EVLA will also need support from the existing software base.
AIPS will be expected to read EVLA data from the current and the
prototype correlators.  This may require changes since the computers
and software used to write these data will change from those used at
present.  New calibration modes may be required including doing
combined total intensity and polarization calibration and using
fringe-fitting techniques across the wide bandwidths of the EVLA.
There is already a brand new task in AIPS to fit a spectral index
image to a spectral cube of total intensity images.  The output of
this task will be used in IMAGR to correct for spectral index in
imaging across a significant frequency band.  This is a simple start
to studies in which the AIPS Group is expected to participate of the
imaging problems which will be caused by the wide bandwidths of the
EVLA.  Alternative algorithms for mosaicing will be studied as well.
The amount of time available for algorithm studies is limited by the
need to support the user base.  But the fact that any algorithms
developed become available to a large number of users almost
immediately gives some leverage to those studies.

Studies of software systems in geophysics have shown that different
systems, implementing the same published algorithms and starting with
the same data, produce apparently realistic answers that differ almost
completely.  In that particular study, the answers were found to
diverge at the rate of 1% per 6000 lines of code.  It is therefore
useful to keep AIPS around for some time as a comparison and sanity
check on the answers being produced by other software systems.