[daip] NSF review

Wed Sep 7 18:33:12 EDT 2005

I would probably reduce the detail, and hence the length.  In particular, I
would reduce substantially the description of new tasks in 2004/5.
What I would say is something like "Examples of the modifications to
AIPS in the last year to support VLA and VLBA observers include
the following:
(1) Models for primary flux calibration sources at many frequencies
are now included with AIPS distributions.
(2) Calibration tasks have been improved to offer more robust gain
solution methods, and flag discrepant solutions.
(3) The primary imaging task has been modified to enable more
efficient use of CLEAN algorithms.
(4) The VLA data-filling task was modified to read archive data from
disk, in support of the NRAO on-line archive."

Then I would take out the paragraphs on the same subjects and on
FIXBX.  My inclination is to leave in the full paragraph on ATMCA and
DELZN, perhaps with an introductory sentence stating that
"Improved astrometric results for the VLBA have been enabled by the
implementation of new tasks that make use of observations of multiple
reference sources.  A new task ..."

I'd go ahead and send in the Plan part more-or-less as is.  It might be
that this should be shortened somewhat, but I think it's a reasonable
description and I'd rather not try to wordsmith it at the moment.
It may in fact be that the plan will end up in Section 5 of the document
under VLA/VLBA, but I'm not sure.

Jim

Eric Greisen wrote:

>I need your comments on this section to go into the "white paper"
>asap.
>
>Eric
>
>4.5   AIPS
>
>4.5.1  Overview and resources
>
>In 1990, the NRAO decided to replace the AIPS package with more modern
>software.  The AIPS Group was asked to carry on supporting the users
>of the VLA and the VLBA until such time as the new software replaced
>the functionality of AIPS.  Although new development in AIPS was
>discouraged for a time, the current AIPS package is vastly better and,
>in many ways, more modern than it was in 1990.  Supporting current
>users of current instruments is still the main role of the group.
>This includes assisting with installation of the system, answering
>queries about the capabilities of AIPS, correcting bugs, and fixing
>minor (and not so minor) inconveniences in the system.  Where
>possible, the group does add new capabilities to improve calibration
>and imaging techniques or to correct newly discovered instrumental
>problems.
>
>The official AIPS Group consists of some fraction of four individuals:
>   1.  EG - Scientist 0.75 FTE, management and all software areas
>   2.  AM - Asst. Scientist 0.5 FTE, VLBI expert, some software
>   3.  LK - Scientist 0.5 FTE, mathematician, VLBI software
>   4.  WY - Software Engineer 0.15 FTE, system support
>with an additional person who works with the Group:
>   5.  LS - Assoc. Scientist  0.5 FTE, pipeline processing
>The total is 2.4 FTE/yr, where the effective times have been
>estimated.  The official times differ from these numbers but the sum
>is about the same
>
>
>
>4.5.2  Status and details
>
>The 31DEC04 version of AIPS was developed through 2004 and then frozen
>in late December.  It was available for download (and update via the
>"midnight job" or MNJ) during development and the frozen version may
>now be downloaded.  The new test version, 31DEC05, was started in
>December 2004 and is available for download and update.  We have
>developed tools to count downloads of full AIPS versions and to count
>access to the main "cvs" site.  The latter reflects both initial
>installation and updating of the development version; frozen versions
>do not generate cvs accesses.  Counting each unique IP address as a
>"site", there were cvs contacts from 797 sites in 2004 of which 231
>appear to have run the MNJ at least occasionally.  The frozen 31DEC03
>version was downloaded by 196 sites and the 31DEC04 version, while
>under development, was downloaded by 808 sites.  A total of 1276
>unique IP addresses downloaded a copy of AIPS and/or accessed the cvs
>site.  At this writing (7 September 2005), the frozen 31DEC04 version
>has been downloaded by 213 sites and the development 31DEC05 version has
>been downloaded by 629 sites.  The total number of IP addresses is
>already 1136.  41 sites have downloaded 31DEC04 in the binary form,
>while 193 sites have downloaded 31DEC05 in binary.
>
>We have found that the Fortran compiler developed by IBM for MacIntosh
>systems generates code that is 50% faster than that produced by the
>GNU compilers.  Unfortunately, the IBM compiler is moderately
>expensive.  Therefore, we have made available a binary distribution of
>AIPS.  This binary form is available both for the frozen 31DEC04
>release and the development 31DEC05 version, including periodic
>updates (daily are possible) via the MNJ.  Similarly, the SUNWspro
>compiler produces faster binaries on Solaris machines and the Intel
>ifort compiler produces 35% faster binaries on Linux Pentium IVs.
>We have therefore made binary installations available also for Solaris
>and Linux systems.  The Linux binaries are produced in a manner that
>allows them to run on most types of CPU with particularly good
>performance on Pentium IVs.
>
>Steps are being taken to support greater use of pipeline and other
>procedures in AIPS.  A large task FLAGR was written to use the
>internal statistics in a data set to flag that data set.  The new task
>FINDR was written to determine some of the same statistics, returning
>values to the AIPS user.  Studies are now underway to determine how
>these tasks may be used in AIPS' VLA and VLBA data reduction
>pipelines, particularly in flagging calibration sources.  Several
>other tasks were modified or created to allow information to be
>computed and passed back to AIPS for use by procedures.
>
>Models for the primary flux calibration sources are now provided with
>AIPS.  There are four sources, 3C48, 3C138, 3C147, and 3C286, at the
>three highest VLA frequency bands, K, Q, and U.  These models were
>provided by Claire Chandler.  X-band (3.6-cm) and C-band (6-cm) models
>for 3C48 and 3C286 have been provided so far by Amy Mioduszewski.  The
>L-band (21-cm) model for 3C48 has been released recently and the 3C286
>L-band model is being prepared.  We expect, in the long run, to
>provide models for all four sources at all VLA frequencies.  The
>pipeline procedures for the VLA are being revised to use these models.
>The verb CALDIR to list available models and the task CALRD to read in
>models were added to AIPS.
>
>The calibration and bandpass tasks, CALIB and BPASS, were revised to
>offer "robust" gain solution methods.  Such methods progressively
>refine solutions by ignoring significantly discrepant data at each
>iteration.  This change in the fitting routines provides data to allow
>CALIB to flag visibilities with significant closure error.  BPASS was
>changed to handle channel-dependent flagging correctly.
>
>A new task, ATMCA, was written to refine the calibration during
>phase-referencing observations through the use of additional
>calibration sources.  The direction-dependency of phase error is
>fit from the multiple calibration observations and used to refine the
>gain solution on the target source.  AIPS Memos on ATMCA and the
>earlier DELZN astrometric-level calibration tasks were released.
>Note that both these tasks require observations to be scheduled to
>provide the additional calibration information required to model the
>atmospheric direction dependency of phase.
>
>The AIPS CookBook was kept up to date as always.  In addition, it was
>developed so that, in addition to the usual PostScript version, both
>html and pdf versions exist.  The latter provide full cross-reference
>capability including using the web browser to examine cross-linked
>help files.  This XHELP facility was corrected and greatly enhanced
>during 2004.
>
>In the past, when the Midnight Job detected changes in certain
>system-like files, all the AIPS Managers were alerted and instructed
>to perform a variety of manual operations.  The MNJ procedures were
>changed to do almost all of these automatically, greatly simplifying
>the Managers' job.  The installation from CDrom was tested and
>corrected.  It is actually pretty slick.
>
>The new task FIXBX converts Clean windows from one set of cell size
>and facet locations to another.  New tasks to renumber frequency IDs
>and sources were written.  New verbs to draw Clean boxes on the TV, to
>return random numbers, and to provide direct access to the operating
>system were added.  The last has interesting implications for
>pipelines and can even run an AIPS beneath the current one.
>
>Recently, a path was developed by scientists in Green Bank with the
>assistance of the AIPS group to bring GBT data to AIPS for its
>single-dish imaging.
>
>Task upgrades included a number of changes to IMAGR to enable the
>Steer-Dewdney-Ito Clean algorithm to be used efficiently.  The complex
>Clean task and procedure (CXCLN and CXPOLN) were corrected.
>On-the-fly imaging is still done at the 12-m telescope, formerly
>operated by the NRAO, and AIPS can now handle the new, changed data
>format as well as the older one.  All plot tasks and verbs that used
>the Tektronix-emulation window now offer a TV-window equivalent.  The
>VLA data filling task FILLM was changed to read archive data from disk
>and, in numerous small ways, to handle various error conditions, to
>deal with multiple data sets from the archive, and to precess
>coordinates in a manner consistent with the rest of AIPS.
>
>
>
>4.5.3  Plan
>
>Any plan for the AIPS project begins with the assumption that
>personnel in the group will not increase and the hope that management,
>health concerns, aging (retirement), and career advancement will not
>cause it to decrease.  Furthermore, the group's main role will remain
>the support of the existing software and user base.  That task alone
>will occupy most of the official AIPS Group's resources.
>
>At present, there is a project to enhance the existing VLA calibration
>and imaging pipeline and to employ it is a demonstration project on a
>selected portion of the VLA archive.  This project is currently
>limited to observations of total intensity in the continuum.
>Polarization and spectral-line observations will be addressed in
>coming years.  The test version of the pipeline will consist of a
>script that fetches data from the archive, loads it into AIPS, does
>preliminary flagging, writes an AIPS input RUN file, runs that file,
>and then stores the results in the processed-data archive.  This
>script invokes a pipeline procedure which will be made available to
>all users of AIPS.  This procedure edits the data automatically,
>applies model images for calibration, carries out more standard
>calibrations, selects imaging parameters, images the data Cleaning to
>three time theoretical noise, corrects for the primary beam, and
>writes out the results as calibration tables, calibrated UV data, and
>Cleaned images in FITS format.  Multi-facet imaging and
>self-calibration will become optional portions of this pipeline in
>time.
>
>This pipeline project is well underway, but there is much to do.
>After the structure is completed, the effectiveness of the editing and
>imaging will need to be evaluated and almost certainly improved.
>Adding additional capabilities will also take time.  Much of the work
>will be done by the adjunct member of the group mentioned above.
>Testing and any needed modifications to the normal AIPS code will be
>done by the other members of the group.
>
>The VLBA has received a variety of algorithmic support.  The
>astrometric tasks DELZN and ATMCA allow special calibration
>observations to enhance the accuracy of the calibration applied to the
>target sources.  Recently, the group has worked on a correction to the
>data to account for improved knowledge of Earth orientation
>parameters.  When the VLBI data are correlated these parameters may
>not be fully determined and predictions of their values must be used.
>At a later date, better values become known and the data may be
>corrected substantially.  We expect that other astrometric
>capabilities will be needed over the coming years.  In particular,
>there are needs to add the "calc" astrometric package to AIPS for
>recomputing whole phase solutions.
>
>The EVLA will also need support from the existing software base.
>AIPS will be expected to read EVLA data from the current and the
>prototype correlators.  This may require changes since the computers
>and software used to write these data will change from those used at
>present.  New calibration modes may be required including doing
>combined total intensity and polarization calibration and using
>fringe-fitting techniques across the wide bandwidths of the EVLA.
>There is already a brand new task in AIPS to fit a spectral index
>image to a spectral cube of total intensity images.  The output of
>this task will be used in IMAGR to correct for spectral index in
>imaging across a significant frequency band.  This is a simple start
>to studies in which the AIPS Group is expected to participate of the
>imaging problems which will be caused by the wide bandwidths of the
>EVLA.  Alternative algorithms for mosaicing will be studied as well.
>The amount of time available for algorithm studies is limited by the
>need to support the user base.  But the fact that any algorithms
>developed become available to a large number of users almost
>immediately gives some leverage to those studies.
>
>Studies of software systems in geophysics have shown that different
>systems, implementing the same published algorithms and starting with
>the same data, produce apparently realistic answers that differ almost
>completely.  In that particular study, the answers were found to
>diverge at the rate of 1% per 6000 lines of code.  It is therefore
>useful to keep AIPS around for some time as a comparison and sanity
>check on the answers being produced by other software systems.
>
>_______________________________________________
>Daip mailing list
>Daip at listmgr.cv.nrao.edu
>http://listmgr.cv.nrao.edu/mailman/listinfo/daip
>  
>