[asac] Telecon (FEB03) memo
Toshikazu Onishi
ohnishi at a.phys.nagoya-u.ac.jp
Tue Feb 23 20:52:28 EST 2010
Dear all,
I am sending herewith a memo of the ASAC telecon on 3rd of February.
Please inform me if there are any problems/errors.
Sorry for the delay in distribution. I caught a cold just after the
telecon, and there was no time to compile the items until now due to
the accumulated duties...
Best regards,
Toshikazu Onishi
---
ASAC Telecon memo
3rd February 2010
Attending: Baker, Johnson, Johnston, Stacey, Williams, Afonso, Gueth, Hogerheijde, Ivison, Kuno,
Momose, Onishi, Hills, Wootten, Morita, Peck, Willson
Agenda: https://safe.nrao.edu/wiki/bin/view/ALMA/ASAC3Feb10Agenda
1. Membership
- New member: Jonathan Williams, Extension: Michiel Hogerheijde
2. Project Report
- January 22nd: CSV started
- Three antennas at high site
- Deployment of the new software system (R7.0 version)
- All 4 basebands (full 16GHz bandwidth)
- Phase 1 configuration(?): 100-200m baseline -> Test in compact baseline in the near future.
- Astronomical Holography expected: Measurement of gravitational deformation of main dishes
- Two more antennas are testing at OSF(?), which enables another interferometry test other than at AOS
- Control room equipped with archive
- Three new people joined to the commissioning scientist, another one is coming
- Frontend delivery schedule slipping.
- Eight antennas in August are needed for early science test -> Frontend will not be available at the time
- Important changes in the Project Management on the North America side
- Adrian Russell leaving in July for ESO. Mark McKinnon will take over.
http://science.nrao.edu/about/news_20100201_mck.shtml
- Management of Frontend: Bill らんど？???
- Frontend IPT: Charles Cunningham??
- Many more...
3. Face-to-Face meting: Charges and preliminary discussion
- Proposal Review Process: Draft distributed
+ Comments from Rob, Andrew, Johnston, and Momose-san are attached below.
+ Major issues
- Number of proposals one panel member should be reviewed per year: ~125
- Burdensome?
- Assumption: 1000 proposals/year
- One calls per year: for cost saving
- Two calls per year will not significantly reduce the referee's load.
- In any case, nearly 1000 proposals/call will be expected
- Some percentage of proposals will be rejected as not being technically feasible
- IRAM: ~100 proposals per semester already
- Investigate other TAC's loads by asking your colleagues for further discussion at f2f meeting
- The way of the estimation can be further explained at f2f
- Science Categories
- Four categories: Appropriate?
- We need to build in some flexibilities in the system
- There may be big in-valance between the areas
- Duplication
- A lot of duplications are expected for the 1st round of proposals.
Many people are thinking the same subjects as urgent science.
- Duplications need to be stated to the proposers before the deadline.
- Proposers should know the rules for the duplications
- Investigate other TAC's handling by asking your colleagues for further discussion at f2f meeting
- De-scope or change proposals
- skipped
- Rigid approach of queue draining
- See Andrew's comments
- Low-rated proposals can eat up all the time for high-rated proposals depending on the weather
- Same types of the thing might be already thought about. Ask Lars and their people on this.
- Accounting of time for proposals which belong to multiple executives: Open sky and Taiwanese proposals
- It should be automatically accounted according to the executive contributions.
- Andrew's idea for accounting of time
- CoPIs and CoIs: CoPIs share scientific leadership and the time accounting
regardless of the number of hours requested
- The idea that the responsible persons share the accounting of time itself feels great
- Still there is a possibility that the sharing will be considered politically.
- In the scheme of the current draft, 100+ hours proposals may be artificially inflated
because proposers can split their risk that comes from the queue draining scheme.
- There is no reason that 100+ proposals should be differently handled compared with smaller ones.
- Very big proposals can be handled differently compared with smaller ones.
- Open Sky Policy
- No time to discuss during this telecon.
- Cost saving
- A lot investigation for construction and operation savings
- Additional financial support from Taiwan?
- Current construction cut plan does not include the items that leads to a scientific loss
- Different operation plans are being investigated. Any decisions were not made yet.
- One call for proposal per year.
- Two months shutdown at summer time?
- Comment: Should not do this. Very sensitive issue.
- Software
- There will be two people from computing coming to Japan.
- Report from region Project Scientist
- East asia
- Two ARC scientists and two CSV scientists in selection
- North America
- Proposal to the NSF: 2012-2015
-Regional SAC
- ESAC
- f2f at March 1st
- ANASAC
- Workshop in fall?: Long wavelength spectroscopy in Astrophysics: ALMA, EVLA, Herschel,...
- EASAC
- ALMA User's meeting last December: ~150 people
- Next meeting
- f2f meeting 9-10 March, Tokyo
- Next Telecon: 12 May 2010; not confirmed
-----
Comments for the Proposal Review Process
*****
1. From Rob Ivison
here are some thoughts on the ALMA Proposal Review Process draft. i've listed them in the order that i encountered the various issues in the document, rather than by importance.
science categories: these appear sensible, both in number and scope, though i couldn't help wondering which box high-redshift radio galaxies might fall in? presumably "Cosmology and the high-redshift Universe" rather than "Galaxies and galactic nuclei". a definition of "high redshift" is all that is required (z>0.1?)
review panels: i can see that panels consisting of 9 members is appealing from the point of view of regional representation. however, in my experience members of panels comprising >6 people tend to assume that other folk will do the hard work and find the important little flaws. i would advise more, smaller panels (which is, apparently, contrary to previous ASAC advice).
125 proposals per reviewer seems, to me, absurdly high.
i like the idea that panel members serve 3 seasons. the biggest complaint about HST TACs is that the churn rate is too high. the biggest complaint about UK TACs is that the churn rate is too low (Chairs are often on panels for 6 years). i think the numbers here are about right.
grading system: seems sensible, with lessons learnt from previous weather-sensitive facilities.
review process: why two secondary assessors? reducing this to one secondary assessor would be an easy way to significantly reduce the overall burden on the assessors and thereby increase the quality of the assessing.
it would help to write down explicitly whether a proposal that is "borderline technically infeasible" can or cannot be "re-written" to some degree by the panel. i don't care either way, so long as the rule applies to everyone.
i would recommend that the science grades for each assessor are normalised to a common mean and stdev, and that assessors are allowed to give a null ranking if they feel it is appropriate for whatever reason.
i didn't get a clear view from the document about how the recommendations of the different review panels will be merged. we should not ring-fence time for themes, so allocations based on the number of panels or the over-subscription rate are to be avoided. i would suggest that the APRC must explore a certain number of proposals near the A/B, B/C, C/D borderlines of each panel and form a conclusion re: their relative merits.
this appears to be weakest part of the document. it ducks for cover when faced with the thorny issue of how to avoid ring-fencing time for themes.
the issue of how time (and good weather) is to be shared accurately between regions is handled simply and straightforwardly.
the formulae for sharing time amongst co-Is appears to be sensible.
i am not happy with the 5% cap on open skies, nor about the idea that NA should pick up the bill if this is exceeded. i do not believe that a cap should be discussed in the document, or that one is justified. however, i am prepared to accept that no-one will give a stuff about what i think on this issue ;-)
*****
2. From Doug Johnstone
Here are my quick hit thoughts on the proposal review process
1) the number of proposal calls per year has dropped from two to one - this may be financially and organizationally necessary but I am not convinced it is best in terms of getting excellent science on the telescope
2) Director's Discretionary Time has not been adequately addressed - how much time will be available ? I think a hard upper limit should be stated here
3) there is a statement that ARC staff will work with proposers of problematic proposals in order to get technical details correct - how will this be implemented, passively or actively ? I'd suggest more careful wording - "ARC staff will be available to work with ..."
4) each reviewer will read 125 proposals - Wow that is an extreme amount. I wonder if anyone will agree to sit on the review panels?
5) grade A proposals will remain in the queue for how long? If there is only one call per year than I see no reason to keep a Grade A proposal in the queue longer than an extra term. All programs go stale eventually and if programs aren't observed after two years they should be rechecked.
6) while not explicit to this document, the grading scheme will need to be watched carefully - dynamically scheduled telescopes can suffer significantly from either hyping expectations or discouraging users.
7) the ARP meetings seem like a huge amount of work for the reviewers
8) the stringency criteria should be available to users writing the proposals (if possible) - might help them choose most observable sources for example
9) not clear how duplications and overlaps will be handled which could lead to arguments later on - perhaps some words of advice here would be helpful
10) also not clear how much leeway there should be to de-scope or change proposals - given only one call a year this may be necessary to get excellent projects with small flaws on the telescope but it also opens up tampering charges
11) why is September 1st chosen as the proposal due date? This seems a horrible date. Is it really the best of a bad bunch?
12) is it realistic to believe that the Mnote-Carlo ALMA Scheduling software will be available in time - this is a very important part of the scheduling process
13) why have Taiwan split its time individually rather than simply put the time in both queues proportionately
14) if a program has under 100 hours and is led by an unaffiliated scientist does it end up in Open Skies?
15) why is a specific amount of time, 5%, set aside for Open Skies - I would have thought this could be left to be determined by the quality of the proposals
*****
3. From Andrew Baker
Summary
-------
The "vagaries of weather" argument for two proposal cycles per year
may not be relevant if the rate of cycling through configurations is
reduced (per recent suggestions from the operations group).
Proposal Types
--------------
A further argument for not allowing Key/Legacy projects early on is
that these make no sense while the array's capabilities are still
evolving rapidly (why spend hundreds of hours early on a project that
can be done in tens of hours later?).
DDT: I would like to suggest that the regional members of the committee
that advises the JAO Director should be appointed by, and report back
to, their respective executives. This was the sentiment expressed
in the last ASAC report.
Joint proposals: I think endorsement of "joint" (interobservatory)
proposals is premature. In North America, I would say that the
programs for joint NRAO/NASA proposals have not been resoundingly
successful; moreover, if access to partner facilities is conditioned
partially or fully on one's home institution (ESO, Gemini, etc.)
then the "great benefit to the community" is not going to be shared
very widely. This point needs more discussion.
Management, Science Categories, ...
-----------------------------------
If past experience is any guide, the fourth category will receive
vastly fewer proposals than the third category. In some respects,
4 x Galactic + 4 x extragalactic ARPs seems like a more appealing
and flexible scheme.
The footnote on election of additional APRC members in the event
that there are more than eight panels is confusing.
The Proposal Review Process
---------------------------
Technical assessors drawn from the pool of "JAO astronomers" should
be primarily operations astronomers, to make sure that AIV/CSV
team members are not overburdened during initial proposal cycles.
I completely agree with Rob that referee grades should be rescaled
for identical mean and variance before they are fed into ARP
discussions. This might impose some degree of control on where
the A/B and B/C cutoffs fall across all ARPs.
ARPs need to be aware of which targets/proposals are implicated
in duplications extending outside their own panels (e.g., so that
they can comment on whether the scientific goals of a given
proposal would be irreparably damaged by dropping one of several
targets).
The statement that duplications, overlaps, and descopes will be
implemented "taking into account regional preferences" is
extremely vague. I would think we can do better than this; in
any case, the details need to be communicated with the proposers
in advance of the deadline.
I am leery of a rigid approach to queue draining: if different
partners have different distributions of stringencies among their
approved proposals (as I think should be allowable), then it's
possible that a run of good weather or bad weather would eat up
all of a partner's quota with proposals of one stringency and
"strand" a few highly rated proposals with different stringency.
Also, if a rigid approach to queue draining is adopted, it's
not clear to me why the HSO needs to monitor and report on the
shares of observing time scheduled for each region.
Accounting of Time to the Executives and Chile
----------------------------------------------
I would favor a scheme like that adopted by IRAM. Proposers
fall in two categories: CoPIs and CoIs. CoPIs (of whom there
can be up to two for IRAM) are assumed to share scientific
leadership of a project. For ALMA proposals, I think we should
allow up to N = 4 CoPIs (one per region), with 1/Nth of the time
attributed to each region, regardless of the number of hours
requested. The long tail of CoIs then becomes irrelevant.
I would argue strongly that proposers with rights to ALMA
through more than one region should *not* be allowed to choose
which region time should be charged to. Instead, time for
proposers from countries like Taiwan should be charged to
the partners according to a formula that is related to the
terms under which access to ALMA has been negotiated. If
there is no limit on the amount of time that Taiwanese
astronomers can claim from either the North American or the
East Asian share, then time should be charged in proportion
to the relative contributions made to North America and
East Asia for construction and operations (however the Board
defines these). If there is a ceiling on the amount of
East Asian time that Taiwanese astronomers can claim, then
the time should be charged "in proportion" until this
ceiling is reached, with the remaining time charged to
North America only. (Note that this is rather similar to
the treatment recommended for open skies proposals...)
Open Skies
----------
I am very unhappy with the undue burden that the North American
share will bear from "overflow" open skies proposals. However,
this seems to be mainly an argument with my own government.
Handling of Duplications
------------------------
Details need to be provided to proposers in advance of the
deadline, not just to the ARPs before they meet.
*****
4. From Munetake Momose
One is about the number of proposals which each panel member should
review. As Rob has already pointed out, 125 per call seems
formidable. If my memory is correct, the original idea about the
proposal cycle was twice per year, at least in the full-operation
phase. If once per year proposal cycle will be adopted, some
mitigation to reduce the number of proposals per each panel member
must be required (e.g., more than 8 ARPs: perhaps each ARP consists
of less than 9 members ?).
The other issue is about proposers with rights to ALMA through more
than one region; this issue was also mentioned by Andrew, and I
would like to support his argument. I believe time for proposers
from a country like Taiwan should be charged automatically
according to some formula which is defined beforehand.
(There is no ceiling on the amount of East Asian time that
Taiwanese astronomers can claim.)
More information about the Asac
mailing list