Proposal review rankings: the influence of reviewer discussions on proposal selection

The telescope time allocation process for NASA's Great Observatories involves a substantial commitment of time and expertise by the astronomical community. The annual review meetings typically have 100 external participants. Each reviewer spends 3-6 days at the meeting in addition to one-two weeks of preparation time, reading and grading proposals. The reviewers grade the proposals based on their individual reading prior to the meeting and grade them again after discussion within the broad, subject-based review panels. We summarize here how the outcome of the review process for three Spitzer observing cycles would have changed if the selection had been done strictly based on the preliminary grades without having the panels meet and discuss the proposals. The changes in grading during the review meeting have a substantial impact on the final list of selected proposals. Approximately 30% of the selected proposals would not have been included if just the preliminary rankings had been used to make the selection.