Building Trust: Defining Subject Matter Expertise through U.S. Federal
Peer Review Policy
Abstract
There is no quantifiable definition of what comprises a peer reviewer,
leading to a lack of trust in the outcomes of peer review by the public
and some government officials. Using contextual content analysis, this
study uncovered concepts from federal science policy and agency peer
review guidance documents to begin the creation of such a definition.
The first stage determined which term is most often used for a peer
reviewer as seen in the U.S. Office of Management and Budget’s (OMB)
Final Information Quality Bulletin for Peer Review 2005. The second
stage analyzed the U.S. Department of Energy, Energy Efficiency &
Renewable Energy, the U.S. Department of Interior, Bureau of Safety and
Environmental Enforcement and the U.S. Department of Justice, Office on
Violence Against Women peer review guidelines to determine the number of
mentions of each concept (knowledge, skills, experience), if a
definition of a merit peer reviewer is provided, and descriptors
associated with qualities, or sub-concepts pertaining to peer reviews
including discipline, teaching, education, postdoctoral positions,
certifications, grant funding, publications, presentations, awards and
consulting work. Stage one results found that the term reviewer is the
most used in describing a peer reviewer for all three federal agencies
and the OMB bulletin. Results of stage two show that both knowledge
(M=9) and experience (M=14.33) categories were the largest reported with
few mentions of skills (M=3.33). Each agency provided a definition of
reviewer and qualifications they must meet. This analysis of three
federal peer review guidance documents shows that the federal guidance
on expertise, if measured by both experience as well as knowledge, is
being followed. It would be a worthy effort to do an analysis of the
publications, experience and knowledge of those selected for peer review
and compare it to each agencies criteria to build a profile of reviewer
characteristics. In addition to contributing to the scant literature on
peer review, these findings will help narrow the qualities merit peer
reviewers need and will assist with future research exploring all U.S.
Federal Agency peer review manuals in which qualities in both knowledge
and experience could be quantified, helping to defining an accurate
measurement of goodness of fit for reviewers.