As the practice that distinguishes academic publications from every other kind, peer review is supposed to guarantee that only works featuring sound research and rigorous analysis get published. By inviting others to provide feedback on a piece of new research, it is meant to enact a scholarly community created by and for peers with a shared commitment to the intellectual enterprise. Pure, disinterested meritocracy. But in reality the playing field isn’t level and players don’t come to the pitch with equal resources.
Who got published before structures the intellectual landscape of the present and therefore the paths available to researchers now.
By nature and design, peer review is intensely conservative: we evaluate a manuscript in terms of what we already know, and what we already know depends on what got through peer review in the past. Who got published before structures the intellectual landscape of the present and therefore the paths available to researchers now. In short, peer review can and does reproduce inequalities.
What can we do to ensure that peer review functions to encourage sound, challenging scholarship while reducing the effects of social inequalities?
As a recent study by the Committee on Publication Ethics shows, “Assuring fair representation of new voices and diverse perspectives” (14) and “Recognising and dealing with bias in reviewer comments” (15) are important concerns for journal editors in the arts, humanities, and social sciences. University press acquisitions editors share these concerns. What can we do to ensure that peer review functions to encourage sound, challenging scholarship while reducing the effects of social inequalities?
Editors can do a lot by being conscious of the way social inequalities have shaped and continue to shape what counts as scholarship and who counts as a scholar, especially the senior scholars preferred as reviewers. We need to deliberately seek a diverse range of qualified scholars and to value a broad range of perspectives. Admittedly, this is not always easy to do and may require extra effort to broaden our networks and become familiar with relevant emerging or marginalized areas of scholarship. Drawing on resources like Women Also Know Stuff (political science), POC Experts Directory (also political science), and Women Also Know History is one way to get started.
...“nothing about us without us” should inform the selection of reviewers as well as the evaluation of the manuscript.
As the AUPresses’ Best Practices in Peer Review handbook points out, who qualifies as an expert depends on the nature of the manuscript being evaluated. Especially (but not only) when a work treats a specific community or a sensitive social issue, the injunction “nothing about us without us” should inform the selection of reviewers as well as the evaluation of the manuscript.
Incompetent reports—those that do not provide an adequate evaluation of the manuscript or are ignorant of the manuscript’s research material, methods, or the relevant literature—should be disregarded.
Another opportunity to make sure peer review functions as intended is in dealing with reports that may be biased or incompetent. Rather than simply passing biased reports along to authors, editors need to evaluate whether they offer any benefit to the project. If not, they should be disregarded and new reviews commissioned. If they do offer some benefit—if only in preparing authors to respond to similar criticisms post-publication—then the editor needs to guide the author in interpreting them and clarifying how the journal or press expects the author to address the reports. Incompetent reports—those that do not provide an adequate evaluation of the manuscript or are ignorant of the manuscript’s research material, methods, or the relevant literature—should be disregarded. In these cases, the editor needs to rethink who counts as a qualified reviewer.
At another level, we can clarify for prospective authors just what “peer review” means and how it will be conducted. The white paper Just Ideas? The Status and Future of Publication Ethics in Philosophy, by Yannik Thiem, Kris F. Sealey, Amy E. Ferrer, Adriel M. Trott, and Rebecca Kennison, offers insights from a series of surveys and focus groups with philosophers and publishers of philosophy journals. The authors write that
while journals often requested that submissions be prepared for anonymous review, more than a third did not publicly specify the actual review process that they follow—for example, how many referees the manuscript is sent to, how many positive judgments are required for acceptance or “revise and resubmit,” or whether one negative review is sufficient for rejection…. Few journals (15.1%) had a clear protocol for how to proceed if a reader or an author suspects unethical activity either in the peer-review process or after publication.
Providing a clear statement of a journal’s policy and practices in an easily accessible place on the journal’s website would address prospective authors’ concerns and offer practical steps to take if the author suspects that something has gone awry.
The Just Ideas? team also notes that reviewers were often uncertain about what the editor wanted to see in a reader’s report, since few journals offer training or a clear statement of their expectations and few graduate students receive any training on this important scholarly practice. Making a journal’s policy and practices public would be helpful, as would incorporating peer review into professional development for PhD candidates.
On the books side, most university presses do offer guidelines to reviewers, though these can vary widely. The acquisitions editor inviting you to do a review can (and should) answer questions about length, focus, and process if the guidelines aren’t clear. Though few presses publish their peer review policies online (as we noted last week in discussing transparency), the AUPresses Best Practices in Peer Review outlines the processes and expectations common among North American university presses.
The Just Ideas? team also suggests that journal editors (and we would add editorial directors and acquisitions editors at presses) “consider how their current peer-review practices and policies serve their values and goals, and whether experimenting with new models and strategies for peer review might be worthwhile.” To encourage more inclusive practices in citation (thus combatting the historical legacy of inequality), editors can ask “reviewers to include in their evaluation and recommendations whether there is significant but underrecognized scholarship by underrepresented groups that the article may have overlooked or not sufficiently engaged.”
We may also want to examine the way reviewer anonymity functions: does it foster constructive candor, or is it enabling the grinding of axes? Is it not realistic in some smaller areas of inquiry, in which researchers will be able to guess whose work they are reading? Might some other, more open form of review work better in such cases? (We’ll talk more about open review in a later post.)
Finally, the Just Ideas? authors call on scholarly societies and colleges and universities to “set and communicate expectations about how much reviewing scholars should do” in order to alleviate the disproportionate burden of reviews that some scholars in smaller fields or underrepresented groups carry.
Questions for Feeding the Elephant Readers
What else can we do? What other issues do you have with peer review, and what solutions do you suggest? Use the Reply box at the bottom of this post to add your thoughts.
In the next Feeding the Elephant post, we’ll speak with Kathleen Fitzpatrick about her experience of open review and prospects for the future. Stay tuned!