Peer Review, Diversity, and Equity

Catherine Cocks's picture

A post from Feeding the Elephant: A Forum for Scholarly Communications

As the practice that distinguishes academic publications from every other kind, peer review is supposed to guarantee that only works featuring sound research and rigorous analysis get published. By inviting others to provide feedback on a piece of new research, it is meant to enact a scholarly community created by and for peers with a shared commitment to the intellectual enterprise. Pure, disinterested meritocracy. But in reality the playing field isn’t level and players don’t come to the pitch with equal resources.

Who got published before structures the intellectual landscape of the present and therefore the paths available to researchers now.

By nature and design, peer review is intensely conservative: we evaluate a manuscript in terms of what we already know, and what we already know depends on what got through peer review in the past. Who got published before structures the intellectual landscape of the present and therefore the paths available to researchers now. In short, peer review can and does reproduce inequalities.

What can we do to ensure that peer review functions to encourage sound, challenging scholarship while reducing the effects of social inequalities? 

As a recent study by the Committee on Publication Ethics shows, “Assuring fair representation of new voices and diverse perspectives” (14) and “Recognising and dealing with bias in reviewer comments” (15) are important concerns for journal editors in the arts, humanities, and social sciences. University press acquisitions editors share these concerns. What can we do to ensure that peer review functions to encourage sound, challenging scholarship while reducing the effects of social inequalities? 

Editors can do a lot by being conscious of the way social inequalities have shaped and continue to shape what counts as scholarship and who counts as a scholar, especially the senior scholars preferred as reviewers. We need to deliberately seek a diverse range of qualified scholars and to value a broad range of perspectives. Admittedly, this is not always easy to do and may require extra effort to broaden our networks and become familiar with relevant emerging or marginalized areas of scholarship. Drawing on resources like Women Also Know Stuff (political science), POC Experts Directory (also political science), and Women Also Know History is one way to get started.

...“nothing about us without us” should inform the selection of reviewers as well as the evaluation of the manuscript.

As the AUPresses’ Best Practices in Peer Review handbook points out, who qualifies as an expert depends on the nature of the manuscript being evaluated. Especially (but not only) when a work treats a specific community or a sensitive social issue, the injunction “nothing about us without us” should inform the selection of reviewers as well as the evaluation of the manuscript.

Incompetent reports—those that do not provide an adequate evaluation of the manuscript or are ignorant of the manuscript’s research material, methods, or the relevant literature—should be disregarded.

Another opportunity to make sure peer review functions as intended is in dealing with reports that may be biased or incompetent. Rather than simply passing biased reports along to authors, editors need to evaluate whether they offer any benefit to the project. If not, they should be disregarded and new reviews commissioned. If they do offer some benefit—if only in preparing authors to respond to similar criticisms post-publication—then the editor needs to guide the author in interpreting them and clarifying how the journal or press expects the author to address the reports. Incompetent reports—those that do not provide an adequate evaluation of the manuscript or are ignorant of the manuscript’s research material, methods, or the relevant literature—should be disregarded. In these cases, the editor needs to rethink who counts as a qualified reviewer.

At another level, we can clarify for prospective authors just what “peer review” means and how it will be conducted. The white paper Just Ideas? The Status and Future of Publication Ethics in Philosophy, by Yannik Thiem, Kris F. Sealey, Amy E. Ferrer, Adriel M. Trott, and Rebecca Kennison, offers insights from a series of surveys and focus groups with philosophers and publishers of philosophy journals. The authors write that

while journals often requested that submissions be prepared for anonymous review, more than a third did not publicly specify the actual review process that they follow—for example, how many referees the manuscript is sent to, how many positive judgments are required for acceptance or “revise and resubmit,” or whether one negative review is sufficient for rejection…. Few journals (15.1%) had a clear protocol for how to proceed if a reader or an author suspects unethical activity either in the peer-review process or after publication.

Providing a clear statement of a journal’s policy and practices in an easily accessible place on the journal’s website would address prospective authors’ concerns and offer practical steps to take if the author suspects that something has gone awry.

The Just Ideas? team also notes that reviewers were often uncertain about what the editor wanted to see in a reader’s report, since few journals offer training or a clear statement of their expectations and few graduate students receive any training on this important scholarly practice. Making a journal’s policy and practices public would be helpful, as would incorporating peer review into professional development for PhD candidates.

On the books side, most university presses do offer guidelines to reviewers, though these can vary widely. The acquisitions editor inviting you to do a review can (and should) answer questions about length, focus, and process if the guidelines aren’t clear. Though few presses publish their peer review policies online (as we noted last week in discussing transparency), the AUPresses Best Practices in Peer Review outlines the processes and expectations common among North American university presses. 

The Just Ideas? team also suggests that journal editors (and we would add editorial directors and acquisitions editors at presses) “consider how their current peer-review practices and policies serve their values and goals, and whether experimenting with new models and strategies for peer review might be worthwhile.” To encourage more inclusive practices in citation (thus combatting the historical legacy of inequality), editors can ask “reviewers to include in their evaluation and recommendations whether there is significant but underrecognized scholarship by underrepresented groups that the article may have overlooked or not sufficiently engaged.”

We may also want to examine the way reviewer anonymity functions: does it foster constructive candor, or is it enabling the grinding of axes? Is it not realistic in some smaller areas of inquiry, in which researchers will be able to guess whose work they are reading? Might some other, more open form of review work better in such cases? (We’ll talk more about open review in a later post.)

Finally, the Just Ideas? authors call on scholarly societies and colleges and universities to “set and communicate expectations about how much reviewing scholars should do” in order to alleviate the disproportionate burden of reviews that some scholars in smaller fields or underrepresented groups carry.

Questions for Feeding the Elephant Readers

What else can we do? What other issues do you have with peer review, and what solutions do you suggest? Use the Reply box at the bottom of this post to add your thoughts.

In the next Feeding the Elephant post, we’ll speak with Kathleen Fitzpatrick about her experience of open review and prospects for the future. Stay tuned!

This is all very well and good, but as someone who has been a department chair, served on many tenure and promotions committees both at my own and other institutions, evaluated grant and fellowship applications, reviewed manuscripts journal and book publishers, etc.. I hope we all remember that there is no substitute for reading submissions ourselves rather than relying on the fact that something has been peer reviewed in a highly rated journal. Lots of excellent pieces never make it through and often get rejected for other-than-quality reasons. As scholars, we must never forget that, when we are evaluating people and making important decisions about their future, WE are the most important peers. To be honest with ourselves, we all have experienced the elimination of applicants for positions or awards because the publications on their CV didn't have enough 'heft,' without even perusing them; not to mention totally ignoring unpublished manuscripts. European academics have gone so far as to point evaluation systems which makes it even easier to overlook excellent candidates. This is not an argument against peer review and improving the system, but a reminder of not abusing the privileged position some of us hold by relying on others to make decisions for us.

Thank you for another thoughtful and immensely helpful post reviewing a key issue with peer review. I really enjoyed reading it. Let me suggest that their are some really complicated, underlying issues involving the intellectual authority of an editor over a field of study embedded in some of these proposed policies. For example, the "nothing about us wihtout us" principle is easy to apply in some cases but gets much more difficult in others.  To apply the principle, an editor or group of editors first have to decide which groups it applies to and which it doesn't. It is tempting to point to the principle of under-representation to do this work, but in fact that requires a preliminary set of judgments over the critieria for representativeness. Is it demographic representativeness across all peer reviews at a journal? All peer reviews in a field?  Representativeness within each peer review? Is it relative to the population of the United States? The world? The current demography of a field?  What is the mechanism by which an editor knows of and affirms the membership of an individual within a group? Does the "nothing" part of the criteria mean that an article that offers some quick discussions of several under-represented groups needs one reviewer for each under-represented group mentioned? Doesn't this impose a higher review standard than is applied to articles that avoid comparative methods or address multi-group topics?

My suspicion is that there are many ways in which these various criteria would themselves be subject to debate and would, if truly adhered to, require editors to do extensive work to know the precise composition of their field of study in terms of all sorts of variable (ethnicity, gender, religious affilaition, politics, etc.).  Since many journals and book presses run on a shoe-string budget, a really robust, field-specific understanding of representativeness may be much harder to attain than it sounds. To think this though in one area where I have published, are Mormon scholars under-represented in the field of 19th-century United States? U.S. history more generally? The historical profession? In making that appraisal, how would Mormon historians ex-communicated by the Church be counted (admittedly a more pressing issue a few decades ago)? Does an editor overseeing an essay on Mormon history need to ensure that there is at least one Mormon reviewer of it? Intuitively, I would see this as a less pressing issue than gender or racial diversity, but is my intuition being consistent? And can I really know the gender identity of every possible reviewer? What if some prefer not to share this?

But even as a more general rule-of-thumb, it still poses some problems. What do we do, for example, with cases where the "us" is defined by a long-standing identifier that in fact refers to an internally diverse and changing group of people? If the article being reviewed is about a minority group in 1750, does the review process need to include someone who uses the same term for self-identification in the present? What if a potential reviewer claims that the population being studied in a book manuscript is part of her or his "us" but the scholar submitting the book claims that this is not the case? I could easily imiagine this happening in scholarship on ethnogenesis and on national identities, which many scholars see as "invented" and "imagined." Modern national cultures, some of which we likely would deem to be oppressed and under-represented, sometimes make methodologicallly debatable claims about the antiquity of their group. How do editors handle such claims if dealing with an manuscript about the history of people centuries ago? I'm sure most editors would feel comfortable making some case-by-case decisions, but do we have an explicit principle for dealing with such cases? If identity is fluid, constructed, and contested, how do editors manage competing claims about peoplehood from peer reviewers and authors? Perhaps we're best off having the broad principle of "nothing about us without us" and leaving it to the editors to try to uphold it... but then again, the premise of this discussion is that we cannot fully trust the judgment of the editors overseeing the peer review process.  

Thanks to both Jerome and David for thoughtful replies, and my apologies for being slow to respond. I'm afraid I was traveling last week. Indeed, peer review is not meant to replace other kinds of scholarly and professional evaluation. The AUPresses's handbook of best practices in peer review states: "The review process for proposals and manuscripts is intended to be entirely distinct from any professional review authors may be undergoing. For this reason AEs are strongly discouraged from sharing materials with authors’ hiring, tenure, and promotion committees. Peer reviewers are not being asked to comment on an author’s professional experiences beyond what is conveyed in the proposal or manuscript itself, so repurposing reader reports for any professional situation beyond the publishing world constitutes misuse." (Section 5; David's point about how complicated it can be to identify appropriate reviewers is very well taken and something acquisitions editors grapple with every time they send a project out. The "nothing about us, without us" injunction adds complexity to an already complex process of evaluation. The day-to-day solution, though it can't address every potential issue, is for editors to attend carefully to the context of the manuscript and the field or fields it is contributing to, and to provide thoughtful guidelines for reviewers. As (peer-reviewed) scholarship has shown us frequently of late, we are all subject to unconscious bias; but we are also capable of learning to do better.