Double blind peer review is widely considered the gold standard for peer review of new research in the arts, humanities, and social sciences, as well as in the STEM fields. As previous posts in this series have alluded, however, peer review has been subject to criticism for its inherent conservatism and gatekeeping functions, and its susceptibility for abuse by unscrupulous reviewers and those who take forever to turn around reviews. Open review in its various iterations is one proposal for improving peer review processes by making reviewers known to the author, to fellow reviewers, or to the wider public. Experiments in open review, though still relatively rare, have launched a conversation about what reviewer accountability and recognition might look like if the invisible service of peer review were made more open and public. At the same time, open review requires more overall labor for everyone involved in the publishing enterprise and may even pose a challenge to authors who are early in their careers or working in relatively narrow fields. Whether wider adoption of open review in the arts, humanities, and social sciences will help improve publication outcomes and scholars' and reviewers' experiences with the peer review process is yet to be seen. Feeding the Elephant spoke with Kathleen Fitzpatrick, a longtime advocate of open review, about her own experiences with the process and where she sees opportunities for improving peer review overall. The interview has been edited for length and clarity.
Yelena Kalinsky: You made the drafts of your last two books, Generous Thinking and Planned Obsolescence, public in order to solicit review from the scholarly community, and your publishers also sent your manuscripts out for traditional review. Can you talk about the different kinds of feedback you received from those two forms of review?
Kathleen Fitzpatrick: Absolutely! The kind of response that I got online to Planned Obsolescence was, as you might expect, pretty open. There were about forty commenters who came from a vast range of different areas of the academy. Some of them weren’t academics at all and were just really interested in the topic. Overall the comments were really supportive and generative and designed to help push the thing forward. They were like the comments on a blog post, in which there’s a conversation taking place between the author and the commenters.
The comments that I received through the traditional peer review covered the entirety of the manuscript. They took a slightly higher-level view of the structure and addressed things like, Are these the right chapters? Are they in the right order? But there were only two of them and those two reviewers couldn't cover everything in the way that 40 commenters were able to do online. The two sets of reviews work together well and they really did help me see what I needed to do.
With Generous Thinking, Johns Hopkins was interested in my doing this again and seeing whether I could involve a community. This time some of those commenters took a broader view. They would pick a point locally in the manuscript and ask large structural questions about the project as a whole that pushed the review in some important ways. The manuscript did go to at least one outside reviewer, but the press ultimately decided that the 30 commenters were sufficient for peer-review and took the project to the editorial board that way. It was really a positive process for me and some of that has to do with a particular community that is interested in the kind of work that I'm doing and some of that has to do with the feeling of openness that kind of review generated from the reviewers.
I think that for an open review process like this one to be successful the work needs to be addressing a community that preexists the review process. But at the same time, the review process can be used to build out that community, to bring more people into it and to make it more inclusive.
YK: This issue of community is really crucial. In a recent blog post, you write about peer review as “an act of attention to the work of another that makes a powerful contribution to...its community of practice.” It seems like peer review is actually just one piece of the puzzle when it comes to maintaining one's scholarly network or community of practice. What kind of preparatory work within your own scholarly network have you found useful to encourage a productive, generative, open review process?
KF: In both cases, with Planned Obsolescence and with Generous Thinking, I had been blogging on and off about the project as it was in development. I had a group of people who had commented on some of those blog posts and with whom I had been in conversation both via the blog and on Twitter. Those were the folks that I first reached out to when I started this review process. It was important to me to have that readymade community, but they also then passed on what they were doing to others. News of the review spread through networks and people came in to comment with whom I wasn't familiar. I think that for an open review process like this one to be successful the work needs to be addressing a community that preexists the review process. But at the same time, the review process can be used to build out that community, to bring more people into it and to make it more inclusive.
YK: I was thinking about how different scholarly communities might use open review. In the case of a scholar transforming a dissertation into a first book, that scholar may still be building a community. They may feel isolated from their peers geographically or due to a job outside of academia. Can folks working on more specialized topics or ones with narrower audiences or smaller networks take advantage of open review?
KF: I do think that authors working in narrower areas or with smaller communities of practice can take advantage of open review. It may take more effort on the part of the reviewers, but those narrower communities are often tighter because of their shared concerns. It may sound a little less impressive to say I had five commenters who read the manuscript and left me a bunch of really good comments and I improved the manuscript, than it is to say I had 40 people come in and read it. But if you can demonstrate that those five are the right five, then that's exactly the community that it should be. Size of audience is not really the key driver of openness in this sense, it's more about getting the right people with the right ideas into the space.
YK: It seems like a large part of this is on the scholar to grow their community, to reach out to their community. Where do the editor and the press fit into this?
KF: In my case, my Generous Thinking editor did a lot of retweeting and helping to spread the word about this set conversations, which indicated that the press was behind it and enthusiastic about it. It mattered a lot, I think, to people being willing to take the time.
There is an enormous variety of flavors of open in open peer review...and figuring out for a particular publication, for a particular community, for a particular publisher, what the best combination is...is important.
YK: In 2011-12, you worked on a white paper with MediaCommons and NYU Press about the "technologies, practices, and desires for open, online peer-to-peer review in humanities-based scholarly communication.” Can you talk about what you learned about scholars’ needs and expectations about peer review and online open review through that process?
KF: Yes, absolutely. We launched into that white paper process planning to come away with a set of best practices or guidelines for how these things should be done that other publications and authors could take forward. And by the time we got even a quarter of the way into the study, we realized that was profoundly misguided. What we’re going to come out with is a whole bunch of questions for authors and publishers and communities to ask themselves about what it is they’re trying to do and what the best way for them to do that is.
Part of what we realized is that there is an enormous variety of flavors of open in open peer review. It can be, as in the case of the journal Kairos, a very large editorial board that discusses together the work that’s submitted to them and then puts together a subgroup of the editorial board that works directly with the author. Or it can be, as in the case of Planned Obsolescence and Generous Thinking, a community-oriented, totally public, with everybody’s name attached discussion online. And then there are other things that can happen in between. For the first volume of Debates in the Digital Humanities, everybody whose work was going to be in the volume got put into a community where we were all commenting on one another’s work, but it was only visible to the folks who were going to be in that volume, and so everybody was super motivated to do it because they wanted the volume to be as good as it could be. So it was a supportive closed--but open to the group--process. So there are a bunch of different flavors, and figuring out for a particular publication, for a particular community, for a particular publisher, what the best combination is--complete transparency, or behind password wall, or complete anonymity, or totally open and everybody’s name is attached to everything, and all the gradations in between--is important.
There’s a lot more recognition that review needn’t be just one particular moment--that it can happen in different ways and at different points in the development of the work.
YK: Eight years out from Planned Obsolescence, I'm curious where you see open review getting taken up (if it’s being taken up), and how much of the change you called for is underway?
KF: I think it’s developing slowly. I am hearing more and more about journals and other kinds of publications that are adopting more open practices, but they’re doing it cautiously and I think for good reason. Finding ways to ensure that a community is on board before making that leap is important. There’s a lot more recognition that review needn’t be just one particular moment--that it can happen in different ways and at different points in the development of the work. One of the challenges in making more review more open is just the ongoing labor problem. It’s difficult for journal and university press editors to find two to three people with the time and the willingness and the inclination and the generosity to sit down and read something and give it their attention. It’s hard to mobilize; it’s even harder to mobilize an entire community. When you start mobilizing more of a community to do more stuff, it becomes exponentially more difficult. But I do feel like the value of open review processes is becoming more visible, and that’s gradually starting to change the ways that we look at peer review in general. And recognition that peer review can be a whole lot of different things is a good thing.
YK: Speaking of the labor problem, in 2006, you speculated about the possibility of decoupling peer review from the function of credentialing and devoting the time spent on peer review into "open discussions that not only improve the individual texts under review but are also, potentially, productive of new work.” In some ways, the transformation of labor in higher education has created a much more hybrid workforce, for whom the traditional marker of credentialing (tenure) has disappeared. Have you noticed a concomitant change in peer review, such as more openness to open review, or different kinds of scholarship that people put out, a more hybrid scholarly ecosystem?
KF: I think there has been a kind of change. I don’t want to pin this to the decline in the percentage of tenured and tenure-track faculty within institutions like ours, because I don’t want to make anything about that decline sound like it’s a good thing. It’s not. On the other hand, I do think that there are a lot of folks with PhDs in academic fields who have for their own reasons decided to go in different directions with their work. They’re still within the academy, but they’re not in conventional positions that have all the traditional markers and requirements for the kinds of credentialing that peer review can provide. And I think a lot of those folks look at the traditional journal and university press publishing practices and say, you know, maybe that’s not the best thing for my work. Maybe what I need to do is just make the work as public as possible, and I can reach more people through this online platform where I publish everything and then they comment on it and then I revise it. Or I can publish in more popular venues rather than in academic venues and I can reach much larger audiences that way. I think the possibilities for what publication is and what benefits it provides to a career have opened up as some of those strictures about the very specific requirements for peer-reviewed scholarship within the academy have come to apply to a smaller and smaller percentage of folks in the academy.
All of that having been said, I still think we need to take a really hard look at those requirements for tenure and promotion, because they help to distance the tenure-stream faculty from everybody else working on campus. And they don’t always reflect the real values that we bring to the field, the real reason for communicating the work and the goals of writing and publishing. So I don’t know if I’m quite willing to say today, thirteen years later, that we need to totally decouple credentialing from the peer review process. But I do think that there are some benefits to looking at those two processes as being related but different, and thinking about how peer review can help create better work, and credentialing as a whole other thing that sometimes involves knowing that people have read this stuff and have helped it become better.
Feeding the Elephant readers, share your experiences with open review--either as editors or as authors. How did you mobilize reviewers to comment on the work? Did tenure and promotion (or lack thereof) make a difference in your decision to pursue open review (or not)? Does open review offer potential solutions to peer review's worst offenses, like lack of transparency, unconstructive criticism, and gatekeeping? Will you seek out open review in the future? Use the Reply box at the bottom of this post to join the conversation.