Sunday, August 31, 2008

Peer Review, or Who the Hell Do I Think I Am?

So I just finished reading, writing the reader's report for, and submitting the reader's report for a manuscript that I agreed to evaluate for Very Good Journal, and I'm left feeling... Oh, I'm left with a lot of feelings.

First, can I just say that this is service I really, really like. In theory. Even though it's basically invisible. I like reading stuff that may be the Newest Thing in my field, I like feeling like an expert who has something to contribute to scholarship seeing the light of day, and I like feeling like I can offer feedback that writers can actually use.

But. I've done this now exactly two times (for I'm not all that much of an expert), and in both instances I recommended that the articles be rejected. And that kind of nullifies most of what I listed above about liking this kind of service. Because it sucks to know that I may be responsible for somebody not getting a publication. I don't like that at all. And it also sucks to wonder whether my thoughts about the relative publishability of something are fucked up, and to wonder whether I'm being an asshole, and if I am, whether that actually makes editors think that I'm an asshole and so they won't ever ask me to review again. And in the case of the report that I just finished, the journal does double-blind review, and so what if I was an asshole to somebody fancy? (Though I really don't think somebody fancy would have submitted something that cited - or, rather, failed properly to cite - Wikepedia, so that's some small consolation.) To make a long story short, this whole reviewing manuscripts thing for me is bound up with a lot of my own insecurities and problems with dealing with criticism.

See, I hate nothing like a reader's report. Even the good ones make me shudder upon first looking at them. I hate the impersonal tone - "the author offers insightful analysis of x"; "Crazy fails to address y." I hate that feeling of dealing with the suggestions for revision, like I'm attempting to read a map in the dark or like I'm trying to use telepathy to figure out what some ethereal entity wants from me. I hate that feeling that sometimes happens where one wonders whether the person actually read the essay that I wrote (though admittedly, I've only really felt that one time). My problem is not with complying with the comments so much - for I am typically fully willing to change anything I've written in the service of it being published and have no love affair with my sentences or any such thing - but with the fact that the comments exist at all. I hate reading what strangers think about what I've written. Absolutely hate it.

So when I read a manuscript at the request of an editor, I bring all of that baggage with me. And in some ways, that's probably good, because I really am committed to trying to offer practical advice, even if my recommendation is to reject. I care a lot about being a useful reviewer, and I care a lot about offering suggestions that somebody can use going forward.

But so what? Somewhere, some academic, who really did have an interesting idea even if the execution wasn't so grand, is ultimately going to read comments from a hostile reviewer, and that hostile reviewer will be me. Even though I wasn't really hostile: I was disappointed. I so wanted to be able to say, "this is the most fantastic essay in the land! publish away!" But instead, well, I couldn't in good conscience even offer a revise and resubmit. It just wasn't good enough, or so I thought.

But who the hell am I to make that kind of decision? When I look at myself, at my CV, whatever - I still don't feel qualified. I feel like a total fraud. Because sure, I've had some modest success with publication or whatever, but who the hell do I think I am?

I think what makes me feel even more... insecure... about this most recent report is that I've currently got an essay out for review with this journal. And so I imagine getting back reader's reports about my own essay that are as harshly critical of my work as I was of the manuscript that I read. And I imagine the editor reading my report and then getting reports back that my own essay is a piece of garbage and deciding that I'm a bullshit scholar who doesn't know anything. And yes, all of this has nothing to do with the manuscript that I reviewed, but I sure would feel better if I'd already heard back about my own essay. (Apparently one report is already back and the other is overdue; the overdue thing worries me, because I feel like it means that the person hasn't had time to articulate all of the ways in which I suck but with substantial revision could maybe be acceptable, but of course what it probably means is that the person just hasn't bothered to look at it yet and the overdueness means nothing about my work, positive or negative.)

But so the point of all of this is that it is totally weird to be in a position where one is called upon to review the work of others when one is subject to the same review of one's own work. If you let yourself think too much about it - which I don't recommend people do, so you may actually want to skip this part - probably the same 20 people are all reviewing each other's work at the same time, passing judgment on one another simultaneously. How horrifying is that?

And yet, the report is done. That is one thing I can check off of my list of things to do, and ultimately, I do think that my response to the manuscript was fair and that I offered some suggestions that would make the essay hang together better, make the argument tighter, and make the thing really a super-interesting piece of scholarship. And maybe my lack of confidence when it comes to thinking about my authority in this area will not last forever. Maybe at some point I'll actually feel qualified to do this work. Maybe it's like teaching - that the more you do it the more authority one feels. One can only hope.

But it still sucks to know that I'm a person who could be responsible for somebody else's professional disappointment. The idea was really a good one. I hope the person realizes that I thought so.


life_of_a_fool said...

my favorite part about peer reviewing is when the journals send the reviewers copies of the final decision and the other reviewers' comments -- because then I can see how my comments matched the others, and hopefully be reassured that I am not a complete idiot.

I agree it's awkward to review something for a journal that's currently reviewing your own work (though from a journal's perspective, how could one possible say no to doing a review in those circumstances?). It's also awkward to do a review for a journal that's already rejected you -- which I've now been asked to do twice this year. I would like to think this means that even though I was rejected, they don't think I am a complete idiot. More likely, it probably just means they need reviewers, but whatever. They wouldn't ask if they thought I was a complete crackpot, right?

Earnest English said...

This is what I love about your blog, Crazy. That now that I'm sending out my work, I get to imagine that all the people who reject my work feel as badly about it as you do! =)

gwinne said...

I really appreciated this post! Out of five essays that I've placed recently (all with varying degrees of revision required), only one left me feeling good about the process. And that was an essay that required *substantial* revision before it was accepted. I would have done anything for that editor because she was so *nice* in communicating with me. I think what I'm saying is even negative reviews can be framed by the editor in such a way that the writer doesn't end up feeling demoralized. It's the nasty editors who do the real damage, at least in my experience. I will never again submit to a major journal in my field--I have an essay forthcoming--because the process made me feel worse than any time in my career, even if the outcome was publication.

Liz said...

They cited wikipedia? Just as an example of somethng in pop culture or did they really think it was a worthy source of information?

Dr. Crazy said...

As a source of information. For a fact that they could have easily confirmed with a more viable source.

You know, I'm not against wikipedia for a lot of things, but I do feel pretty strongly that a journal that accepts less than 15% of submissions probably shouldn't accept articles that use wikipedia as if it is a strong source of fact. At the very least if you found the information first on wikipedia you should hunt down a better source to be the one that you cite. But maybe that's just me.

(Unless of course one is analyzing something on wikipedia as a primary source. That, to my mind, would be a different thing.)

Dr. Crazy said...

I should note, the wikipedia thing was the least of the problems that I had... but that was the icing on the cake.

helenesch said...

The Wikipedia citing does seem weird!

Keep in mind that if they accept less than 15% of submitted essays, you're in the majority in judging that the essay be rejected. So you shouldn't feel bad!

And at least you *read* the essay carefully, and somewhat charitably. What I find most frustrating is getting back very short reviews that make it clear that the reviewer hasn't carefully read my paper, or that she/he isn't really working in relevant areas. This seems to happen all too frequently.

Dr. Crazy said...

EE: I aim to please! :)

Gwinne: You're so right that how the editor frames the reports makes all the difference in the world. I've been really lucky in this regard thus far.

Helenesch: I have a suspicion that perhaps what I read was a heavily edited down version of a not-yet-passed diss chapter or of an MA thesis. A lot of things make me think this, in combination with the wikipedia thing, notably an "as I argued elsewhere" in the manuscript that didn't actually refer to anything that was argued elsewhere within the essay and that didn't cite any previously published thing. There were other things as well that made me suspect this. Now, I could be wrong about that, but it's my strong suspicion. But you are so right about the 15% thing! I am normal! The journal has high standards! Most things are rejected! I'm not a total meanie! (And really, I know I'm not a total meanie, otherwise I'd not have read so carefully or offered actual suggestions for revision even though I recommended "reject." I mean, I was hard on the essay, but I wasn't nit-picky, and I did try to be constructive even as I pointed out what I think are clear problems. But you're so right: it's horrible when the review is super-short and/or clearly unfamiliar with the area of the essay. At least I was guilty on neither of those counts :)

Psych Post Doc said...

I try to combat feeling badly about writing a rejection review by putting all the good stuff about the manuscript upfront.

I'll start by listing the strengths and the things I like about the manuscript, in hopes that it takes the sting out of the comments regarding what wasn't up to par.

Susan said...

I know the sometimes icky feeling of recommending rejection, but I also think of myself as a reader of journals who sometimes says, "HOW did that get published?"

If you give constructive advice, that's great. Though I'm with you. I had reading readers reports almost as much as I hate reading student evaluations. If they don't say I walk on water, I have real trouble dealing with it!

Dr. Crazy said...

Psych post doc: You know, I wish I were the sort of person who could use praise to soften the blow. The fact is, if I had substantial praise to offer, this thing would have gotten a revise and resubmit. As any student of mine would tell you, I'm not great with the positive feedback. Not that I don't give it, but I'm much more oriented toward focusing on improvements as opposed to accomplishments. I don't think this is necessarily bad, but it does mesh better with some personalities than others.

I think a lot of this goes back to my own feelings about dealing with responses to my work. I don't do particularly well with positive reactions, because if they're followed up by negative ones I think all the positive stuff was a lie. I'm suspicious. This isn't how all people operate, but it is how I operate. So that informs my criticism of others, for better or worse.

Susan: I agree about the feeling of "HOW did this get published?" Which probably colors my responses somewhat, but which also makes me feel like a jerk, if that makes sense, as so many people get published with standards that I wouldn't approve. I've got to say, though, even if they think I'm walking on water, I hate the reports. See above :)