An Empirical Look at the Peer-Review Process

"Everybody talks about the weather," Mark Twain once famously declared, "But nobody ever does anything about it." He then ate a knife to show the seriousness of his statement.

The same sentiment applies to reviewers in academia. Young researchers are frequently traumatized by their first exposure to the peer-review process, in which a paper is scrutinized and criticized by one's colleagues, ostensibly to judge whether the paper is acceptable for publication and to suggest improvements. However, as every one of us scientists intuitively understands, in reality it serves as nothing more than a vicious catharsis for the shattered dreams and failed career and (probably) frustrated sex life of the reviewer, unable to distinguish between his personal neuroses and legitimate criticism, by ruthlessly excoriating an otherwise brilliant exposition of scientific reasoning and experimental methodology. This trauma is then propagated along to other reviews in a never-ending cycle of misery and despond, with the abused becoming the abusers. Conrad summed up this mentality in his book The Secret Agent, in which he characterized a particularly violent group of terrorists plotting against a repressive government as not necessarily opposed to tyranny per se; they were merely unhappy that they were not the ones doing the tyrannizing.

The whole review process has been thoroughly criticized by many, but one of the first systematic exposés of peer-review was done back in the 1980s by two young researchers. To better understand their motivations, let's turn the clock back a few decades to the United States of the '80's: Top Gun, one of the supreme achievements of world cinema, had just been released in theaters to universal acclaim; cocaine flowed like crack through the streets of America; and the nation was beginning to feel its oats after successfully invading and destroying the hated island of Grenada. Life was so good, you could taste it in your spit.

However, even in the midst of all this success and prosperity, problems with the peer-review process still loomed large in the minds of the public. This is where our two young heroes, a couple of spitfire hoydens named Peters & Ceci, entered the picture, intent on producing concrete evidence of systematic bias among reviewer. Although subsequently tortured and murdered for their heresy by the peer-review mafia, their publication still circled clandestinely among groups dedicated to changing how the review process works.

Before their ghastly deaths, Peters & Ceci had taken papers previously accepted for publication and changed a few trivial details on the manuscript - for example, changing the author affiliation from a prestigious university, such as Yale, to a less prestigious-sounding, totally fictitious institution, such as the Nelson Center for the Study of Bodacious Ta-Ta's. Everything else, including materials and methods, remained exactly the same. After resubmitting these manuscripts, surprisingly few were reaccepted; in fact, out of the nine journals sampled, only one accepted the same papers it had accepted earlier! Keep in mind that these were all papers that had been accepted at respectable journals; virtually nothing had been changed, except for trivial details that had no bearing on the quality of the reported results. Reviewer comments were focused on methodological details ("Analysis of variance was improperly used in a post-hoc fashion"), writing style ("The theoretical focus of the introduction could be tightened up considerably"), and even, incredibly, features of the individual authors ("Just where can I meet this Nelson fellow?").

On the face of it, these results may seem to be cause for suffering and despair, as we find ourselves at the mercy of only one more facet of an openly hostile universe. One may well ask whether anything can be done, or whether it would instead make more sense to curse God, and die.

I offer a few suggestions. First, make sure that you are at a prestigious university; and if you are not, begin to associate with people who are at prestigious universities. Offer to make them authors on your paper, even if they haven't done anything. Ingratiate yourself with well-known figures in your field, even going so far as to openly debase yourself by performing menial tasks for them, or by humiliating yourself by offering to eat one of their used tissues. By stripping away any vestige of self-respect, you will increase your chances of their agreeing to be a co-author on your paper, as long as they don't have to do any work and never have to see you again. This, in turn, will increase your chances at publication.

The word sycophant has had negative connotations, long associated with spineless, weak, craven poltroons who would not hesitate to pawn their souls for the most trivial of earthly gains. Our job is to make it a positive word, associated with success, advancement, and - above all - publication.

EPILOGUE

Ceci and Peters laid on the grimy floor of their cell, wasted, defeated, only a putrid dish of water sitting between them. Their eyes were hollow and sunken and their frames emaciated, their faces dirty and wizened, their hair wild and bedraggled. Ceci's eyes roved around the room and he gibbered in an incomprehensible argot, thin strands of drool running down his cheeks in viscous rivulets. They had not eaten in five days and they awaited the hour of their execution with neither apprehension nor resolve but exhaustion.

There was the sound of a key turning in the door and an enormous Swede entered the room, holding a maul in his right hand. He seemed to fill up the entire room with his frame and he had two little pig's eyes set into the sockets of his skull and a cheesy rictus pasted on his face like a doll's. When he moved he gave the impression of a boulder set into motion. He gestured with the maul for Peters to get up but Peters merely glared at him and would not move. The Swede seemed to expect this and he bent down and picked him up by the meatless bone of his arm and dragged him to his feet.

Peters spit in the face of the Swede but it seemed as though it was all part of his day for he did not blink or wince but merely smiled, his right hand raising up slowly and methodically and then snapping down like the lid of a box and the maul made a sound like hitting a pumpkin with a club. Peters' eyes bulged as he collapsed to the floor like his bones had turned to jelly and the blood began to stream out of his ears.

The Swede bent down to look at him and watched as the capillaries in his eyes began to break up. Ceci's eyes darted zigzag across the room, his mouth open and his lower jaw flapping up and down uncontrollably. The Swede looked over at him and smiled.

*   *   *

The next day their lifeless bodies were dumped in a shallow arroyo along with everyone else who had dared to challenge the might of the peer-review process. For the next two days the air was filled with the whine of flies and the wolves and buzzards wallowed in the carrion, and when the feast was over there was nothing left save for their bleached skulls and the bones of their briskets curving in upon themselves like grotesque half-closed traps.

FIN

Link to the paper can be found here.

Thanks once again to Keith Bartley, who is quickly becoming an Andy's Brain Blog VIP. VIP status includes a free year-long subscription to the blog, shampoo samples, Nutella aroma therapy, and a summer vacation to North Dakota, including a tour of the legendary sugarbeet processing plants, a pitchfork steak fondue, and tickets to the one-hour acting monologue Bully! The Teddy Roosevelt Story.