Sunday, January 29, 2017
Are All FMRI Results Wrong?
False positive rates in science have been an issue recently; and although we all had a good laugh when it happened to the social psychologists two years ago, now that it's happening to us, it's not so funny.
Anders Eklund and colleagues published a paper last summer showing that cluster correction - one method that FMRI researchers use to test whether their results are statistically significant or not - can lead to high false positive rates, or saying that a result is real, when actually it is a random occurrence that looks like a real result.
Their calculations showed that about 10% of FMRI studies are affected by this error (http://tinyurl.com/jaomsgs). However, keep in mind that even if a study is at risk for reporting a false positive, doesn't mean that their result is necessarily spurious. As with all results, one must go to the original study and take into account the rigor of the experimental design and whether the result looks legitimate.
These flaws have been addressed in recent versions of AFNI, an FMRI software package. The steps to use these updated programs can be found on the blog here: http://tinyurl.com/j5vafsb
Thursday, January 19, 2017
Commentary on Cluster Failure: Inflated False Positives in FMRI
Why did the old Folly end now, and no later? Why did the modern Wisdom begin now, and no sooner?
-Rabelais, Prologue to Book V
=======================
Academia, like our nation's morals, seems to be forever in peril. When I first heard of the replication crisis - about how standards are so loose that a scientist can't even replicate what he ate for breakfast three days ago, much less reproduce another scientist's experiment - I was reminded of my grandpa. He once complained to me that colleges today are fleshpots of hedonism and easy sex. Nonsense, I said. Each year literally tens of students graduate with their virtue intact.
Nonetheless, the more I learned about the replication crisis, the more alarmed I became. You know a crisis is serious when it has its own Wikipedia page, and when its sister phrase "questionable research practices" gets its own acronym: QRP (pronounced "Quirp"). These practices range from data "peeking" - ogling a p-value before she's half-dressed - to fabricating data, one of academia's gravest sins. (Apparently grant agencies have gotten very picky about researchers lying about their results and wasting millions of dollars.) One of my friends suggested campaigning for a War on QRPs, with scientists in white labcoats stamping out QRPs like cockroaches. Who knows - someday QRP Exterminator may become as respected and honorable a position as Lab Manager.
One such QRP was recently flushed out of hiding in an article by Eklund and colleagues published in the journal PNAS (pronounced "P-NAS"), with the klieg lights thrown on cluster correction, the most common FMRI multiple comparison technique. If you're in a neuroimaging lab, or if you've read about FMRI in a magazine, you've come across these clusters - splotches of reds and yellows, color-coded to show the strength of the result. They look attractive, especially on glossy magazine pages, and they make for good copy.
There is another reason for the popularity of clusters: As each FMRI dataset contains tens or hundreds of thousands of voxels, and as each voxel is similar to its neighbor, it is useful to test whether a cluster of voxels is significant instead of each voxel individually. This increases the sensitivity of detecting activation, although at the cost of reduced spatial sensitivity. If I point to a cloud of smoke in the distance you will assume there is a fire somewhere, although seeing the cloud itself gives you only a vague idea of where or how big the fire is - or, for that matter, whether its the result of a dozen individual campfires, or of a single inferno.
Problems with Cluster Correction
The authors don't take issue with cluster correction per se; rather, they claim that the assumptions behind the technique, and how the correction is used, can lead to inflated false positives. In other words, using cluster correction makes one more likely to say that a cluster represents a true activation, when the cluster is actually noise. The assumptions they question are:
1) That spatial smoothness is constant over the brain; and
2) That spatial autocorrelation looks like a Gaussian distribution.
Spatial smoothness is in fact not constant; it varies over the brain. Midline regions such as the precuneus, anterior cingulate cortex, and prefrontal areas show more spatial smoothness than the the outer regions - possibly because the midline areas of cortex, with the gray matter of the hemispheres in close proximity to each other, have similar structure and similar timecourses of activity which are averaged together. The underestimation of spatial smoothness in these areas can lead to inflated false positives, which may be why midline areas (such as the anterior cingulate cortex) turn up as a significant result much more often than other areas.
It is the second assumption, however, which is the main cause behind the increase in false positives demonstrated in the Eklund paper. The idea behind spatial autocorrelation is simple: A voxel is more similar to its immediate neighbors and less similar to voxels farther away, the similarity falling off in concentric spheres around a given voxel. This correlation between each voxel and its neighbors is traditionally modeled with a Gaussian distribution - the same bell-shaped curve that models most distributions, such as IQ or height - when in reality the autocorrelation distribution has much thicker tails:
When the authors ran simulations on large datasets under different experimental conditions (e.g., block design and event-related), they found that this assumption of a Gaussian shape led to false positives far above the expected 5% rate across all the major software packages (SPM, AFNI, and FSL). The commonly used p = 0.01 threshold for the individual voxels of the cluster led to much higher false positive rates than a more stringent p = 0.001 threshold. (Notably, permutation testing, a nonparametric method that does not rely on assumptions of normality, did not show any substantial difference in false positives under any conditions.) These inflated false positive rates led to the following figure, which was picked up by the popular media and discussed in a fair, evenhanded way which emphasized that although false positive rates could be increased in some cases, it would be hasty to conclude that all neuroimaging results are wrong.*
*Or not.
Solutions
There are three solutions to this problem; solutions to make neuroimaging respectable again, to cease being laughed at as cranks, charlatans, raiders of the public fisc, and attention-seeking, unscrupulous bloggers.
One solution is to use voxel-wise thresholds; although, as shown in the Eklund paper, these are often far too conservative, and can lead one to conclude there are no results when they actually do exist. I, for one, cannot stand the thought of those poor results slowly suffocating in their little gray coffins, alive but unable to signal for help. With voxel-wise thresholds we stop up our ears with wax; we think we hear the dim scratching of nails, the whimpers for succor, but dismiss them as figments of a fevered imagination; and we leave those tiny blobs and their tiny blob families to know that their last moments will be horror.
The second, more humane solution is to use more conservative cluster-forming thresholds, such as p = 0.001 uncorrected, in conjunction with more accurate modeling of the spatial autocorrelation. I recommend updating your AFNI binaries to the latest version (i.e., the beginning of 2017 or later), and using the -acf option in 3dFWHMx. For example:
3dFWHMx -mask mask+tlrc.HEAD -acf tmp.txt errts+tlrc.HEAD
Which will generate parameters for a more accurate estimation of the empirical ACF; you will see something like this:
++start ACF calculations out to radius = 37.68mm
+ACF done (0.74 CPU s thus far)
11.8132 13.748 12.207 12.5629
0.260939 6.36019 16.6387 19.1419
The first line of numbers are the traditional smoothness estimates in the x-, y-, and z-directions, as well as a weighted average of them (bolded). The second line of numbers are the a, b, and c parameters for the ACF model,
a^(-r*r/(2*b*b))+(1-a)^(-r/c)
With the last number being an updated estimate of the smoothness. These are then input into 3dClustSim:
3dClustSim -acf 0.26 6.36 16.64 -mask mask+tlrc.HEAD -athr 0.05 -pthr 0.001
The difference between the two smoothness estimates leads to substantial differences in significant cluster thresholds. In this case, the traditional estimates lead to a cluster threshold of 668 contiguous voxels, while the ACF version gives a threshold of 1513. This was using an initial cluster-defining threshold of p = 0.01, which can lead to such large differences; more stringent thresholds (e.g., p = 0.001) have less of a difference, but still a significant gap between them.
The third solution is to use non-parametric methods, such as permutation tests; which, since they do not rely on assumptions of normality, are immune to the autocorrelation assumptions listed above. Some popular tools include SPM's SnPM (Statistical non-Parametric Mapping); FSL's randomise; and Eklund's BROCCOLI. Out of the three, I've only used randomise, and found it straightforward to use (see the tutorial here).
Conclusions
Have we been profiting too long from dubious methods in neuroimaging? Have we become lax and complacent in our position as a period science, knowing that no matter how poorly designed the experiment is, no matter how ludicrous the question we are investigating - such as where the word YOLO is in the brain, or how the consumption of Four Loko affects delayed discounting - no matter whether our subjects are technically alive or not, we will probably still get a significant result, most likely in the precuneus?**
The answer to all of these, I would say, is yes. But just because I was able to get the results of my first paper to squeak by with an outdated version of AlphaSim, and consequently have an easy time in graduate school, make tons of money, and be envied by all of my friends, doesn't mean that you should have the same privilege. The way forward will be a hard one for the neuroimagers of today; standards will be higher, methods tighter, reviews harsher. But if you put in enough work, if you exercise enough diligence, and if you live long enough into the ninth year of your graduate program, you may just be able to take advantage of the next major statistical loophole that comes along. Have faith.***
**For the record, I still don't know where the precuneus is. My best guess is: "Somewhere right before the cuneus."
***In all seriousness, AFNI has been responsive, helpful, and, given the circumstances, even gracious with any issues that have come up with their programs - specifically, 3dFWHMx (the smoothness estimator) and 3dClustSim (the cluster threshold estimator). The problems are not so much with AFNI, as with how it is used; and even though it is being patched, it is worth keeping in mind that there will likely be other issues discovered in the future, as with all methods - even non-parametric ones. (Their response to the Eklund paper can be found here.) The responsibility of the user is to make sure that the results that are reported follow the most up-to-date correction methods; that suspicious-looking activations that don't make sense in the experimental context, or consist of several blobs connected by threads of voxels, should be interpreted with caveats; and that even results which are weak and do not pass correction with the current methods, may still be real but underpowered effects. As with any result, it's up to the researcher to use their best judgment.
Labels:
cluster correction,
eklund,
false positives,
fMRI,
studies
Thursday, November 17, 2016
On Academic Talks
I am beginning to suspect that most academics do not want to be heard. This is a paradox, given the number and regularity of academic talks. Indeed, some experts estimate that at least eighty thousand academic talks occur at any given moment, and that some three quarters of these are designed to fill up a time slot during the week which simply must be filled, lest we begin to think about what we used to do before meetings were invented. Most of the remainder are to distract the attendees from distressing events such as grant rejections, increases in conference fees, or foreign armies parachuting onto campus. Academic talks, like most variations of committee work, are becoming a polite form of debauch.
No matter. Regardless of how many talks they give, I still think these students, these professors, these men and women, do not want to be heard. Which is too bad. A talk, for me at least, is a chance to see, hear, and possibly smell the person behind the research. A talk injects red-blooded humanity into an enterprise which would rapidly become sterile without it. Journal articles and book chapters at best radiate a kind of cool elegance. Academic writing only tells me about what they think; I want to know more about what they feel. Where their passions lie. And when someone is talking to me in the same room they can't help but show some of those emotions, giving me a sense of why they are doing what they do in the first place.
Yet some of them would deny me even that pleasure. They talk to their laptop, or to the slides, but not to me. They use small font I can't read; complicated diagrams I can't follow; color schemes and animations that are in bad taste. I would be willing to forgive most of this if I could actually hear what they were saying. Usually I can't.
That has led me to outline six simple ways to improve a talk, any of which can be used profitably:
1. Suspense! If it is to have any worth at all, if it is to function as a stimulant and not a narcotic, a talk needs to have that essential ingredient of any effective story, which is suspense. When hearing a good story, the listener always wants to know what happens next: Who is this character, and what are his motivations? Why is there a note on the table and what does it say? What's the deal with those last two or three pages at the end of a book that are intentionally blank? So it is with good talks.
2. Dress well! A well-dressed speaker is a confident speaker, and a confident speaker is an engaging speaker. When in doubt, follow these "quick tips" on how to dress:
- For Men:
- Right way: Sport coat, open collar dress shirt, loafers, good pair of jeans.
- Wrong way: Graphic tee with the words, "Legalize It."
- For Ladies:
- Right way: Pantsuit
- Wrong way: Jeggings
3. Take notes! To be specific, have a friend take notes about you during your talk. Ideally, this should be a friend who is "brutally honest," also known as "being insufferable and pretentious when pointing out your flaws, and having it hurt even more because, deep down, you know he's right." Some people also refer to this person as "Dan."* No matter how much it hurts, remember that if you want to improve, you need to learn how to take criticism graciously! Also remember that when it's all over you can cut "Dan" out of your life completely.
4. Practice! The speaker putting together his slides at the last minute is guaranteed to embarrass himself and waste the audience's time. Since one of the major goals in life is to not be that one person everybody hates, avoid this by having your slides ready days in advance and doing at least two or three run-throughs ahead of time - preferably, in front of a live audience that includes Dan.
5. Eye contact! Try making eye contact with different people during your talk, but be careful of making eye contact for too little or too long. When in doubt, consult the following chart to make sure you hit the "sweet spot":
- 1-2 seconds: Too little!
- 2-5 seconds: Just right!
- 5-10 seconds: Too much!
- 10-15 seconds: Restraining order!
- More than 15 seconds: This might be normal for, say, a serial killer
6. Lastly, find speakers you admire - and then imitate them. Really. Nothing helps your style more. Note their posture, the cadence of their voice, the way they carry themselves. Some rely more on humor, others have more of a quiet confidence; some tell stories that move the talk along, others tell you the point of the talk right away and never let you lose focus of it. But they all keep you wanting to know what happens next. They keep you in suspense.
Do not mistake this as recommending the substitution of humor for substance, of histrionics for actual thought. (The results are usually awful.) All I ask is that a speaker think about how he would want to be talked to, and then think about how seriously he has to prepare to meet that standard. Only then he begins to realize the gravity of his situation, no matter how small the talk or how low the stakes. In fact the stakes, when he thinks about it, are quite high. A typical one-hour talk attended by twenty of his colleagues can be either satisfyingly filled or grossly wasted, and a few simple calculations reveals a spread of forty attention-hours - a tremendous responsibility - all hinging on one man's decision about whether he actually wants to tell something and be heard. His choice is one between a successful talk and a crashing bore.
Now, I've witnessed more crashing bores than I would like to remember, and I've delivered more than I would like to admit. The fact is that there are far, far too many, and each one inflicts a serious loss on the audience - the loss of an hour they won't get back. (Were we not so complacent about what to expect out of a talk, we might feel more indignation.) Would they have wasted that hour anyway? Maybe. But a bad talk guarantees it.
*I specifically refer here to my labmate.
Saturday, November 12, 2016
Society for Neuroscience 2016!
Dear friends,
I will be attending this year's Society for Neuroscience conference, arriving Saturday evening. I have no plans in particular, aside from catching up with my old lab on Sunday evening, and presenting a poster Monday morning about my latest paper in The Journal of Neuroscience. (By the way, the poster is 362.05/LLL21, 8am-12pm Monday; you should all come!) So if you see me walking around the posters with a vacant stare and you want to talk to me about something, feel free to do so, because:
1) I love hearing about any questions or comments you have about the blog, the videos, or anything related to academic life and neuroimaging analysis, as it gives me a fuller, more comprehensive understanding about what graduate students and scientists are going through in this fast-changing era; and
2) I need attention.
See you in San Diego!
-Andy
Labels:
san diego,
society for neuroscience,
travels
Friday, November 11, 2016
Updated Functional Connectivity Tutorial using AFNI
As part of a new course in neuroimaging methods at Haskins Laboratories, I've begun updating videos on topics such as functional connectivity, context-dependent correlations, and how to accept bribes as a reviewer. The first topic we've covered is resting-state functional connectivity, a sophisticated-sounding name designed to make the subject think he is doing something of immense scientific importance by lying still and doing nothing, when in reality it's to distract him while we find out how to hell to hook up the experimental laptop.
Aside from its usefulness as a stall tactic, resting-state connectivity can also reveal resting-state networks, or correlations between the signal of distant regions of the brain. This provides clues to how structural connectivity - i.e., white matter connections - interact with the BOLD signal, as well as whether differences in resting-state connectivity is a marker for mental disorders such as Alzheimer's or schizophrenia.
The following video takes you step-by-step through functional connectivity analysis, using an online dataset from openfmri.org. One major change from my previous tutorials is condensing all the information into one long video, and providing time markers for each segment in the "Show More" box. This way the viewer can jump around to the information that they need, without having to keep track of several different videos detailing different steps. I hope it's an improvement, and I would like to get feedback.
I've also posted the lecture on resting-state analysis given at Haskins Laboratories on November 3rd. You won't learn much new here that isn't in the video above, but it does have more information. For most of the lecture you can only see the top of my head bobbing around, but that's OK. Eyes on the slides, not the hair.
Exercises:
Aside from its usefulness as a stall tactic, resting-state connectivity can also reveal resting-state networks, or correlations between the signal of distant regions of the brain. This provides clues to how structural connectivity - i.e., white matter connections - interact with the BOLD signal, as well as whether differences in resting-state connectivity is a marker for mental disorders such as Alzheimer's or schizophrenia.
The following video takes you step-by-step through functional connectivity analysis, using an online dataset from openfmri.org. One major change from my previous tutorials is condensing all the information into one long video, and providing time markers for each segment in the "Show More" box. This way the viewer can jump around to the information that they need, without having to keep track of several different videos detailing different steps. I hope it's an improvement, and I would like to get feedback.
I've also posted the lecture on resting-state analysis given at Haskins Laboratories on November 3rd. You won't learn much new here that isn't in the video above, but it does have more information. For most of the lecture you can only see the top of my head bobbing around, but that's OK. Eyes on the slides, not the hair.
Exercises:
- Set the errts dataset as the underlay, and select "Graph". From the "Opt" menu, select "Write Center." Rename the output 1D file, and use 1dplot to see the timecourse. This can be used as a seed for another connectivity analysis.
- Other resting state networks include the somatosensory network, the visual network, and the language network. Research one of these networks, determine where the hubs are, and run a resting state analysis on a seed placed in that hub.
- Run correlations for a group of subjects, convert to z-scores, and do a second-level t-test using uber_ttest.py.
- Modify the afni_proc.py script to apply 3dRSFC to your data (see Example 10b in afni_proc.py -help)
Wednesday, August 24, 2016
An Interview with Reviewer #2
Peer review, the cherished academic tradition of having your work criticized by anonymous angry people, is an excellent chance for you to see your prose violated in public. According to one publisher, peer review also helps "increase networking opportunities," in that after having your paper reviewed you will become very, very interested in finding out the names and addresses of all your reviewers.
The reviewing panel consists of two or three reviewers, known by their pseudonyms "Reviewer #1," "Reviewer #2," "Johnny Two-Knuckles," "Icepick Willie," and so on. As everyone knows, the review process tends to be a "good-cop, bad-cop" routine, with Reviewer #1 being nice and lenient - pointing out you shouldn't use Comic Sans font, for instance - while Reviewer #2 is so offended to read your paper that he thinks you should, in so many words, die.
Reviewer #2 is in fact a man named Gary who owns a hardware store in Winnipeg. Although no longer in academia, Gary is still the man who can be counted on, when the chips are down, to write a scathing review of whatever he's reading. Editors scramble to recruit Gary when confronted with a paper they may have to accept, and he is regularly solicited for freelance reviewing. This week we caught up with Gary at his summer home on the banks of Lake Manitoba.
Andy's Brain Blog: How did you become interested in reviewing?
Gary: I took a psychology class in college where we critiqued each other's class projects that were written like scientific articles. Then we anonymously reviewed each other's papers. The professor was impressed that I managed to reject every single paper that I read, and that I also managed to make unnecessary remarks about the author's intelligence and work ethic. At the time I didn't even know what rejecting a paper meant. It just came naturally to me. He put in a good word for me at Elsevier.
ABB: And what happened then?
Gary: Well, I began reviewing everything I read. One time I got so into it that I ended up reviewing the back of a cereal box. It was an accident, but the review was accepted anyway. Two employees at General Mills got fired because of it.
ABB: Wow.
Gary: Yeah. There were grammatical mistakes on there like you wouldn't believe. I couldn't follow the logic of how solving a word game would help Buzz escape from a bank vault full of honey. And the figures were atrocious.
ABB: What was the most memorable review you ever did?
Gary: It's funny you ask, because just last week I returned from the annual Reviewer's Gala in Manhattan. It's a private party for those who have the highest rate of rejecting manuscripts, with awards given for achievements like Most Papers Rejected, Most Brutal Review, Most Irrelevant Comment, and so on. This year I won the prize for Most Hurtful Comment, which went something like: "Writing this paper didn't make you a terrible scientist - you were born one." When the emcee read that line, the audience went wild.
ABB: What is the most ridiculous comment you've ever gotten someone to address?
Gary: I'm not that good at making crazy requests, but one of my fellow reviewers - Carl - can get people to do almost anything. One of his comments was, and I quote: "This is a strong paper, but I think it would be even stronger if, for some reason, all of the authors did the gallon challenge, and uploaded a video of it to YouTube. Now obviously you don't have to do this, but you should, because I am a reviewer."
ABB: They actually did that?
Gary: Yeah. One of them had to go to the hospital. Carl felt pretty bad about that one.
ABB: What advice would you give to a first-time reviewer?
Gary: Rejecting a paper takes a tremendous amount of courage. We've all had the temptation to accept a paper because the science was "solid," or because the logic was "air-tight," or because one of the authors secretly gave us "money." Be firm! I find that I write my best reviews when I'm pissed off about something that has nothing to do with the paper, such as getting something in the mail about taxes.
ABB: You owe a lot of taxes?
Gary: No, I just found out about them, as a concept. They're ridiculous. That's the kind of thing I'm talking about that will get you in the right mood to review a paper.
ABB: Have you ever accepted a paper?
Gary: No.
ABB: Never?
Gary: Never-ever.
ABB: Never even come close?
Gary: Well, there have been a few times. Maybe if one of the authors had the same last name as a celebrity I like, such as Barry Manilow or Kenny G. But other than that, no.
ABB: How long does it take you to write a review?
Gary: Not long. I have a template that I follow, which is a lot like Mad-Libs. For example, "This is an interesting [study / review / prophecy], but I find the [results / figures / theology] unconvincing because I am [an expert / a skeptic / a nun]." Things like that.
ABB: So, how long does it take to get back to the authors? A couple of days? A week?
Gary: No, no, nothing like that. The review takes a couple of days at the most, but you can't let the editor think that you're just blowing through it. I sit on it for at least a few months.
ABB: What are your strategies for writing a review? Is it to always go negative, or what?
Gary: Well, you have to be careful about that. Writing only negative comments raises suspicions that you're taking out your own frustrations and lack of success on the authors instead of addressing their arguments. I aim for a mix of negative comments, nitpicking, and vague sentences. Vague sentences are great, because the authors aren't going to admit that they don't understand what you're saying. Asking an academic to be clear is like asking him to take his clothes off - it's a rude request, almost obscene. So instead they reply as though they understood perfectly what you were saying. It's amazing to see how they try to interpret what is in fact nonsense.
ABB: Can you give an example?
Gary: Sure. Let me see - here's one: "Among the considerations that arise at this stage are the likelihood that the manuscript would seem of considerable interest to those working in the same area of science and the degree to which the results will stimulate new thinking in the field, although we cannot be persuaded of the justifiability, synergy, or translatability of how these results integrate with the conclusions and narrative of Fensterwhacker et al, 2009. Are you professional. Also, you spelled 'their' wrong (should be 'they're': p. 19)."
ABB: I have no idea what that means.
Gary: Exactly.
ABB: How do they respond?
Gary: Usually they begin with something like "We thank the reviewer for their insightful comment," or "We are just thrilled by this excellent suggestion," or "I simply cannot wait to meet this reviewer in person and show him how incredibly, insanely grateful I am, which in no way would include kidnapping his dog." It's interesting how far someone will bend over backwards to address a comment that could've been written by a complete space loon.
ABB: Why do you keep doing this? You're not in academia anymore.
Gary: I try to focus on the big picture. I think that by irritating so many people, everybody will have something in common to talk about. Then they can bond over their shared frustrations and challenges. It makes academia more like a family, except in the sense of being related to or liking or caring about one another.
ABB: Gary, thanks for your time.
Gary: You spelled "your" wrong.
The reviewing panel consists of two or three reviewers, known by their pseudonyms "Reviewer #1," "Reviewer #2," "Johnny Two-Knuckles," "Icepick Willie," and so on. As everyone knows, the review process tends to be a "good-cop, bad-cop" routine, with Reviewer #1 being nice and lenient - pointing out you shouldn't use Comic Sans font, for instance - while Reviewer #2 is so offended to read your paper that he thinks you should, in so many words, die.
Reviewer #2 is in fact a man named Gary who owns a hardware store in Winnipeg. Although no longer in academia, Gary is still the man who can be counted on, when the chips are down, to write a scathing review of whatever he's reading. Editors scramble to recruit Gary when confronted with a paper they may have to accept, and he is regularly solicited for freelance reviewing. This week we caught up with Gary at his summer home on the banks of Lake Manitoba.
===============
Andy's Brain Blog: How did you become interested in reviewing?
Gary: I took a psychology class in college where we critiqued each other's class projects that were written like scientific articles. Then we anonymously reviewed each other's papers. The professor was impressed that I managed to reject every single paper that I read, and that I also managed to make unnecessary remarks about the author's intelligence and work ethic. At the time I didn't even know what rejecting a paper meant. It just came naturally to me. He put in a good word for me at Elsevier.
ABB: And what happened then?
Gary: Well, I began reviewing everything I read. One time I got so into it that I ended up reviewing the back of a cereal box. It was an accident, but the review was accepted anyway. Two employees at General Mills got fired because of it.
ABB: Wow.
Gary: Yeah. There were grammatical mistakes on there like you wouldn't believe. I couldn't follow the logic of how solving a word game would help Buzz escape from a bank vault full of honey. And the figures were atrocious.
ABB: What was the most memorable review you ever did?
Gary: It's funny you ask, because just last week I returned from the annual Reviewer's Gala in Manhattan. It's a private party for those who have the highest rate of rejecting manuscripts, with awards given for achievements like Most Papers Rejected, Most Brutal Review, Most Irrelevant Comment, and so on. This year I won the prize for Most Hurtful Comment, which went something like: "Writing this paper didn't make you a terrible scientist - you were born one." When the emcee read that line, the audience went wild.
ABB: What is the most ridiculous comment you've ever gotten someone to address?
Gary: I'm not that good at making crazy requests, but one of my fellow reviewers - Carl - can get people to do almost anything. One of his comments was, and I quote: "This is a strong paper, but I think it would be even stronger if, for some reason, all of the authors did the gallon challenge, and uploaded a video of it to YouTube. Now obviously you don't have to do this, but you should, because I am a reviewer."
ABB: They actually did that?
Gary: Yeah. One of them had to go to the hospital. Carl felt pretty bad about that one.
ABB: What advice would you give to a first-time reviewer?
Gary: Rejecting a paper takes a tremendous amount of courage. We've all had the temptation to accept a paper because the science was "solid," or because the logic was "air-tight," or because one of the authors secretly gave us "money." Be firm! I find that I write my best reviews when I'm pissed off about something that has nothing to do with the paper, such as getting something in the mail about taxes.
ABB: You owe a lot of taxes?
Gary: No, I just found out about them, as a concept. They're ridiculous. That's the kind of thing I'm talking about that will get you in the right mood to review a paper.
ABB: Have you ever accepted a paper?
Gary: No.
ABB: Never?
Gary: Never-ever.
ABB: Never even come close?
Gary: Well, there have been a few times. Maybe if one of the authors had the same last name as a celebrity I like, such as Barry Manilow or Kenny G. But other than that, no.
ABB: How long does it take you to write a review?
Gary: Not long. I have a template that I follow, which is a lot like Mad-Libs. For example, "This is an interesting [study / review / prophecy], but I find the [results / figures / theology] unconvincing because I am [an expert / a skeptic / a nun]." Things like that.
ABB: So, how long does it take to get back to the authors? A couple of days? A week?
Gary: No, no, nothing like that. The review takes a couple of days at the most, but you can't let the editor think that you're just blowing through it. I sit on it for at least a few months.
ABB: What are your strategies for writing a review? Is it to always go negative, or what?
Gary: Well, you have to be careful about that. Writing only negative comments raises suspicions that you're taking out your own frustrations and lack of success on the authors instead of addressing their arguments. I aim for a mix of negative comments, nitpicking, and vague sentences. Vague sentences are great, because the authors aren't going to admit that they don't understand what you're saying. Asking an academic to be clear is like asking him to take his clothes off - it's a rude request, almost obscene. So instead they reply as though they understood perfectly what you were saying. It's amazing to see how they try to interpret what is in fact nonsense.
ABB: Can you give an example?
Gary: Sure. Let me see - here's one: "Among the considerations that arise at this stage are the likelihood that the manuscript would seem of considerable interest to those working in the same area of science and the degree to which the results will stimulate new thinking in the field, although we cannot be persuaded of the justifiability, synergy, or translatability of how these results integrate with the conclusions and narrative of Fensterwhacker et al, 2009. Are you professional. Also, you spelled 'their' wrong (should be 'they're': p. 19)."
ABB: I have no idea what that means.
Gary: Exactly.
ABB: How do they respond?
Gary: Usually they begin with something like "We thank the reviewer for their insightful comment," or "We are just thrilled by this excellent suggestion," or "I simply cannot wait to meet this reviewer in person and show him how incredibly, insanely grateful I am, which in no way would include kidnapping his dog." It's interesting how far someone will bend over backwards to address a comment that could've been written by a complete space loon.
ABB: Why do you keep doing this? You're not in academia anymore.
Gary: I try to focus on the big picture. I think that by irritating so many people, everybody will have something in common to talk about. Then they can bond over their shared frustrations and challenges. It makes academia more like a family, except in the sense of being related to or liking or caring about one another.
ABB: Gary, thanks for your time.
Gary: You spelled "your" wrong.
Labels:
academia,
gary,
interview,
mad libs,
peer review,
reviewer #2,
reviews,
winnipeg
Thursday, August 4, 2016
New Website
Readers have complained I haven't updated in a while. Do you know why I haven't updated? Too much Andy's Brain Blog is bad for you. It's like cigarettes, booze, or Nutella. It should be enjoyed in moderation - if at all.
Many of you probably felt that the writing here was slowly petering out. I don't blame you. I've come across sites like that - sites that make me feel as though I'm walking through an abandoned house. What's unsettling is that the writing didn't end; it stopped. That makes me think something terrible happened to the author. Maybe he said everything he had to say; maybe he lost interest; maybe he simply lost inspiration - and couldn't bear to look at those half-stitched monstrosities he began but never finished. I understand. There are many posts that I began to write, but then abandoned - they didn't sound right. You would be surprised how many of these limbless horrors I have buried in my graveyard.
There are two other reasons why I haven't written. One, long periods of absence tend to filter out the fair-weather readers and leave me with only the fanatics. Two, I have been building a new website - a professional website, complete with photos of me doing professional things, such as posing for the camera. I felt that it was time to move; some may disagree. I hate to disappoint them.
Regardless, my posts will continue on the new website; and, to smooth the transition, new writings will be posted to both sites for the next few months. I haven't decided yet what I'll do with this blog; I am too fond of it to simply press "delete" and see it vanish into the electricity. There's history here. Perhaps I'll write something here once in a while with my more unprofessional thoughts. I don't intend to stop anytime soon.
Yet I know that, whatever happens to me, there are others who carry the flag; that there are others who are doing what I do. A few examples come to mind: Mumford Brain Stats; Crash Log; Diffusion Imaging. And that is why this blog, being what it is - a desire to help you understand, to get you excited about neuroimaging; above all, to make you see - will survive even if it die.
Many of you probably felt that the writing here was slowly petering out. I don't blame you. I've come across sites like that - sites that make me feel as though I'm walking through an abandoned house. What's unsettling is that the writing didn't end; it stopped. That makes me think something terrible happened to the author. Maybe he said everything he had to say; maybe he lost interest; maybe he simply lost inspiration - and couldn't bear to look at those half-stitched monstrosities he began but never finished. I understand. There are many posts that I began to write, but then abandoned - they didn't sound right. You would be surprised how many of these limbless horrors I have buried in my graveyard.
There are two other reasons why I haven't written. One, long periods of absence tend to filter out the fair-weather readers and leave me with only the fanatics. Two, I have been building a new website - a professional website, complete with photos of me doing professional things, such as posing for the camera. I felt that it was time to move; some may disagree. I hate to disappoint them.
Regardless, my posts will continue on the new website; and, to smooth the transition, new writings will be posted to both sites for the next few months. I haven't decided yet what I'll do with this blog; I am too fond of it to simply press "delete" and see it vanish into the electricity. There's history here. Perhaps I'll write something here once in a while with my more unprofessional thoughts. I don't intend to stop anytime soon.
Yet I know that, whatever happens to me, there are others who carry the flag; that there are others who are doing what I do. A few examples come to mind: Mumford Brain Stats; Crash Log; Diffusion Imaging. And that is why this blog, being what it is - a desire to help you understand, to get you excited about neuroimaging; above all, to make you see - will survive even if it die.
Subscribe to:
Posts (Atom)






