Thursday, April 25, 2013

The Noose Tightens: Scientific Standards Being Raised

For those of you hoping to fly under the radar of reviewers and get your questionable studies published, I suggest that you do so with a quickness. A new editorial in Nature Neuroscience outlines the journal's updated criteria for methods reporting, which removes the limit on the methods section of papers, mandates reporting the data used to create figures, and requires statements on randomization and blinding. In addition, the editorial board takes a swipe at the current level of statistical proficiency in biology, asserting that

Too many biologists [and neuroscientists] still do not receive adequate training in statistics and other quantitative aspects of their subject. Mentoring of young scientists on matters of rigor and transparency is inconsistent at best. In academia, the ever-increasing pressures to publish and obtain the next level of funding provide little incentive to pursue and publish studies that contradict or confirm previously published results. Those who would put effort into documenting the validity or irreproducibility of a published piece of work have little prospect of seeing their efforts valued by journals and funders; meanwhile, funding and efforts are wasted on false assumptions.

What the editors are trying to say, I think, is that a significant number of academics, and particularly graduate students, are most lazy, benighted, pernicious race of little odious vermin that nature ever suffered to crawl upon the surface of the earth; to which I might add: This is quite true, but although we may be shiftless, entitled, disgusting vermin, it is more accurate to say that we are shiftless, entitled, disgusting vermin who simply do not know where to start. While many of us learn the basics of statistics sometime during college, much is not retained, and remedial graduate courses do little to prepare one for understanding the details and nuances of experimental design that can influence the choice of statistics that one uses. One may argue that the onus is on the individual to teach himself what he needs to know in order to understand the papers that he reads, and to become informed enough to design and write up a study at the level of the journal for which he aims; however, this implies an unrealistic expectation of self-reliance and tenacity for today's average graduate student. Clearly, blame must be assigned: The statisticians have failed us.

Another disturbing trend in the literature is a recent rash of papers encouraging studies to include more subjects, to aid both statistical reliability and experimental reproducibility. Two articles in the last issue of Neuroimage - One by Michael Ingre, one by Lindquist et al - as well as a recent Nature Neuroscience article by Button et al, take Karl Friston's 2012 Ten Ironic Rules article out to the woodshed, claiming that small sample sizes are more susceptible to false positives, and that instead larger samples should be recruited and effect sizes reported. More to the point, the underpowered studies that are published tend to be biased to only finding effects that are inordinately large, as null effects simply go unreported.

All of this is quite unnerving to the small-sample researcher, and I advise him to crank out as many of his underpowered studies as he can before larger sample sizes become the new normal, and one of the checklist criteria for any high-impact journal. For any new experiments, of course, recruit large sample sizes, and when reviewing, punish those who use smaller sample sizes, using the reasons outlined above; for then you will have still published your earlier results, but manage to remain on the right side of history. To some, this may smack of Tartufferie; I merely advise you to act in your best interests.

4 comments:

  1. "Too many biologists [and neuroscientists] still do not receive adequate training in statistics and other quantitative aspects of their subject."

    Couldn't agree more ... I've done well in statistics courses, and I still feel rather clueless. It's the fault of academic institutions that gloss over it as an afterthought ...

    ReplyDelete
    Replies
    1. Actually, I'd say it's true of scientists in general, not just biologists and neuroscientists.

      Delete
  2. As for sample sizes, if one can quantitatively justify a smaller sample size because there is very little expected variability and therefore you can still get good statistical power with the smaller sample size, then I think that should be fine (but what do I know?).

    On the other hand, what if you want to study something and there are only few samples available in the first place? Should you avoid the inquiry altogether?

    ReplyDelete
    Replies
    1. Hey Sherif,

      I see no problem with using smaller sample sizes, provided that a power analysis shows that a typical effect for your study will probably reject the null most of the time. This is the motivation behind pilot studies and power analyses, although I think that most people tend to omit them; see one of my earlier posts discussing Jeanette Mumford's fmri_power tool in Matlab.

      This relates to your other question about exploratory analyses restricted by small sample sizes. I don't think they should be discouraged, necessarily, but I do think that researchers need to have some idea about the effect that they expect to find before simply testing any hypothesis with a small sample. Since most null effects from small samples go unreported, the ones that do get reported are often very big; they have to be, in order to reject the null with relatively few subjects. Again, pilot studies could help this problem by giving some idea about whether the question is worth pursuing in the first place.

      As for statistical knowledge in general, I think it would help if courses examined the statistics used in actual studies, so that students get a better idea about why they are used and in what contexts, and discuss whether the statistics are appropriate in a given paper...but that's just something I've been thinking about in my free time.

      -Andy

      Delete