The test of replicability: are you practicing fact or fiction?

In December, The New Yorker ran an article which I found extremely interesting and intriguing.  The article was titled, “The Truth Wears Off: Is there something wrong with the scientific method?” ,and examined the difficulty for scientists to replicate well-established studies.   The author, Jonah Lehrer, highlights how the foundation of modern research, the test of replicability, is revealing that many believed facts are losing their truth.  He states the difficulty of scientists to replicate well-known studies and get similar results.

He sites a possible reason for this from research completed in the 1930’s by Joseph Rhine, a psychologist at Duke University.

Rhine had an interest in extrasensory perception (esp) and Rhine would use a deck of 25 cards printed with one of five different symbols on each card.  Rhine would ask a subject which symbol was on the card and most of his subjects would guess 25% of the cards correctly which is mathematically expected.  One day, Rhine came across a subject named Adam Linzmayer who guessed nearly 50% of the cards correctly, even guessing nine in a row!  The odds of this happening by chance, without esp, is 1/2,000,000 and Linzmayer repeated this three times.

Rhine studied these results and was preparing an article for publication on esp when he invited Linzmayer back to guess 1000 more cards.  This time, Linzmayer’s success rate went down significantly and it appeared he was guessing correct by chance.   Rhine later noticed this effect would occur with other individuals who he believed had ESP and called this “the decline effect“.  It simply represented a decline in his data to replicate.

Now to some math…a decline in data to replicate most likely occurs from a regression toward the mean.  As the experiment is repeated, it cancels out statistical flukes or extreme phenomenon. So in the example of Rhine, the statistical fluke of possible esp was an illusion which vanished over time.

Another possible flaw to replication is in the peer-reviewed process.  There is currently a fad in which when a new idea or paradigm is proposed, the peer-reviewed process is skewed or more entrenched toward positive results.  In time, the incentives shift and the most notable results are to disprove the once exciting, new theory.  This can be referred to as publication bias.  But as publication bias has an effect, it doesn’t account for the researchers findings which in many cases, may be skewed from selective reporting of results.

After reading this article, which is non-peer reviewed, I am intrigued at the replicability of much research we currently follow as clinicians.  One article that particularly comes to mind was published in the January 2011 JOSPT. The article was titled “Immediate Effects of Lumbar Spine Manipulation on the Resting and Contraction Thickness of Transversus Abdominis in Asymptomatic Individuals”.  This article looked at the effects of lumbar spinal manipulation on the thickness of the transversus abdominis (TrA) in asymptomatic subjects.  In previous studies, including the infamous clinical prediction rule for lumbar manipulation, examiners found the thickness of the TrA changed following a lumbar manipulation.  The subjects in previous studies were those with LBP, so it could be speculated that the thickness may have been altered due to descending pain inhibitory influences which might have allowed for greater relaxation.   In this current study, the thickness of the TrA did not change with manipulation.

Could this study be a possible example of the decline effect or a possible peer-reviewer excitement to disprove a once-exciting theory? Possibly.  The results could also be correct, and non-flawed, and simply different due to the examination of subjects without LBP vs. those with LBP and the neurophysiological mechanisms effected by the manipulation are different in those with pain vs. those without pain.   But in either case, we must continue to reevaluate and replicate our current research and techniques so that we can further define ourselves as effective clinicians.

Lehrer J. The Truth Wears Off: Is there something wrong with the scientific method. The New Yorker. Dec. 13, 2010.

Puentedura EJ, Landers MR, Hurt K. Immediate Effects of Lumbar Spine Manipulation on the Resting and Contraction Thickness of Transversus Abdominis in Asymptomatic Individuals. JOSPT: 41 (1); 13-21.