- Stay Connected
Follow Us
90% of everything is crud
October 4, 2012
CALGARY, AB, Oct 4, 2012/ Troy Media/ – Responding to a critics observation that most science fiction is junk, writer Theodore Sturgeon noted that the same could be said for just about everything. Sturgeon’s Revelation, also referred to as Sturgeons Law, is that; Ninety per cent of everything is crud.
Everything includes fiction and non-fiction writing, film, television, news stories, consumer products, rules and regulations and what we hear from business and government. Probably no surprise there.
What is a surprise is that there’s a scientific basis for Sturgeon’s Revelation, making it true for formal scientific and academic research because of how statistical significance is misused.
We’ve all become familiar with the language of statistical significance in poll results peppering the news around election time. When told that, ’74 per cent of Canadians are in favor of something, plus or minus 3 per cent, 19 times out of 20′, that’s statistical significance. So is news that U.S. President Barack Obama’s four-point lead is beyond the ‘margin of error’. Polling is one of the few places where statistical significance calculations are applied and interpreted properly, informing us of poll accuracy.
But while it tells us about poll accuracy, statistical significance says nothing about the importance of those results. Is Obama’s four point lead big, insurmountable, or important? That’s a question for political pundits, not statistics. Statistical significance answers ‘How Much?’ but can’t answer ‘So what?‘ Where research goes off the rails is pretending otherwise.
For example, recently, the Memorial Sloan Kettering Cancer Center released a study concluding “the most robust evidence to date that acupuncture is a reasonable referral option“. A lead researcher is quoted as saying; “The effects of acupuncture are statistically significant . . . so we conclude aren’t due merely to the placebo effect“. Whatever, you may think about acupuncture, conclusions like this are delusional.
The misuse or misinterpretation of statistical significance is Why Most Published Research Findings Are False. This classic paper, by John P. A. Loannidis, notes;
Several methodologists have pointed out that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05. . . . It can be proven that most claimed research findings are false.
How often are they false? How often are statistically significant findings of no practical importance? Results vary but here’s a nice symmetry with Sturgeon’s Revelation, a rough estimate is about 90 per cent of the time.
When next you hear or read about some latest health research claiming statistically significant results, remember, the results are probably wrong. Also, the more a study pumps its statistical significance, the greater the likelihood its crud. Good research mentions statistical significance in passing, focusing instead on material significance.
The same is true for government policy and program evaluation research in areas such as economics, housing, education, social and human services. If a study trumpets statistical significance of the findings, it’s probably crud.
So why are statistical significance still used in this way?
Because it’s so easy. Statistical insiders know it’s crud, but so what? There’s no cost to this lying with statistics.
Until now perhaps. The United States Supreme Court, recently rejected the claim by Matrixx Initiatives that it wasn’t required to report side effects of Zicam, because of a lack of statistical significance. The Court sided with science saying ‘medical professionals and regulators act on the basis of evidence of causation that is not statistically significant’.
In other words, statistical significance, or the lack of it, is not the same thing as a scientific finding and, therefore, not a defense for anything. This rightly opens the door for lawsuits against those who lie to people by selling statistical significance as practical importance.
Theodore Sturgeon was more right than he knew.
Troy Media columnist Robert Gerst is a Partner in Charge of Operational Excellence and Research & Statistical Methods at Converge Consulting Group Inc. He is author of The Performance Improvement Toolkit: The Guide to Knowledge-Based Improvement and numerous articles in peer-reviewed publications.
Read more Bottom Line
Follow the Bottom Line via RSS
This column is FREE to use on your websites or in your publications. However, Troy Media, with a link to its web site, MUST be credited.
© Troy Media
| ADVERTISEMENT |
Facebook
Reddit
Twitter
StumbleUpon
Sign me up to your newsletters!