Post by Ruthanna Gordon
New science isn’t hard to find. Open the newspaper, and you’ll find articles about the latest medical breakthrough and the newest environmental research. Turn on the TV, and the Discovery Channel will tell you all about sharks or space travel. Read a blog like this one, and enthusiastic writers will happily ramble about their latest scientific obsessions. But with all this information out there, how much can we trust? How should we think about the stories we read over morning coffee?
Most science reporting is a little simplified, by necessity. The full write-up of a scientific study may be 10 or 20 pages long, and heavily dependent on technical terminology. The best reporters simply distill research to its essence, sharing the most intriguing highlights. The worst make the research unrecognizable. Here are a few translation tips that I’ve found useful for figuring out what studies actually show:
Ignore the Headline. Headlines are written for eyeball grabs first, and accuracy second. I’m going to pick on the New York Times here, because they’re bigger than us, and generally provide excellent science reporting. Flipping to their latest Health section, I find “Brain Substance May Help Retrieve and Fortify Old Memories.” Helped along by the opening paragraph, my first assumption is that someone has developed a memory pill—maybe an artificial neurotransmitter?—that will help me regain the Calculus skills I let slide after college. Alas, no. Reading a little further, they have discovered a substance that improves memory for the past six days—in rats.
Insert Disclaimers. One easy way to simplify a story is to say that something is true or will happen, rather than giving percentages and likelihoods. Sometimes quotes from the researchers will hint at these qualifiers, but sometimes even that is left out. I am completely enamored of this idea from Popular Science. But I suspect that if you talked to the people who came up with it, they’d have a lot more to say about the remaining barriers to development. Unfortunately, I’m not expecting my street to be lit by glowing gold leaves any time soon.
…But Distrust Controversies. This is something you can only pick up after reading a few articles on a topic. At which point you’ll notice that, in spite of media claims about huge scientific debates, vaccine refusers and climate change skeptics never produce new studies. If someone isn’t constantly producing new supporting data, they are not doing science, however often they get interviewed. A good theory doesn’t come from one possibly flawed study, but from ten studies that all show the same thing in different ways.
Question Correlations. When you hear that two things are correlated, treat it as a fill-in-the-blank puzzle. The odds are against one of the things causing the other—it’s much more likely that some third thing caused both. One of my students delightfully calls this the lurking factor. For example, shoe size is strongly correlated with math ability.
I’ll give you a minute.
The answer, if you didn’t guess, is that age is the lurking factor, correlated with both of them. As a child grows, so do their feet, and so does their exposure to algebra. Most correlations are like this. If you look closely at the seemingly simple relationship, you find that it masks a larger and more interesting pattern.
Ultimately, that’s the most important thing to bear in mind about popular science stories. The full truth is usually a little more complicated than what you’re reading—and a lot more interesting!