Scientific American
Doing Good Science
Evaluating scientific claims (or, do we have to take the scientist’s word for it?)
By Janet D. Stemwedel | September 30, 2011

The views expressed are those of the author and are not necessarily those of Scientific American.

Recently, we’ve noted that a public composed mostly of non-scientists may find itself asked to trust scientists, in large part because members of that public are not usually in a position to make all their own scientific knowledge. This is not a problem unique to non-scientists, though — once scientists reach the end of the tether of their expertise, they end up having to approach the knowledge claims of scientists in other fields with some mixture of trust and skepticism. (It’s reasonable to ask what the right mixture of trust and skepticism would be in particular circumstances, but there’s not a handy formula with which to calculate this.)

Are we in a position where, outside our own narrow area of expertise, we either have to commit to agnosticism or take someone else’s word for things? If we’re not able to directly evaluate the data, does that mean we have no good way to evaluate the credibility of the scientist pointing to the data to make a claim?

This raises an interesting question for science journalism, not so much about what role it should play as what role it could play.

If only a trained scientist could evaluate the credibility of scientific claims (and then perhaps only in the particular scientific field in which one was trained), this might reduce science journalism to a mere matter of publishing press releases, or of reporting on scientists’ social events, sense of style, and the like. Alternatively, if the public looked to science journalists not just to communicate the knowledge claims various scientists are putting forward but also to do some evaluative work on our behalf — sorting out credible claims and credible scientists from the crowd — we might imagine that good science journalism demands extensive scientific training (and that we probably need a separate science reporter for each specialized area of science to be covered).

In an era where media outlets are more likely to cut the science desk than expand it, pinning our hopes on legions of science-Ph.D.-earning reporters on the science beat might be a bad idea.

I don’t think our prospects for evaluating scientific credibility are quite that bad.

Scientific knowledge is built on empirical data, and the details of the data (what sort of data is relevant to the question at hand, what kind of data can we actually collect, what techniques are better or worse for collecting the data, how we distinguish data from noise, etc.) can vary quite a lot in different scientific disciplines, and in different areas of research within those disciplines. However, there are commonalities in the basic patterns of reasoning that scientists in all fields use to compare their theories with their data. Some of these patterns of reasoning may be rather sophisticated, perhaps even non-intuitive. (I’m guessing certain kinds of probabilistic or statistical reasoning might fit this category.) But others will be the patterns of reasoning that get highlighted when “the scientific method” is taught.

In other words, even if I can’t evaluate someone else’s raw data to tell you directly what it means, I can evaluate the way that data is used to support or refute claims. I can recognize logical fallacies and distinguish them from instances of valid reasoning. Moreover, this is the kind of thing that a non-scientist who is good at critical thinking (whether a journalist or a member of the public consuming a news story) could evaluate as well.

One way to judge scientific credibility (or lack thereof) is to scope out the logical structure of the arguments a scientist is putting up for consideration. It is possible to judge whether arguments have the right kind of relationship to the empirical data without wallowing in that data oneself. Credible scientists can lay out:

Here’s my hypothesis.
Here’s what you’d expect to observe if the hypothesis is true. Here, on the other hand, is what you’d expect to observe if the hypothesis is false.
Here’s what we actually observed (and here are the steps we took to control the other variables).
Here’s what we can say (and with what degree of certainty) about the hypothesis in the light of these results.
Here’s the next study we’d like to do to be even more sure.
And, not only will the logical connections between the data and what is inferred from them look plausible to the science writer who is hip to the scientific method, but they ought to look plausible to other scientists — even to scientists who might prefer different hypotheses, or different experimental approaches. If what makes something good science is its epistemology — the process by which data are used to generate and/or support knowledge claims — then even scientists who may disagree with those knowledge claims should still be able to recognize the patterns of reasoning involved as properly scientific. This suggests a couple more things we might ask credible scientists to display:

Here are the results of which we’re aware (published and unpublished) that might undermine our findings.
Here’s how we have taken their criticisms (or implied criticisms) seriously in evaluating our own results.
If the patterns of reasoning are properly scientific, why wouldn’t all the scientists agree about the knowledge claims themselves? Perhaps they’re taking different sets of data into account, or they disagree about certain of the assumptions made in framing the question. The important thing to notice here is that scientists can disagree with each other about experimental results and scientific conclusions without thinking that the other guy is a bad scientist. The hope is that, in the fullness of time, more data and dialogue will resolve the disagreements. But good, smart, honest scientists can disagree.

This is not to say that there aren’t folks in lab coats whose thinking is sloppy. Indeed, catching sloppy thinking is the kind of thing you’d hope a good general understanding of science would help someone (like a scientific colleague, or a science journalist) to do. At that point, of course, it’s good to have backup — other scientists who can give you their read on the pattern of reasoning, for example. And, to the extent that a scientist — especially one talking “on the record” about the science (whether to a reporter or to other scientists or to scientifically literate members of the public) — displays sloppy thinking, that would tend to undermine his or her credibility.

There are other kinds of evaluation you can probably make of a scientist’s credibility without being an expert in his or her field. Examining a scientific paper to see if the sources cited make the claims that they are purported to make by the paper citing them is one way to assess credibility. Determining whether a scientist might be biased by an employer or a funding source may be harder. But there, I suspect many of the scientists themselves are aware of these concerns and will go the extra mile to establish their credibility by taking the possibility that they are seeing what they want to see very seriously and testing their hypotheses fairly stringently so they can answer possible objections.

It’s harder still to get a good read on the credibility of scientists who present evidence and interpretations with the right sort of logical structure but who have, in fact, fabricated or falsified that evidence. Being wary of results that seem too good to be true is probably a good strategy here. Also, once a scientist is caught in such misconduct, it’s entirely appropriate not to trust another word that comes from his or her mouth.

One of the things fans of science have tended to like is that it’s a route to knowledge that is, at least potentially, open to any of us. It draws on empirical data we can get at through our senses and on our powers of rational thinking. As it happens, the empirical data have gotten pretty complicated, and there’s usually a good bit of technology between the thing in the world we’re trying to observe and the sense organs we’re using to observe it. However, those powers of rational thinking are still at the center of how the scientific knowledge gets built. Those powers need careful cultivation, but to at least a first approximation they may be enough to help us tell the people doing good science from the cranks.

Janet D. Stemwedel About the Author: Janet D. Stemwedel is an Associate Professor of Philosophy at San José State University. Her explorations of ethics, scientific knowledge-building, and how they are intertwined are informed by her misspent scientific youth as a physical chemist. Follow on Twitter @docfreeride.
The views expressed are those of the author and are not necessarily those of Scientific American.

Scientific American is a trademark of Scientific American, Inc., used with permission

© 2013 Scientific American, a Division of Nature America, Inc.

All Rights Reserved.

The problem now is all the scientists are associated with Colleges or private interests like pharmaceuticals.
If they don't play ball then the funding stops and the FDA approvals get held up.
I found a statement made by a top economist in the country very telling and even he didn't realize what he had exposed. He stated "Before we can come up with those analysis we have to get the numbers from the BLS"
In other words they have to wait till Washington tells them what to say to come up with the numbers that support the headlines Washington is aiming for.

Explain Carbon Dioxide as a pollutant, collecting Carbon Credits/Fines for the Federal Government's E.P.A.,D.E.C. and any other imbecile entity, part of the science-fiction branch of reality, no answers but the same parameters of the Eugenics and social hygiene of the Physicians Reich Chancellors, from 1934-1945, as far as it goes, wh**ever happened to science and facts? Scientific Sharia or plain stupidity, no only the c*****e-c****e i***ts who have never questioned why it is called weather, if the object of this is to control every aspect of climate, doesn't this sound kind of arrogant? Maybe ask God, once he stops laughing.

Take for example the Architects & Engineers for 9/11 T***h. They are 3000 of the finest most experienced minds in the country and have told us that the official explanation for 911 is not possible and even gotten the NIST to admit that the free fall scenario is unexplainable outside of controlled demolish.
Even though Thermite has been proven to be on the site and the video of molten steel pouring from the ruins has been clearly seen weeks after they fell people still refuse to believe those supports were c*********d.
The most common sense analogy has been presented to people that the temps that were found on the beams that were tested from the floors that first started falling are only half the temp that is created when your self cleaning oven is fully heated. Does their oven melt on the kitchen floor.
Even the phone calls that are impossible to have been made from a plane 30,000 feet in the air and messages left on the answering machines of loved ones saying "Its a frame" have been dismissed. These are the dying words of the people of that plane that are being disregarded as lies.
The t***h some times is much to horrifying for some to let seep into their minds and the comfort of the lie that has been provided for them is less disturbing to their existence. The lie requires no change on their part to believe.
Listen to the last 5 minutes of DVD #1 and tell me that this woman who knows she is going to die having heard that the other 2 planes had flown into the towers is lying.
http://www.luogocomune.net/site/modules/sections/index.php?op=viewarticle&artid=167