Seeing is Not Always Believing: Why Anecdotal Evidence is Not Proof
Recently, the medical journal ‘Pediatrics’ released a consensus report stating that after evaluating all of the scientific and medical literature (studies), the consensus was that special diets do not provide any special benefit to children with autism. ABC News’s Diane Sawyer then interviewed Jenny McCarthy, the one time Playboy model and television personality and now anti-vaccination activist who has all sorts of opinions on autism based on her expertise as a “mommy.” McCarthy’s response was to dismiss the survey of scientific studies, and her statement on the subject was: “Until doctors start listening to our anecdotal evidence, which is—it’s working, it’s going to take so many more years for these kids to get better.”
In addition to being a bit hypocritical—if you want people to listen to you, you must also listen to others yourself and consider what is said with as much respect as you would like given to yourself–this kind of thinking is a problem and it’s important to understand why.
Everyone has heard the age-old dismissal of “I’ll believe it when I see it.” If you are or ever were a budding skeptic, this may have been your motto at one time. Unfortunately, seeing something or experiencing it is not proof enough to believe in it. Personal experience is known in scientific terms as “anecdotal evidence.” Personal experience can be very convincing. Unfortunately, human perception is subject to many errors and weaknesses.
What seem to be logical connections sometimes aren’t really connections at all.
Correlation Does Not Always Equal Causation
One reason that it’s important to separate anecdotal evidence from concrete proof is the truism that “correlation does not always equal causation.” What this means is that if you take action “A,” and then “B” happens, it’s not necessarily true that “A” caused “B” to happen.
For example, let’s say that you have a rash on your skin. Someone tells you to rub butter on it to help it heal. If you follow that instruction and the rash goes away within two hours, you might come to the conclusion that the butter really did heal the rash. But if you had gone to a doctor, you might have learned that you had a type of rash that cleared up on its own after a few hours. The butter had nothing to do with it, and you would have achieved the same result either way.
Alternatively, imagine a psychiatrist who sees people suffering from schizophrenia that began in their late teens. During the course of therapy, he notices that a significant number of them state that they started smoking cigarettes when they were under the age of 14. If this pattern continues, one may come up with a hypothesis that cigarette smoking at an early age causes schizophrenia. If you just stop there, you may not be getting the whole story. Perhaps, if tested, it will be discovered that it is not the cigarettes causing the schizophrenia, but that the schizophrenia was present much earlier than it was detectable, and the early smoking was a primitive form of self-medication.
Human brains work by seeking out patterns and this is just another example of that. We want to see patterns, and sometimes we pick out patterns, but are not able to interpret them correctly with the information we have. The patterns we pick out might be a sign that there is a relation that deserves further study — but once the study is made, anecdotal evidence doesn’t prove the study wrong.
Failures of Memory and Perception
Not only are we susceptible to making false connections between events, but our perception and memory are quite malleable. Psychology studies have shown how easy it is to manipulate people’s perception of events so that they remember or experience something different than what actually happened. The book ‘Quirkology’ by Richard Wiseman describes some experiments that show how easily memory and perception are manipulated.
In the first, 20 people were asked to persuade a relative to participate in a study about childhood memories. The recruiters were asked to supply a photograph of the person as a child, and three other photographs of the person as a child participating in events. The first photograph was manipulated to produce a false photograph of a trip in a hot air balloon. The participants were then shown the three real photographs and one false one and asked to describe the events in the photos. About one third “remembered” the event in the hot air balloon. By the third interview, half of the participants were able to “remember” and describe in great detail their trip in the hot air balloon. Because the people thought they should remember the event, their brains helpfully supplied them with a vivid and detailed memory.
That experiment dealt with the unreliability of memory … but what about something that is not a distant memory? Another experiment described in ‘Quirkology’ deals with how easy it is to manipulate perceptions. The researchers purchased a simple brass curtain ring and a chrome light pull, which cost a total of £2.50. If they asked the people to hold the objects and describe what they felt, the people unsurprisingly described feeling nothing. However, all it took was for them to say the objects had been designed to make people feel unusual, and people began to describe all sorts of varied feelings, from tingling to feeling more relaxed. When asked how much they would pay for the items, the responses were between five and eight pounds. When the experimenters went one step further, donned lab coats and put the objects in nice boxes, the reactions became even more extreme with people reporting to feel “high” or that the objects were magnetic or electric. People were now willing to pay between 15 and 25 pounds for the objects. They thought they should feel something, so their brains helpfully supplied them with feelings that felt quite real to them.
This ease of manipulating perception with suggestion is what people selling the modern-day equivalent of snake oil count on: that you will hope and expect to see an effect, therefore you will see one. Ethical medicine and studies cannot lie. They cannot sugar coat a situation to make it better or to make you think something is happening when it isn’t. People who want to sell you false hopes have no such qualms.
It is all too easy to imagine that unfortunate mother, distraught over a child’s medical condition, easily gravitating toward the hope that her problems can be solved permanently by a simple change of diet. Someone who wants to believe her child will get better can be manipulated by suggestions that her child is getting better. If her child also suffers from digestive problems or food sensitivity it may appear that he is getting better, as he may be relieved of some discomfort even while the underlying condition remains. However, there is a difference between improvement and cure. Until some of these babies and toddlers who are showing such allegedly tremendous results reach an age where they must begin to fend for themselves and integrate in society, it will be hard to tell how truly miraculous this cure is . . . when the only evidence we have are the subjective reports of the parents.