The titular phrase “anecdotal evidence shows that…” is very familiar to us – with good reason. Not only is it very commonly used, but it is subject to a great deal of misuse. It generally makes an assertion that something is probably true because there is some observed evidence to support it. While that evidence does not rise to the level of proof, it does at least create some factual basis for wishful thinking.

Anecdotal evidence is important. It is often the only evidence we can obtain. In many areas, scientists cannot practically conduct a formal study, or it would be ethically wrong to do so. It may simply be an area of study that no one is willing to fund. Therefore, even scientists often have no alternative but to base conclusions upon the best anecdotal data they have.

Anecdotal evidence is essential to making everyday decisions as well. We don’t normally conduct formal studies to see if our friend Julie is a thief. But if ear rings disappear each time she visits, we have enough anecdotal evidence to at least watch her closely. Likewise, even court proceedings must often rely upon anecdotal evidence, which is slightly different than circumstantial evidence.

Knowing when anecdotal evidence is telling, when it is simply a rationalization for wishful thinking, and when it is the basis for an outright con job is not always easy. The fact that sometimes all we have to work with is anecdotal evidence makes it all that much more dangerous and subject to misuse and abuse.

All too often, anecdotal evidence is simply poor evidence. I once presented anecdotal evidence of ghosts by relating a harrowing close encounter that I had. The thing was, I totally made it up (see here). People don’t always intentionally lie when they share an anecdote, but those people who in good faith repeated my story to others were nevertheless sharing bad anecdotal information.

Testimonials are a form of anecdotal claim. Back in the 1800’s a Snake Oil Salesman would trot out an accomplice to support his claims of a miracle cure. Today we see everyone from television preachers to herbal medicine companies use the same technique of providing anecdotal evidence through testimonials. Most of these claims are no more legitimate than my ghost story.

We also see anecdote by testimony performed almost daily in political theatre. The President points to the crowd to identify a person who has benefitted greatly from his policies. In Congressional hearings, supposedly wronged parties are trotted out to give testimony about how badly they were harmed by the actions of the targeted party. Both of these individuals are put forth as typical examples yet they may be exceedingly unusual.

So here’s the situation. We need anecdotal evidence as it is often all we have to work with to make important decisions that must be made. However, basing decisions on anecdotal information is also fraught with risk and uncertainty. How do we make the wisest use of the anecdotal information that we must rely upon?

First, consider the source and the motive of the anecdote. If the motive is to try to persuade you to do something, to support something, to accept something, or to part with your cash, be particularly suspect of anecdotal claims or testimonials. One great example are the Deal Dash commercials. You hear a woman claim that she “won” a large screen television for only $49. Sounds great, until you realize that the anecdote doesn’t tell how many bids she purchased to get it for $49, how much she wasted on other failed auctions, and how much was spent in total by the hundreds of people bidding on that item. Anecdotal evidence are not always an outright lies, but they can still tell huge lies by omission and by cherry-picking.

Second, consider the plausibility of the anecdote. If the anecdote claims to prove that ghosts exist, someone made it up. Likewise with god or miracles or angels or Big Foot. Just because someone reports something incredible, no matter how credible that person may be, demand credible evidence. As Carl Sagan pointed out, “extraordinary claims require extraordinary evidence.”

Third, consider the scope of the anecdotal claim. Does it make sweeping generalizations or is it very limited in scope? If the claim is that all Mexicans are rapists because one Mexican was arrested for rape, we end up with a Fallacy of Extrapolation which is often the result of the misuse of anecdotal information.

Finally, consider the cost/benefit of the response to the anecdotal claim. If the anecdote is that eating yoghurt cured Sam’s cancer, then maybe it’s reasonable to eat more yoghurt. But if the anecdote is that Ed cured his cancer by ceasing all treatments, then perhaps that should be considered a far more risky anecdote to act upon.

Anecdotal information is essential. Many diseases such as AIDS have been uncovered by paying attention to one “anecdotal” case report. In fact, many of the important breakthroughs in science have only been possible because a keen-eyed scientist followed up on what everyone else dismissed as merely anecdotal or anomalous data.

Anecdotes are best used to simply make the claim that something may be possible, but without any claims as to how likely it is. For example, it may be that a second blow to the head has seemed to cure amnesia. However, this cannot be studied clinically and it is not likely to occur often enough to recommend it as a treatment. Still, sometimes it is extremely important to know that something has been thought to happen, no matter how uncertain and infrequent. If a severe blow to the head MAY have cured amnesia at least once, this can help to inform further research into it.

Don’t start feeling overwhelmed. We don’t actually need to stop and consciously analyze every anecdote in detail. Our subconscious pattern-recognition machines are quite capable of performing these fuzzy assessments for us. We only need to be sure to consciously internalize these general program parameters into our pattern recognition machines so that they produce sound conclusions when presented with claims that “anecdotal evidence shows.”