The Perils of Data Story Telling: The Virtues of Data Documentaries

Features

  • Author: Thomas Speidel
  • Date: 04 Aug 2015
  • Copyright: Image 1 ©iStockphoto.com/MyosotisRock; Google, Weather.com; New York Times; Harding Center for Risk Literacy

Many times, my young children ask me to read them a story before sleeping. We have wide variety of children's books from which to choose. My children tend to go back to the same stories: those that capture their attention, spark their imagination, and have an appealing ending. They enjoy stories for which I inject theatrical creativity, a thick accent, a singing tone, or want participation from my small audience to become active part of the story. Good stories capture our attention and imagination, engage the audience. They are cohesive and become memorable. We all like a good story.

Several articles and blog posts have highlighted how data storytelling is so important for the organization. Being able to captivate the audience and persuade them to change their mind is an attribute that requires the ability to communicate a good story. Or so it is argued. It makes perfect intuitive sense.

thumbnail image: The Perils of Data Story Telling: The Virtues of Data Documentaries

We need to highlight the detrimental side of storytelling, which can undermine good decision making. When we are trying to build a story from data we fail to recognize that most stories we are supposed to glean from data are cognitively dissonant. Wikipedia defines cognitive dissonance as a "discomfort experienced by an individual who holds two or more contradictory beliefs, ideas, or values at the same time, or is confronted by new information that conflicts with existing beliefs, ideas, or values". The more professionals enter the field of data science with limited experience in data analysis, the more we need to become aware of the distinction between information and persuasion.

In one of my favourite articles, Professor David Spiegelhalter illustrates the difference between informing and persuading. He writes: "when designing a communication, the desired outcome must be considered from the start" which means that we cannot inform and persuade at the same time (Spiegelhalter, 2011). By telling a story, we are trying to persuade and influence our audience's behaviour. We do so, not only by way of cherry picking the story, but also by utilizing mental tricks such as positive and negative framing, neglecting to mention conflicting findings, tying the story to cultural beliefs or strong emotions, using strong confident language. In other words, we are devoiding the story of any incoherence.

Author’s illustration of a hypothetical drug test to guide decision making: statistical results are often counter-intuitive. Of 1,000 people tested, only 85 are expected to be true positive despite good test accuracy. Conveying accurate information is crucial for optimal decisions. The information from the diagram should make it clear to the decision maker what drawbacks might surface as a consequence of a decision. For instance, the green icons on the left (false positives) might be unjustly disqualified from a competition on the basis of a drug test, costly litigation may ensue. Cherry picking results, such as the 72% test accuracy, results in poor decisions.

However, when the intent is to make optimal decisions we ought to be transparent by revealing the uncertainties, the ambiguities and the assumptions, not hiding them. In some sense, we are providing a non-story, something more akin to a documentary.

Movies are to data stories like documentaries are to written reports

When analyzing data, the same data contain myriads of stories, often conflicting and even contradicting one another. In fact, it is not all that common to encounter a compelling story from a single dataset or a single analysis.

Most data contain too much uncertainty to lend themselves to coherent stories, ones with a beginning and end. We will always find an alternative explanation conflicting with the main story: analyze data a different way, add a different variable, remove some outliers and exclude some data. Here is an experiment: give the same data to 5 different data scientists and ask them to come up with what the main story the data is telling us. Make sure you blind the analysts to one another. What we will discover from this exercise is that most stories will be different: different actors, different endings, and different plots. Now what?

We now know that humans have a tendency to see patterns where none exist: "Fold the $20 US bill in just the right way and not only the Pentagon but also the Twin Towers in flame are revealed" (Good, Hardin, 2012). Kahneman and Tversky illustrate how easily our judgement is fooled by these and other heuristics and biases. They call it the representativeness heuristic. They also illustrate how humans are inclined to assign causes. The statistician George Box also talked about "special causes", whereby if a deviation from a process exists, such as a manufacturing process, we feel the need to eliminate the causes of such drift, even when they may be a perfectly normal natural variation.

Faced with the incoherence and multitude of challenges, we tend to cherry pick what story to communicate: the one that fulfills a vested interest, the one that impresses our peers and managers, the one that is more likely to be published or simply the one that we believe more credible.

While Kahnmen and Tversky focused on the cognitive aspect, the multiple comparison problem is a well-established, largely unsolved, problem when making inference from data. Loosely said, it means that the more you look at the data, the more likely you are to find interesting patterns even when none exist.

Faced with the incoherence and multitude of challenges, we tend to cherry pick what story to communicate: the one that fulfills a vested interest, the one that impresses our peers and managers, the one that is more likely to be published or simply the one that we believe more credible. Often, the chosen story is elegant and simple (something closely related to the principle of Occam's razor). We use data, many of which are not representative, we look for signals, many of which are false, we are faced with multiple clues, many of which are contradictory, we cherry pick the clues to tie it to a special cause and we tell our story. If our intent is to provide the information needed to make optimal decisions, are we not making a disservice to the organization?

Kahneman (2013) writes: "It is the consistency of the information that matters for a good story, not its completeness. [...] The confidence that individuals have in their beliefs depends mostly on the quality of the story they can tell about what they see, even if they see little. We often fail to allow for the possibility that evidence that should be critical to our judgment is missing".
We might wonder when data storytelling is justified. Stories are justified when our intention is to persuade and we have contextually plausible, compelling scientific evidence from valid data sources supporting the main message of the story.

Data Scientists should not be shy to suggest recommendations. However, the main responsibility of the data scientist is it to increase understanding and accuracy of the information for the audience, supposedly, the decision makers. Seldom is the data scientist the decision maker. And for good reasons. Have you ever wondered why the weatherman gives us the probability of rain (the information), and does not tell us "take the umbrella" (the decision)? Faced with a 60% chance of rain, a 15 year old will think differently than a 70 year old about taking the umbrella. Similarly, the data scientist does not always possess the knowledge related to desire, preference, convenience, reputation than, say, the executive might possess. We ought to relax what Edward Tufte calls the rage to conclude (Tufte, n.d.): while stories encourage conclusions and are memorable we must cautiously remind ourselves of the objective.


Weather forecast provide us with probability or risk of precipitation, that is, the information needed to make a decision, rather than the decision itself (image credit: Google, Weather.com).

In a promising move towards improved risk and statistical literacy, several government agencies are starting to display uncertainty around important key economic indicators, such us unemployment numbers (most noticeably, the UK Government Statistical Service and the Bank of England). In a catchy visualization, the New York Times simulated what that uncertainty might look like (http://goo.gl/ZQFeFS). What does this say about all the commentaries and articles - that is, "stories" - written on minute changes in unemployment numbers? What about the decisions organizations have made as a result of changes that could have been very well attributed to noise? A story containing so much ambiguity would not make for much of a story. We are a 'cause-effect seeking' species and stories tend to satisfy that innate desire. To quote Kahneman again: "We are pattern seekers, believers in a coherent world, in which regularities appear not by accident but as a result of mechanical causality or of someone's intention" (Kahneman, 2013).

A New York Times article visualizing uncertainty around job numbers (http://www.nytimes.com/2014/05/02/upshot/how-not-to-be-misled-by-the-jobs-report.html?abt=0002&abg=1)

We need to keep clear the distinction between data storytelling and communication skills. Choosing not to tell a story does not mean discouraging communication. Furthermore, we need to constantly remind ourselves what the desired objective is: to inform accurately or to persuade. And that is a trade-off, since we know that we cannot achieve both. There is little room for storytelling when we wish to inform our audiences, and while storytelling is an invaluable talent when we want to persuade, it is hard to imagine that all data science and analytic initiative results in such clear and compelling evidence to warrant immediate persuasion. Unless we also happen to be the decision makers, our clients, be it a customer or an executive, deserve first and foremost accurate information.

The Alternative

What are some of the alternatives to data storytelling? Carefully written reports, perhaps interactive ones, supported by plenty of visualizations, tables and annotations are effective tools to communicate findings. Analysts should borrow from health science to improve cognition around findings by utilizing methods that have been shown to minimize distortion and bias.


A fact box for the drug Tamoxifen as suggested by the Harding Center for Risk Literacy (https://www.harding-center.mpg.de/en/health-information/facts-boxes/nolvadex)

Managers, stakeholders and decision makers need to approach data stories with healthy skepticism: what are the alternative endings? What is the story that has not been told? If the role of statistics is to increase our understanding, extract insight from data to make better decisions, then, the legitimate goal is to inform rather than persuade. We aspire to present information as accurately as possible, free of distortions and cherry picking. Anything else makes us less scientists and more fortune tellers.

I thank Randy Bartlett for providing valuable suggestions.

References

[1] Bartlett, R. A Practitioner’s Guide to Business Analytics: Using Data Analysis Tools to Improve You Organization’s Decision Making and Strategy. McGraw-Hill Education, 2013.
[2] Ehrenberg, ASC. The Problem of Numeracy. The American Statistician, 35-2: 67-71, 1981.
[3] Gigerenzer, G. Risk Savvy: How to Make Good Decisions. Viking, 2014.
[4] Good, PI, and Hardin JW. Common Errors in Statistics. Hoboken, NJ: John Wiley & Sons, 2012.
[5] Kahneman, D. Thinking, Fast and Slow. Anchor Canada, 2013.
[6] Kloprogge P, Sluijs J, Wardekker A. Uncertainty Communication Issues and good practice. Copernicus Institute, 2007.
[7] Rosenzweig, P. The Halo Effect: ... and the Eight Other Business Delusions That Deceive Managers. Free Press, 2009.
[8] Spiegelhalter D, Pearson M, Short I. Visualizing Uncertainty About the Future. Science 9 Vol. 333 no. 6048 pp. 1393-1400, 2011
[9] Tufte, E (n.d.). Making better inferences from statistical graphics. Retrieved from https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0003wa
[10] Tufte, E. The Visual Display of Quantitative Information. 2001.

About the Author

Thomas Speidel, P.Stat., is a Statistician working for Suncor Energy in Calgary, Alberta, Canada. He spent ten years working in cancer research before moving to the energy industry. Thomas is often seen writing and commenting on issues of statistical literacy on LinkedIn, Twitter, several blog and is a co-founder of About Data Analysis, a LinkedIn group.

LinkedIn: ca.linkedin.com/in/speidel/en  
Twitter: @ThomasSpeidel
About.me: http://about.me/Thomas.Speidel

Related Topics

Related Publications

Related Content

Site Footer

Address:

This website is provided by John Wiley & Sons Limited, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ (Company No: 00641132, VAT No: 376766987)

Published features on StatisticsViews.com are checked for statistical accuracy by a panel from the European Network for Business and Industrial Statistics (ENBIS)   to whom Wiley and StatisticsViews.com express their gratitude. This panel are: Ron Kenett, David Steinberg, Shirley Coleman, Irena Ograjenšek, Fabrizio Ruggeri, Rainer Göb, Philippe Castagliola, Xavier Tort-Martorell, Bart De Ketelaere, Antonio Pievatolo, Martina Vandebroek, Lance Mitchell, Gilbert Saporta, Helmut Waldl and Stelios Psarakis.