Counterfactuals are perilous. Adam Sandler take notice.

Here’s the misleading headline and opening from Dan Reilly’s article about movies in Fortune magazine:

Netflix’s ‘Murder Mystery’ Would’ve Killed With a $120 Million Opening Weekend—If the Adam Sandler Comedy Ran in Theaters

If Adam Sandler’s new Netflix movie Murder Mystery drew the same U.S. box office numbers as it did via streaming, it would have made $120.5 million domestically in its opening weekend.

Do you see the fallacy?

Always be skeptical of any description that starts with the word “If.” Because once you assume an outcome at odds with the facts, you’re on shaky ground.

If I raised my rate to $5000 per hour, I could make ten times my current income.

If Hillary Clinton campaigned in Wisconsin, she would have won the election.

If Will Smith were white, he’d never be a big star.

The problem is that reality is what it is. Will Smith is black — there is no “white version” of Will Smith. If Hillary Clinton had campaigned in Wisconsin, she might still have lost the state — or whatever state she didn’t campaign in instead. If I charge ten times my normal rate, my clients will desert me — or I might get a different type of clients, but there’s no way to be sure.

What’s wrong with this headline

Let’s pick apart Reilly’s counterfactual. Let’s assume that instead of releasing this movie on Netflix, they released it in theaters.

Narrow opening or wide release?

What does the marketing campaign look like?

Netflix doesn’t release movies in theaters. So they would have had to hire someone else to do it. Who? How would that work? Would it show up on Netflix at the same time? A month later? A year later?

Metacritic’s rollup of critics’ reviews gives “Murder Mystery” a lame score of 38 out of a possible 100. The Globe and Mail called it “dumb, pointless, and completely bereft of laughs.” This is clearly not a very good movie. Would word of mouth have killed it quickly after release?

The shallowest possible analysis is that everyone who streamed the movie would have gone to the theater to see it. And halfway through Reilly’s article, he alludes to this:

Of course, there’s no way to directly measure Murder Mystery’s streaming success to a traditional theatrical release. Netflix, which has 148.9 million subscribers, counts a “view” of its content when an account has watched 70% of a title, though that doesn’t account for any unintentional streams of the flick due to autoplay. Netflix also can’t measure the number of people who actually sat in front of a television to watch one of its titles, nor those who walked away 20 minutes into a movie without hitting stop.

Most importantly, there’s no equivalent—at least not yet—for showing how many people would have actually left their homes and paid $9 for a ticket to see this, as opposed to just randomly choosing to watch it as part of their $13-per-month Netflix subscription. Perhaps Murder Mystery would have matched the success of Sandler and Aniston’s 2011 team-up, Just Go With It, with its $215 million worldwide haul. Or maybe it would’ve matched 2015’s Sandler ensemble film Pixels, which grossed $244 million. It’s impossible to know.

Translation: “I’m just making stuff up. What the heck, it makes for an eye-catching headline.”

How to analyze counterfactuals

Postmortem analysis is useful. Every business scenario includes decisions that could have been made differently. And it’s far from useless to ask the question “What would have happened if we did things differently?”

The answer is: you cannot know, because the connections between cause and effect are always complicated.

So proceed with care:

• Analyze comparable situations to determine the likely effects of decisions different from what actually happened.
• Use statistical analysis to determine the level and significance of correlation between decisions and outcomes.
• Examine multiple scenarios, looking at not just the questioned decision but all the other decisions that would follow from it.
• Avoid blanket statements about what would have happened — replace them with statements about the likelihood of different outcomes, with caveats about uncertainty.
• Recommend methods, not just to make better decisions in the future, but to determine the best way to predict the effect of those decisions.

Counterfactual analysis is fraught with challenges. Glib predictions about what “would have happened if” are always questionable.