Data Science: The Hard Parts

Chapter 1. So What? Creating Value with Data Science

Data science (DS) has seen impressive growth in the past two decades, going from a relatively niche field that only the top tech companies in Silicon Valley could afford to have, to being present in many organizations across many sectors and countries. Nonetheless, many teams still struggle with generating measurable value for their companies.

So what is the value of DS to an organization? I’ve found that data scientists of all seniorities struggle with this question, so it’s no wonder the organizations themselves do so. My aim in this first chapter is to delineate some basic principles of value creation with DS. I believe that understanding and internalizing these principles can help you become a better data scientist.

What Is Value?

Companies exist to create value to shareholders, customers, and employees (and hopefully society as a whole). Naturally, shareholders expect to gain a return on their investment, relative to other alternatives. Customers derive value from the consumption of the product, and expect this to be at least as large as the price they paid.

In principle, all teams and functions ought to contribute in some measurable way to the process of value creation, but in many cases quantifying this is far from obvious. DS is not foreign to this lack of measurability.

In my book Analytical Skills for AI and Data Science (O’Reilly), I presented this general approach to value creation with data (Figure 1-1). The idea is simple: data by itself creates no value. The value is derived from the quality of the decisions that are made with it. At a first level, you describe the current and past state of the company. This is usually done with traditional business intelligence (BI) tools such as dashboards and reports. With machine learning (ML), you can make predictions about the future state and attempt to circumvent the uncertainty that makes the decision process considerably harder. The summit is reached if you can automate and optimize some part of the decision process. That book was all about helping practitioners make better decisions with data, so I will not repeat myself here.

Figure 1-1. Creating value with data

As intuitive as it may be, I’ve found that this depiction is too general and abstract to be used in practice by data scientists, so over time I’ve translated this into a framework that will also be handy when I introduce the topic of narratives (Chapter 7).

It boils down to the same principle: incremental value comes from improving an organization’s decision-making capabilities. For this, you really need to understand the business problem at hand (what), think hard about the levers (so what), and be proactive about it (now what).

What: Understanding the Business

I always say that a data scientist ought to be as knowledgeable about the business as their stakeholders. And by business I mean everything, from the operational stuff, like understanding and proposing new metrics (Chapter 2) and levers that their stakeholders can pull to impact them, to the underlying economic and psychological factors that underly the business (e.g., what drives the consumer to purchase your product).

Sounds like a lot to learn for a data scientist, especially since you need to keep updating your knowledge on the ever-evolving technical toolkit. Do you really have to do it? Can’t you just specialize on the technical (and fun) part of the algorithms, tech stack, and data, and let the stakeholders specialize on their (less fun) thing?

My first claim is that the business is fun! But even if you don’t find it exhilarating, if data scientists want to get their voices heard by the actual decision-makers, it is absolutely necessary to gain their stakeholders’ respect.

Before moving on, let me emphasize that data scientists are rarely the actual decision-makers on business strategy and tactics: it’s the stakeholders, be it marketing, finance, product, sales, or any other team in the company.

How to do this? Here’s a list of things that I’ve found useful:

Attend nontechnical meetings.

No textbook will teach you the nuts and bolts of the business; you really have to be there and learn from the collective knowledge in your organization.

Get a seat with the decision-makers.

Ensure that you’re in the meetings where decisions are made. The case I’ve made for my teams at organizations with clearly defined silos is that it is in the best interest of everyone if they’re present. For example, how can you come up with great features for your models if you don’t understand the intricacies of the business?

Learn the Key Performance Indicators (KPIs).

Data scientists have one advantage over the rest of the organization: they own the data and are constantly asked to calculate and present the key metrics of the team. So you must learn the key metrics. Sounds obvious, but many data scientists think this is boring, and since they don’t own the metric—in the sense that they’re most likely not responsible for attaining a target—they are happy to delegate this to their stakeholders. Moreover, data scientists ought to be experts at metrics design (Chapter 2).

Be curious and open about it.

Data scientists ought to embrace curiosity. By this I mean not being shy about asking questions and challenging the set of accepted facts in the organization. Funny enough, I’ve found that many data scientists lack this overall sense of curiosity. The good thing is that this can be learned. I’ll share some resources at the end of the chapter.

Decentralized structures.

This may not be up to you (or your manager or your manager’s manager), but companies where data science is embedded into teams allow for business specialization (and trust and other positive externalities). Decentralized data science structure organizations have teams with people from different backgrounds (data scientists, business analysts, engineers, product, and the like) and are great at making everyone experts on their topic. On the contrary, centralized organizations where a group of “experts” act as consultants to the whole company also have advantages, but gaining the necessary level of business expertise is not one of them.

So What: The Gist of Value Creation in DS

Why is your project important to the company? Why should anyone care about your analysis or model? More importantly, what actions are derived from it? This is at the crux of the problem covered in this chapter, and just in passing I consider it one of those seniority-defining attributes in DS. When interviewing candidates for a position, after the necessary filter questions for the technical stuff, I always jump into the so what part.

I’ve seen this mistake over and over: a data scientist spends a lot of time running their model or analysis, and when it’s time to deliver the presentation, they just read the nice graphs and data visualizations they have. Literally.

Don’t get me wrong, explaining your figures is super important because stakeholders aren’t usually data or data visualization savvy (especially with the more technical stuff; surely they can understand the pie chart on their report). But you shouldn’t stop there. Chapter 7 will deal with the practicalities of storytelling, but let me provide some general guidelines on how to develop this skill:

Think about the so what from the outset.

Whenever I decide to start a new project, I always solve the problem backwards: how can the decision-maker use the results of my analysis or model? What are the levers that they have? Is it even actionable? Never start without the answers to these questions.

Write it down.

Once you have figured out the so what, it’s a great practice to write it down. Don’t let it play a secondary role by focusing only on the technical stuff. Many times you are so deeply immersed into the technical nitty-gritty that you get lost. If you write it down, the so what will act as your North Star in times of despair.

Understand the levers.

The so what is all about actionables. The KPIs you care about are generally not directly actionable, so you or someone at the company needs to pull some levers to try to impact these metrics (e.g., pricing, marketing campaigns, sales incentives, and so on). It’s critical that you think hard about the set of possible actions. Also, feel free to think out of the box.

Think about your audience.

Do they care about the fancy deep neural network you used in your prediction model, or do they care about how they can use your model to improve their metrics? My guess is the latter: you will be successful if you help them be successful.

Now What: Be a Go-Getter

As mentioned, data scientists are usually not the decision-makers. There’s a symbiotic relationship between data scientists and their stakeholders: you need them to put your recommendations into practice, and they need you to improve the business.

The best data scientists I’ve seen are go-getters who own the project end to end: they ensure that every team plays its part. They develop the necessary stakeholder management and other so-called soft skills to ensure that this happens.

Unfortunately, many data scientists lie on the other side of the spectrum. They think their job starts and ends with the technical part. They have internalized the functional specialization that should be avoided.

Tip

Don’t be afraid to make product recommendations even when the product manager disagrees with you, or to suggest alternative communication strategies when your marketing stakeholder believes you’re trespassing.

That said, be humble. If you don’t have the expertise, my best advice before moving to the now what arena is to go back to the what step and become an expert.

Measuring Value

Your aim is to create measurable value. How do you do that? Here’s one trick that applies more generally.

A data scientist does X to impact a metric M with the hope it will improve on the current baseline. You can think of M as a function of X:

$$\text{Impact}\text{of}X=M\left(X\right)-M\left(\text{baseline}\right)$$

Let’s put this principle into practice with a churn prediction model:

X

Churn prediction model

M

Churn rate, i.e., the percentage of active users in period t − 1 that are inactive in period t

Baseline

Segmentation strategy

Notice that M is not a function of X! The churn rate is the same with or without a prediction model. The metric only changes if you do something with the output of the model. Do you see how value is derived from actions and not from data or a model? So let’s adjust the principle to make it absolutely clear that actions (A) affect the metric:

$$\text{Impact}\text{of}X=M\left(A\right(X\left)\right)-M\left(A\right(\text{baseline}\left)\right)$$

What levers are at your disposal? In a typical scenario, you launch a retention campaign targeting only those users with a high probability of becoming inactive the next month. For instance, you can give a discount or launch a communication campaign.

Let’s also apply the what, so what, and now what framework:

What

How is churn measured at your company? Is this the best way to do it? What is the team that owns the metric doing to reduce it (the baseline)? Why are the users becoming inactive? What drives churn? What is the impact on the profit and loss?

So what

How will the probability score be used? Can you help them find alternative levers to be tested? Are price discounts available? What about a loyalty program?

Now what

What do you need from anyone at the company involved in the decision-making and operational process? Do you need approval from Legal or Finance? Is Product OK with the proposed change? When is the campaign going live? Is Marketing ready to launch it?

Let me highlight the importance of the so what and now what parts. You can have a great ML model that is predictive and hopefully interpretable. But if the actions taken by the actual decision-makers don’t impact the metric, the value of your team will be zero (so what). In a proactive approach, you actually help them come out with alternatives (this is the importance of the what and becoming experts on the problem). But you need to ensure this (now what). Using my notation, you must own $M\left(A\right(X\left)\right)$, not only $X$.

Once you quantify the incrementality of your model, it’s time to translate this to value. Some teams are happy to state that churn decreased by some amount and stop there. But even in these cases I find it useful to come up with a dollar figure. It’s easier to get more resources for your team if you can show how much incremental value you’ve brought to the company.

In the example this can be done in several ways. The simplest one is to be literal about the value.

Let’s say that the monthly average revenue per user is R and that the company has base of active users B:

$$\text{Cost}\text{of}\text{Churn}(A,X)=B\times \text{Churn}\left(A\right(X\left)\right)\times R$$

If you have 100 users, each one bringing $7 per month, and a monthly churn rate of 10% churn, the company loses $70 per month.

The incremental monetary value is the difference in the costs with and without the model. After factoring out common terms, you get:

$$\Delta \text{Cost}\text{of}\text{Churn}(A,\text{baseline},X)=B\times \Delta \text{Churn}(A;X,\text{baseline})\times R$$

If the previously used segmentation strategy saved $70 per month, and the now laser-focused ML model creates $90 in savings, the incremental value for the organization is $20.

A more sophisticated approach would also include other value-generating changes, for instance, the cost of false positives and false negatives:

False positive

It’s common to target users with costly levers, but some of them were never going to churn anyway. You can measure the cost of these levers. For instance, if you give 100 users a 10% discount on the price P, but of these only 95 were actually going to churn, you are giving away $5\times 0.1\times P$ in false positives.

False negative

The opportunity cost from having bad predictions is the revenue from those users that end up churning but were not detected by the baseline method. The cost from these can be calculated with the equations we just covered.

Key Takeaways

I will now sum up the main messages from this chapter:

Companies exist to create value. Hence, teams ought to create value.

A data science team that doesn’t create value is a luxury for a company. The DS hype bought you some leeway, but to survive you need to ensure that the business case for DS is positive for the company.

Value is created by making decisions.

DS value comes from improving the company’s decision-making capabilities through the data-driven, evidence-based toolkit that you know and love.

The gist of value creation is the so what.

Stop at the outset if your model or analysis can’t create actionable insights. Think hard about the levers, and become an expert on your business.

Work on your soft skills.

Once you have your model or analysis and have made actionable recommendations, it’s time to ensure the end-to-end delivery. Stakeholder management is key, but so is being likeable. If you know your business inside out, don’t be shy about your recommendations.

Further Reading

I touch upon several of these topics in my book Analytical Skills for AI and Data Science (O’Reilly). Check out the chapters on learning how to ask business questions and finding good levers for your business problem.

On learning curiosity, remember that you were born curious. Children are always asking questions, but as they grow older they forget about it. It could be because they’ve become self-conscious or a fear of being perceived as ignorant. You need to overcome these psychological barriers. You can check out A More Beautiful Question: The Power of Inquiry to Spark Breakthrough Ideas by Waren Berger (Bloomsbury) or several of Richard Feynman’s books (try The Pleasure of Finding Things Out [Basic Books]).

On developing the necessary social and communication skills, there are plenty of resources and plenty of things to keep learning. I’ve found Survival of the Savvy: High-Integrity Political Tactics for Career and Company Success by Rick Brandon and Marty Seldman (Free Press) quite useful for dealing with company politics in a very pragmatic way.

Extreme Ownership: How U.S. Navy Seals Lead and Win by Jocko Willink and Leif Babin (St. Martin’s Press) makes the case that great leaders exercise end-to-end (extreme) ownership.

Never Split the Difference by Chris Voss and Tahl Raz (Harper Business) is great at developing the necessary negotiation skills, and the classic and often-quoted How to Win Friends and Influence People by Dale Carnegie (Pocket Books) should help you develop some of the softer skills that are critical for success.