Building data-driven organizations, Part 2: Why organizations fail to make data-driven decisions

Case studies from a lean, evidence-based approach

This post is part 2 of a 3-part series on building data-driven organizations. Part 1 is here and part 3 is here.

I previously wrote about what it means to be data-driven, which at its core is about making decisions differently based on data. Companies that want to be data-driven should focus on improving decision-making.

It’s a simple concept to understand, so why is it so difficult to implement? In this post, I share examples that illustrate the most common challenges to data-driven decision-making.

Companies are decision factories

Imagine you’re in charge of quality assurance at a factory that produces thousands of widgets per day. There’s an assembly line, machinery, and many workers that keep the factory running. Sometimes widgets come out the wrong way because a machine malfunctions or because a worker doesn’t carry his weight. Your job is to monitor and fix these problems, and ultimately ensure widgets meet a high quality standard.

Companies are decision factories. And our job as data leaders is to ensure they produce high-quality, data-driven decisions. Just as producing high-quality widgets requires understanding the production line, building data-driven organizations requires deep understanding of decision-making processes and their failure modes.

The best way to build this understanding is with a bottom-up, evidence-based approach. Start by finding case studies of poor decisions within your company. For each case study, apply the 5 Why’s until you have truly understood the root cause of the problem. To have a more representative sample, try picking different types of decisions that cover a range of teams and business contexts.

Decision-making challenges are universal

If we run this exercise enough times on a broad cross-section of companies, we’ll notice some common themes. In hindsight, it’s not surprising that many decision-making challenges are universal. Modern companies share similar talent, technology, and growth challenges – the raw ingredients of decision-making processes.

When it comes to making data-driven decisions, the root causes of problems consistently fall into one of 3 categories: people, products, and partnerships. The rest of this post summarizes the common scenarios in each category.

Problems with people

Data analysts¹ are foundational to data-driven decision-making in the same way that product designers are foundational to well-designed products. It’s possible to make data-driven decisions without them, but you’re guaranteed to get worse results.

Case study 1: Under-resourced team

A product manager mis-prioritized his strategic roadmap. Why? It didn’t address the main growth opportunity for his product. Why? He wasn’t aware of a valuable customer segment that was being underserved. Why? He didn’t do a deep dive analysis during the product planning cycle to uncover the opportunity. Why? He didn’t have an analytics partner to work with, or the time to do the work himself.

The most common reason decision-makers ignore data is that no one is around to do the work. Well-intentioned product managers, marketers, and other stakeholders might attempt their own analyses, but lack the expertise and time to do a good job. Analysts also make their impact felt in meetings by debating points that lead to different outcomes. In fact, just having a “butt in seat” can go a long way: analytics representation builds general awareness of data and leads to better decisions.

Case study 2: Mismatch of talent

A platform team neglected to fix an algorithm that was causing poor user experiences. Why? The team incorrectly believed these were minor edge cases. Why? The team wasn’t aware how often the poor experiences were occurring. Why? The team didn’t have good observability and reporting on the algorithm. Why? The ML engineers who built the model didn’t understand the data engineering tools well enough to build the analytics themselves.

Sometimes there might be a mismatch of skills or seniority. It’s unproductive to staff a machine learning expert on a team that needs a deep dive analysis into customer behavior. It’s also a bad idea to staff the most junior analyst on a high-risk project with lots of visibility.

More commonly, mismatch problems surface as businesses scale over time. The person who built a scrappy v1 of a model may not be the best person to build a more sophisticated v5. It’s important to respond to these shifts to setup teams and business partners for success.

Case study 3: Underperforming team members

A growth marketer misallocated budget across several paid acquisition channels. Why? She didn’t consult her analytics partner to make investment decisions. Why? She didn’t trust the work of the analyst. Why? The analyst had a track record of producing flawed work. Why? The analyst has not been meeting expectations or taking enough ownership of his work.

When analysts underperform, stakeholders will get burned by poor quality work once or twice before losing trust in the analytics partner for future decisions. On the flip side, business stakeholders can underperform as well. Poor judgment and execution drags down the performance of the entire team, regardless of how strong the analytics might be. Holding your team and partners accountable is important not just for decision-making, but for establishing credibility as a leader.

Problems with products

If people are the foundation for making data-driven decisions, then products are the tools that enable them to be effective at their jobs. People and product problems often display similar symptoms. They both result in lower quality work and slower output, which in turn lead to poorer decision-making. It’s important to disambiguate the root cause of the problem.

Case study 4: Lack of self-service

A growth manager sent an email to the wrong list of users. Why? She accidentally included users that had already unsubscribed. Why? There was a bug in her SQL query, which mistakenly included unsubscribed users. Why? She didn’t understand the nuances of the underlying data model and missed an important filter. Why? It’s not her job to understand the data model, but she can’t get the data otherwise.

At every company, there comes a time when data analysts are overrun by ad-hoc questions from stakeholders who are hungry for data. While many people can learn to write SQL, not everyone can or should. To scale decision-making, we have to invest in self-serve data capabilities that empower stakeholders to access data with confidence. Regardless of the specific tool, the end goal is the same: disintermediate the data analysts so they are no longer the bottleneck.

Case study 5: Poor data governance

A product manager sounded a false alarm on a conversion rate decline. Why? The chart on her dashboard incorrectly showed a steep decline in the metric. Why? The SQL query that calculated the metric returned the wrong results. Why? The query referenced an analytics event that had stopped firing, which biased the calculation. Why? A product engineer had deprecated the event for a set of legacy app versions. Why? The engineer didn’t know about the downstream metrics dependency.

There’s an inherent tension between opening data to more consumers and governing how data is used. Growth-stage companies favor more data consumption at the expense of data governance, because the former drives direct business value. But data governance problems will catch up. It’s only a matter of time before broken metrics and dashboards, data quality issues, and unmanaged datasets flare up and impact an important decision.

Case study 6: Lack of productivity tools

The executive team overinvested in growth for next month. Why? The business operations team produced a recommendation based on a forecast that contained a bug. Why? The analyst on the team made a data entry error. Why? Producing a forecast is an error-prone process that requires many hours worth of manual data operations. Why? The team lacked tooling to automate some of the manual efforts.

Analytics productivity tends to lag behind that of peer functions. Unlike engineering teams who can build their own tools and business teams who enjoy large budgets due to their proximity to revenue and profits, analytics workflow challenges often fall by the wayside. Analyst productivity efforts are also neglected not because the impact isn’t there, but because it’s less directly visible. For this reason, upgrading data tools can be one of the highest ROI investments companies can make.

Problems with partnerships

Partnerships describe the relationships between data analysts and decision-makers. While we understand the problems with people and products, we grossly under-appreciate the impact of partnerships on decision-making. Partnership problems are subtle and may seem like process problems at first. But under the surface, processes are created by people with underlying beliefs and experiences working with data.

Case study 7: Lack of alignment

A product manager developed a long-term roadmap that was unsupported by data. Why? The analyst wasn’t involved in the planning process. Why? The product manager hadn’t invited the analyst to the main planning meetings. Why? He decided to task the analyst with gathering individual data points instead. Why? The product manager didn’t appreciate the analyst role as a strategic thought partner. Why? The product manager hadn’t worked with an analyst in a strategic thought partner capacity at previous companies.

Data analytics is new function compared to product management, engineering, and marketing. Titles, organizational structure, and roles for analytics vary significantly across organizations. As a result, business partners carry a range of assumptions on analytics teams based on varied past experiences at other companies. Some will know how to work with you right away, while others may have no idea. As leaders, it’s our responsibility to build trust, create alignment, and define the working relationship for our teams.

Case study 8: Lack of data literacy

An operations manager overstated the revenue impact of a pilot program he launched. Why? He saw revenue increase on his dashboard and incorrectly attributed it to his pilot. Why? He didn’t know the change was mostly driven by unrelated seasonal trends. Why? He didn’t investigate other potential drivers of the change. Why? He didn’t ask an analyst to do the analysis. Why? He was unaware of the possible measurement issues to begin with, and wanted to self-serve.

Giving someone a fishing pole is not the same as teaching someone to fish. Tools can empower decision-makers to self-serve, but a minimum degree of data literacy is required to use them correctly. For example, interpreting data on a chart or understanding correlation ≠ causation. Data literacy is also about knowing when it’s safe to self-serve and when it’s appropriate to ask for help. As leaders, we must give decision-makers the tools, but also train them on how and when to use them.

Case study 9: Personal agendas and biases

A company over-hired for its sales team. Why? Finance awarded extra headcount that should have gone to other teams instead. Why? The sales team overstated its contributions and growth projections to justify extra hires. Why? The sales analyst applied a revenue attribution model that claimed more credit for sales, at the expense of marketing. Why? The sales director had asked him to update the model. Why? The sales director had personal career aspirations to build a large sales organization.

Personal agendas and biases are among the heaviest boulders to move. With some business partners, making a data-driven decision can feel impossibly difficult when data doesn’t support their preconceived, strongly-held views. Business leaders who manage teams can also multiply their personal biases through their teams’ actions. In these situations, it’s important to shine a light on the behaviors and drive accountability for the decisions-making.

Building better decision factories

These case studies capture the most common themes that I’ve seen and heard over the years. Even if some of the themes don’t apply to your specific context, I encourage practicing the 5 Why’s technique as a way to stay mindful of decisions around you.

Now that we’ve understood what it means to be data-driven and why companies fail to get there, we’re ready to turn these insights into action. In my next and final post of this 3-part series, I’ll discuss how to operationalize data-driven decision-making within our organizations and build better decision factories.

Notes:

¹ For consistency, I exclusively use the title “data analyst” as opposed to “data scientist” or other titles. I’m aware that different titles can describe similar roles.