I have at points in my career argued exactly this point, but I have come to see some serious limits in this idea. The fundamental problem is that output accountability without decision-making authority is unstable.
Consider a simple case. A marketing DS manager reviews a launch plan for a product with a push messaging strategy that is extremely abusive; the median user will receive 10 push messages on launch day based on the message trigger logic and forecasted product adoption. She argues the case with her primary marketing partner, she argues the case with the PM, and everyone shrugs. "So what?" they say -- "we'll tune the messaging rules after launch if you can show that it's causing churn."
In the past I might have argued this was a failure "a truly effective analyst would have been able to convince people that this was an issue and to change their plan. If you failed to convince them, you need to consider how to take a different approach that might work better next time." However, I would now diagnose this as a problem with PM and GTM accountability. If they don't think an abusive use of the push notification system is a problem, there is no foundation for influence. No data scientist will win this argument because the PM is accountable for hitting her launch date and adoption targets. Someone else (maybe) is accountable for the push notification system unsub rate.
In other words, holding data staff accountable for outcomes only works in a case where their partners have good accountability models that are the same as the data scientist's. Maybe you have seen that more often than I have! But in my experience, there is constant tension between what a PM cares about and what a data scientist cares about, and that is a persistent barrier to an outcome accountability model.
I have found one useful bit of synthesis between these two perspectives. There is a spectrum of accountability between output and outcome, and the higher level someone is in an org, the more outcome-centric their accountability should be. So for an IC (anything less than principal IC) I prioritize output accountability because they don't own decisions about what to work on, or the decisions made by others that result from their work. For principal and IC managers, I expect outcomes BELOW their level reliably, and AT their level sometimes. In other words, a line manager data scientist should be able to convince IC PMs of important matters. But may or may not reliably get positive outcomes from peer PM managers. And they should only be accountable for securing good outcomes when influencing UP their own reporting chain, one step. But they should not be accountable for influencing "UP and ACROSS" the org. That is the job of someone above them in their own management chain who has the necessary context and peer relationships across the org to reset the accountability of other staff to an appropriate level.
The spectrum of accountability between output and outcome makes sense to me conceptually. And I think you'd already see a pretty meaningful change if senior ICs/IC managers owned the outcomes at their levels. However, I do think there's typically room for people to influence outcomes +1 or +2 levels above them. And often the people who successfully do this are the ones who get promoted, because they are already operating at that level.
Re: accountability structures, completely agree you need good accountability models on the partner side. In your push notification example, what I would expect is for the DS to escalate to their manager, who would then have a conversation with the PM manager. (And continue up the chain until there's a resolution). My general observation is there's a lot more escalation from PM -> DS than the other way around, and that need not be the case.
Huh, I have almost NEVER seen a PM->DS escalation! I wonder why that's different. Maybe relative status of the functions in the respective orgs? Or maybe we weren't pushing hard enough to yield important conflict on issues PMs cared about? Hmm.
It may depend on how we think about "outcomes." You could plausible say "the outcome for the DS is for the PM to observe and accept the risk they identify." In which case the DS in my example DID create an appropriate outcome. Versus saying "the outcome we desire is company-optimal results from the launch given some reasonable resource limits." That's a more reasonable expectation for very senior managers.
The former was more what I leaned towards at the end of my tenure, up to ~the director level. DS are accountable for identifying and clearly communicating their results, observations, and conclusions. We as data staff evaluate our own performance in that regard; if your peers and data management think you did that well, you are succeeding. If the recipient of the results chooses to ignore it, that's their choice for which they are accountable.
Haha! I’d say many of the PM->DS escalations were around resourcing/prioritization when people pushed back. Whereas the DS->PM escalations were around product decisions.
I have at points in my career argued exactly this point, but I have come to see some serious limits in this idea. The fundamental problem is that output accountability without decision-making authority is unstable.
Consider a simple case. A marketing DS manager reviews a launch plan for a product with a push messaging strategy that is extremely abusive; the median user will receive 10 push messages on launch day based on the message trigger logic and forecasted product adoption. She argues the case with her primary marketing partner, she argues the case with the PM, and everyone shrugs. "So what?" they say -- "we'll tune the messaging rules after launch if you can show that it's causing churn."
In the past I might have argued this was a failure "a truly effective analyst would have been able to convince people that this was an issue and to change their plan. If you failed to convince them, you need to consider how to take a different approach that might work better next time." However, I would now diagnose this as a problem with PM and GTM accountability. If they don't think an abusive use of the push notification system is a problem, there is no foundation for influence. No data scientist will win this argument because the PM is accountable for hitting her launch date and adoption targets. Someone else (maybe) is accountable for the push notification system unsub rate.
In other words, holding data staff accountable for outcomes only works in a case where their partners have good accountability models that are the same as the data scientist's. Maybe you have seen that more often than I have! But in my experience, there is constant tension between what a PM cares about and what a data scientist cares about, and that is a persistent barrier to an outcome accountability model.
I have found one useful bit of synthesis between these two perspectives. There is a spectrum of accountability between output and outcome, and the higher level someone is in an org, the more outcome-centric their accountability should be. So for an IC (anything less than principal IC) I prioritize output accountability because they don't own decisions about what to work on, or the decisions made by others that result from their work. For principal and IC managers, I expect outcomes BELOW their level reliably, and AT their level sometimes. In other words, a line manager data scientist should be able to convince IC PMs of important matters. But may or may not reliably get positive outcomes from peer PM managers. And they should only be accountable for securing good outcomes when influencing UP their own reporting chain, one step. But they should not be accountable for influencing "UP and ACROSS" the org. That is the job of someone above them in their own management chain who has the necessary context and peer relationships across the org to reset the accountability of other staff to an appropriate level.
Thanks for the thoughtful reply!
The spectrum of accountability between output and outcome makes sense to me conceptually. And I think you'd already see a pretty meaningful change if senior ICs/IC managers owned the outcomes at their levels. However, I do think there's typically room for people to influence outcomes +1 or +2 levels above them. And often the people who successfully do this are the ones who get promoted, because they are already operating at that level.
Re: accountability structures, completely agree you need good accountability models on the partner side. In your push notification example, what I would expect is for the DS to escalate to their manager, who would then have a conversation with the PM manager. (And continue up the chain until there's a resolution). My general observation is there's a lot more escalation from PM -> DS than the other way around, and that need not be the case.
Huh, I have almost NEVER seen a PM->DS escalation! I wonder why that's different. Maybe relative status of the functions in the respective orgs? Or maybe we weren't pushing hard enough to yield important conflict on issues PMs cared about? Hmm.
It may depend on how we think about "outcomes." You could plausible say "the outcome for the DS is for the PM to observe and accept the risk they identify." In which case the DS in my example DID create an appropriate outcome. Versus saying "the outcome we desire is company-optimal results from the launch given some reasonable resource limits." That's a more reasonable expectation for very senior managers.
The former was more what I leaned towards at the end of my tenure, up to ~the director level. DS are accountable for identifying and clearly communicating their results, observations, and conclusions. We as data staff evaluate our own performance in that regard; if your peers and data management think you did that well, you are succeeding. If the recipient of the results chooses to ignore it, that's their choice for which they are accountable.
Haha! I’d say many of the PM->DS escalations were around resourcing/prioritization when people pushed back. Whereas the DS->PM escalations were around product decisions.