Frequently Asked Questions about Evidence-Based Management
Q: What is the relationship between EBM and OKRs?
A: OKRs are a powerful technique for setting goals and measuring against goals. Some organizations use OKRs as a practice that helps them with EBM. What EBM adds to the OKR technique is an agile and iterative way for the organization to improve their performance in pursuit of OKRs. Specifically, using EBM’s “experiment loop” provides a way for teams to empirically test improvement ideas, and even inspect and adapt objectives while they are working on improvements. In addition, tEBM’s Key Value Areas provide focus for measurements and improvements by encouraging teams to look at specific kinds of measures as they form their ideas for improvement.
For an additional perspective on EBM and OKRs, see OKRs: The Good, the Bad, and the Ugly.
Q: How can an organization get started applying EBM?
A: The obvious place to start is to ask yourself “What is my organization trying to achieve?”, i.e. what is its Strategic Goal. This is often a hard question to answer because many organizations have rather vague and qualitative goals and have a hard time articulating how they would know if they have achieved the goal.
Sometimes organizations find it easier to talk about the goals for their agile initiative or agile transformation. A frequently expressed goal is that the organization wants to “deliver to customers faster”, or to “improve efficiency.” If they do, they should ask themselves “why do we want to do these things?” and keep asking why until they can express the fundamental reason why they want to be more agile. We find that this reason is often related to seizing some customer-related market opportunity. In EBM terms, they seek to realize some currently Unrealized Value in the market. The concept of Unrealized Value is explored more here.
As part of this discussion, they should ask themselves, “how will we know when we have achieved this goal? What measure will tell us that we have achieved the goal?” This will help them achieve focus in pursuit of their goal.
Since the achievement of Strategic Goals often takes years, organizations need nearer-term targets to help them move toward their long-term goals. Discussions about immediate next steps and intermediate steps give rise to Immediate Tactical Goals and Intermediate Goals, each with their associated measures that will tell the organization when they have achieved their goal.
Q: How do EBM’s Key Value Areas help organizations achieve their goals?
A: EBM’s Key Value areas provide organizations with conceptual lenses that can help them focus on specific areas in which they may need to improve. First, an organization should understand the Current Value that it delivers to customers, and whether they have any opportunities to improve the value that their customers experience (Unrealized Value).
If the organization can deliver and measure small increments of value, they may focus on improving Current Value and reducing Unrealized Value. Many organizations are looking to improve their agility because they cannot deliver and measure small increments of value. As a result, they may need to focus first on improving their Time to Market. If they can deliver quickly but each increment of value is very small because the organization is stretched too thin, the organization may need to also improve its Ability to Innovate.
Q: Can EBM be used to manage portfolios of products and services?
A: Yes. There is a white paper discussing this topic. See Investing for Business Agility.
Q: Why are leading and lagging indicator concepts not part of EBM?=
A: Leading indicators imply at least correlation, if not causation, with a lagging indicator. They are useful concepts in simple domains where cause and effect are relatively obvious. In complicated domains, they may still be useful even when causality is not certain but correlation is relatively reliable.
In complex domains, those in which using empiricism to seek toward a goal is the only practical way to proceed, cause and effect, and even correlation, may not be evident until after the event, if they are discernable at all. Consequently, we don’t think focusing on identifying causal relationships or leading indicators is fruitful.
Every assertion about a potential leading-lagging relationship is, in complex domains, simply a hypothesis. EBM uses an experimentation loop to form, test, inspect, and adapt hypotheses, so if a team wishes to explore leading-lagging relationships, they can use this mechanism to do so. In a sense, every experiment is testing the hypothesis that “if we do this thing, that measure will improve”, and teams can use this to gradually progress toward their goals.
What we find, however, is that each experiment loop is testing a different hypothesis. As teams try things, inspect the results, and adapt based on those results, they are constantly testing different ideas. They are not seeking to establish a simple leading-lagging relationship, but rather are gathering clues about what their next improvements might be. Their world is not simple enough for leading and lagging relationships to be very useful for very long.
Q: How does the concept of Unrealized Value help organizations to measure their success?
A: There is a blog that discusses this topic; see Measure Business Opportunities with Unrealized Value
Q: Why these four Key Value Areas and not others?
A: Current Value is important to understand how a product or service delivers valuable outcomes to customers or users of a product. But because it focuses only on what the customer/user experiences today, we added an additional Key Value Area, Unrealized Value, to reflect that there may be outcomes that the customer/user would like to experience but do not today.
The other two Key Value Areas look at the organization’s capability to deliver value from two perspectives: Time-to-Market, or speed, and Ability to Innovate, which focuses on the effectiveness of the efforts of the organization at delivering value.
We have found that these four perspectives provide organizations with a holistic view on areas in which they might need to improve. If we find that other perspectives are also useful, we may add those in the future.
Q: On what intervals should organizations inspect and adapt their goals?
A: We use the experiment loop as the means by which organizations form hypotheses, run experiments, inspect the results, and adapt their thinking on what to do next. We have found that this loop is most effective when it is no longer than a month, and ideally faster. When a team or organization inspects the results of an experiment, they should also inspect their goals to determine whether the goals are still valid. Market conditions, business opportunities, and solution alternatives change all the time, and goals can shift as a result.
Unlike some goal-setting approaches, EBM does not assume that goals are always right; they need to be frequently examined to determine whether they are the right goals. The best time to do this is when the results of experiments are inspected, on at least a monthly basis, if not more frequently.
Q: Do organizations have to use Scrum to use EBM?
A: No. Although our experience with Scrum has certainly informed and shaped our perspectives on empirical management, EBM does not require any particular approach other than forming hypotheses, running experiments, inspecting the results, and adapting based on feedback on at least monthly cycles. If an organization already uses Scrum, this cycle will feel very familiar, but EBM can be used with any approach provided that the organization can run experiments quickly enough.
Q: Why doesn’t EBM talk about measures like profit, revenue, or EPS as Strategic Goals?
A: Business results like profit, revenue, EPS, and the like are indirect measures that have many components. Most of these measures can also be gamed for short-term advantage, in ways that may destroy the long-term viability of the organization.
As an example, an organization might cut costs in a way that shows short-term gains, only to have valuable employees with critical knowledge leave the organization, which would affect the organization’s long-term viability.
EBM is founded on the principle that all business value is created by closing customer satisfaction gaps. If an organization focuses on doing this while also reducing waste by improving Time-to-Market and Ability to Innovate, profits will come. In addition, focusing the areas defined by EBM’s Key Value Areas gives people in the organization concrete and motivating targets for improvement, while purely abstract financial goals do not.
Q: How can I learn more about EBM?
A: Scrum.org has developed a workshop to help students to understand and apply EBM concepts. For more information, look here.