Have you heard the term “lamp post metric”? [i] This is a measurement that is easy to gather and may even seem like common sense. If you lose something on a dark night, where are you going to look? Without a doubt, you will look under the lighted areas of the lamp posts where you last walked. Why did you look there? Because it was easy to see. When you pick metrics because they are comparatively easy for you to gather or based on conventional wisdom (“we have always done it this way”) you are using “lamp post metrics.”
The premise of “Moneyball”[ii] by Michael Lewis was that the collective wisdom and expertise of the baseball elite was more often than not flawed. The metrics they were using such as stolen bases, batting average and runs batted in were not providing the valued insight to help baseball teams achieve the desired outcomes of winning championships given their market constraints. Billy Beane, who is currently executive vice president of baseball operations for the Oakland Athletics, decided to evaluate potential players using measurements that were more closely tied to the outcome of winning games over individual statistics. His suggestions were derided by the established coaches and scouts of baseball because this new suggested way of valuing baseball players required a mindset shift.
Is your firm using “lamp post metrics”? Story points, hours burned per sprint, velocity, code coverage, number of tests executed and budget expended against the project plan are a few examples of measures that many if not most IT shops and Project Management Organizations are focusing on in their management meetings. Is it possible to deliver 100% of planned scope within budget and on time, yet still be unsuccessful? These metrics are often not good proxies to the desired outcome and may contribute to sub-optimization of your ability to deliver value to your customers even while 100% of your internal service level agreements are met. What we measure will be inferred as important by our teams, so it should not be surprising that specialized teams will focus on their measure possibly at the expense of the complete deliverable. Being able to lean into Effectiveness over Efficiency will take a mindset shift and this journey is not for the faint of heart. If your organization is to thrive in the Era of Customer Experience your metrics need to be customer focused.
The Evidence-Based Management framework published by Scrum.org encourages people to think about desired outcomes from your customer’s viewpoint carefully and outlines four Key Value Areas: Current Value; Time to Market; Ability to Innovate; and Unrealized Value. There are several possible Key Value Measures that one could consider in each of these Value areas, but it is essential to focus on one or two key value areas at a time so that progress against a hypothesis or experiment can be evaluated. To help in choosing a measure to work with you would want to consider knowledge about your baseline or current state and what is your goal or desired state. Why do your customers choose your product or service? Do you understand your key customer profiles? How might you measure that? My guess is story points, hours or velocity is not the answer. When working with your customer profiles, have you defined any hypotheses about features that your customers may want and an outcome that can be measured? This approach is called Hypothesis Driven Delivery.
Useful Measurement combined with visualization will provide better insight that will lead to better decisions, and better decisions will help you arrive at better outcomes. That said, when looking to choose a measure, first determine your desired outcome. Start by thinking about your desired outcomes and current constraints to make decisions on what type of insight would be helpful to achieve the desired result. When you understand the outcomes that are important to your stakeholders, you will be in a better position to form a hypothesis and select metrics or measures that are at least a reasonable proxy of the desired outcome.
If you take anything away from this article, please change your focus from being “efficient” and make your aim to be effective. Use your Sprint Retrospectives to reflect on your measures and if the changes made have impacted any of them. Are your defect trends going down such that your ability to innovate is increasing? Have you reduced your Cycle Time, thus improving time to market? Is the Usage Index for a feature improving or Product Cost Ratio decreasing such that Current Value may be growing? This insight will aid your ability to continuously improve as judged by your customer.
[i] http://maccherone.com/larry/2017/09/13/odim-a-framework-for-choosing-the-right-metrics/
[ii] https://venturebeat.com/2011/06/23/book-review-moneyball-by-michael-lewis/