TL; DR: The Agile Metrics Survey 2021
If you value agile practices, it is crucial to know if a job offering or a prospective business partner that claims to be “agile” really keeps its promises. Unfortunately, as agility usually cannot be observed directly, and certainly not from the outside of an organization, there is no way of knowing in advance if you will enter an agile environment that serves your own working needs or if a lot of frustration lies ahead of you. Therefore, we ran an extensive survey throughout 2020 and 2021 with more than 1,000 participants from all walks of agility: the Agile Metrics Survey 2021.
With the Agile Metrics Survey 2021, we present the first results and conclude with some thoughts about possible application scenarios of our instrument as well as possible next steps in our research.
🗞 Shall I notify you about articles like this one? Awesome! You can sign up here for the ‘Food for Agile Thought’ newsletter and join 33,000-plus subscribers.
🎓 Join Stefan in one of his upcoming Professional Scrum training classes!
The Intention Behind the Agile Metric Survey 2021
There are already well-established tools to measure agility at the team level or agile procedures in DevOps. But up to now, a tool is missing that allows measuring overall organizational agility, just as any systematic investigation of agile practices in a broader set of organizations.
Therefore, based on our previous research, comprising interviews and group discussions with practitioners from various disciplines, we decided to take the use of agile metrics as a proxy for overall organizational agility, and we aimed to:
- Develop a tool to measure the use of agile metrics in a broad set of organizations
- Investigate how organizations use specific agile metrics with different characteristics, such as size, industry sector, and agility level.
To reach these goals, we employed a five-step procedure where we:
- Constructed a survey that contained both questions regarding the characteristics of an organization, such as size and sector, and a set of indicators for the use of different agile metrics
- Collected data and evaluated if the instrument worked as expected (in terms of validity and reliability)
- Refined the instrument
- Collected even more data with the refined instrument
- Identified patterns of metric use in organizations with different characteristics.
In this report, we explain each of these subsequent steps in more detail. Additionally, we present the first results and conclude with some thoughts about possible application scenarios of our instrument as well as possible next steps in our research.
Please note: The ZIP-file of the Agile Metrics Survey also contains two HTML-files with high-resolution graphics of the plots and the original survey data.
Preliminary Metric Ensemble and Metric Indicators
Usually, constructing a valid and reliable measurement tool takes a lot of effort and expertise in the respective field of research. Luckily, we had a lot of prior experience from previous interviews and group discussions with agile experts and practitioners. Based on their perspectives, we came up with a list of eight metrics that appeared to be relevant in the context of organizational agility, such as Performance Metrics, Quality Metrics, Engineering Metrics, DevOps Metrics, Team Metrics, Organizational Agility Metrics, Product and Business Metrics, and Flow Metrics.
Due to both different organizational practices and different individual experiences of our prospective participants, we couldn’t be sure that these metrics mean the same for each of them. Therefore, we defined a set of indicators for each of these metrics. For example, Product & Business Metrics initially comprised indicators such as Realized business value, Return on Investment, Stakeholder Net Promoter Score, Customer Net Promoter Score, Number of experiments, Cost per release, Cost of Delay, Cost per story point, Program predictability, and Forecast accuracy.
We constructed a survey where the participants were to rate on a Likert-type scale (Never, Rarely, Sometimes, most of the time, always) whether these indicators are relevant in their day-to-day organizational practices. Additionally, we asked them to provide detailed information about several characteristics of their organization, such as size, sector, team structure, and perceived organizational agility level.
Data Collection
We collected the data in two subsequent phases. In the 1st phase, from 2019-11-17 to 2020-02-13, we collected 534 data sets. In the second phase, from 2020-07-22 to 21-01-30, we collected another 629 data sets. Overall, 1.153 people participated in our survey.<7p>
The data was collected with an online questionnaire administered via Google Forms. The main acquisition channel was the Food for Agile Thought newsletter. Additionally, we distributed the survey link via blog posts and various social media postings. At the beginning of the survey, there was a standardized instruction where we provided further information on data usage and anonymity. If the participants proceeded with the survey, they agreed to include their data in the analysis. There were no incentives for participating in our survey.
First Results
After refining our measurement tool, we took a first look into the dataset to explore the metric use of our participants` organizations. As a result, we found several exciting data patterns we would like to share and discuss in the following.
First of all, we found differences in the organizations` overall metric use. Figure 5 (“Overall metric use”) illustrates these differences on a scale from 1 (the metric is never used) to 5 (the metric is always used). In our sample performance metrics, Defect metrics and Flow metrics were most commonly used by the organizations, followed by Prediction metrics, Time metrics, and Team metrics. A third group, containing relatively less relevant metrics, consists of Engineering, DevOps, Organizational Agility Metrics, NPS, and Cost metrics.
Figure 1: Overall Metric Use
When contrasting agile and non-agile organizations, significant differences in metric use become transparent. (See Fig. 6 “Metric use by Agility.”)
Figure 2: Metric Use by Agility Level
We consider these differences an indicator that further investigation and a more detailed exploration of size and sector-specific metric profiles may be of interest. As you can see in Figure 7 (“Metric Use by Size”), some metrics become more relevant with the increasing size of an organization, such as Defect Metrics, Engineering Metrics, and Performance Metrics.
Figure 3: Metric Use by Organizational Size
Additionally, there seem to be sector specific-metric profiles, as Figure 8 (“Metric Use in different Sectors”) illustrates.
Figure 4: Metric Use by Industry Sector
Taking the sector-specific metric profiles and the differences between agile and non-agile organizations in the overall sample into account, we decided to take a closer look at the largest subsample, software, and internet companies (n = 188) and to investigate if our approach allows for a more fine-grained analysis of agile transitions. Figure 9 (“Metric Use by Agility”) illustrates the metric use of software and internet companies in our sample, depending on the reported agility level.
Figure 5: Metric Use by Agility Level (Software and IT Sector)
As you can see, there is a clear difference between organizations that were rated as rather non-agile (light blue) and those that were rated relatively agile (black) by the participants. In addition, at least for some metrics, separate “stages” or “phases” of the organizations` agile transition con be identified (grey).
Conclusion
With this investigation, we aimed to develop a valid and reliable instrument to measure agile metrics in different organizations. Based on the data from more than 1,100 participants, we refined our initial instrument and were able to reveal several exciting patterns in agile metric use.
Of course, the results presented above only allow for limited insights and detailed analyses, focusing on sector-specific metric profiles need to follow, as the question “what IS an agile organization” apparently needs a sector and maybe even size specific answer.
Further investigations may lead to sector-specific agile benchmarks that can guide organizations` agile transitions. In addition, as we can measure metric use validly and reliably, relationships between metric use and relevant business outcomes can be explored.
One major limitation of our approach is linked to the data collection procedure. As we wanted to measure the use of agile metrics in a broad set of organizations, we, in almost all cases, only surveyed one participant per organization. To allow for sector-specific and more detailed analyses, it would be fascinating to work together with organizations and assess whether all employees share a common perspective or significant differences in perceived agility and metric use.
Do you recognize your organization in the results of the Agile Metrics Survey 2021? If so, please share your learnings with us in the comments.
✋ Do Not Miss Out and Learn more about the Agile Metrics Survey 2021: Join the 10,000-plus Strong ‘Hands-on Agile’ Slack Team
I invite you to join the “Hands-on Agile” Slack team and enjoy the benefits of a fast-growing, vibrant community of agile practitioners from around the world.
If you like to join now all you have to do now is provide your credentials via this Google form, and I will sign you up. By the way, it’s free.
Agile Metrics Survey 2021 — Related Posts
Agile Metrics — The Good, the Bad, and the Ugly
Faking Agile Metrics or Cooking the Agile Books
Spark a cargo cult discussion with the ‘State Of Agile’ checklist for your organization.