In-Depth: The Evidence-Based Business Case For Agile

What is the business case for Agile teams? Is it really that important that they are autonomous, able to release frequently, interact closely with stakeholders and spend all that time on continuous improvement? Or is all that just hype? Or maybe even counterproductive?

Yes, we believe all these things are important. And so does probably everyone else in our community. But what is the evidence we have for this? What proof do we have that we aren’t just selling a modern equivalent of snake oil? We think we do well to base our beliefs about Agile more on evidence. This allows us to test our beliefs and also makes our case stronger in case they are supported.

This post is our attempt to bring an evidence-based perspective to the business case of Agile teams. We will share results from scientific studies that we located through Google Scholar. And we share results from our own analyses based on actual data from stakeholders and Scrum teams that we collected through the Scrum Team Survey.

This post is part of our “in-depth” series. Each post discusses scientific research that is relevant to our work with Scrum and Agile teams. We hope to contribute to more evidence-based conversations in our community and a stronger reliance on robust research over personal opinions.

A working definition of Agile and Stakeholders

Before we begin, we need to define some of the terms we will use throughout this post. The first term is Stakeholder. Throughout this post, we define them as “all users, customers, and other people or groups who have a clear stake in the outcomes of what this team produces, and invest money, time or both in making sure that happens”. So this excludes people who have an opinion about the product but don’t stand to lose anything when the team fails.

The second term I’d like to define is Agile or agility. What do we actually mean when we talk about an “Agile team”? In its broadest sense, we mean that these teams practice the principles of the Agile Manifesto. But this is still rather vague. We could say that these teams practice Agile methodologies, like Scrum or XP, and are therefore Agile. But adherence to a framework or prescribed process does not guarantee agility. In fact, we’ve written the Zombie Scrum Survival Guide specifically to shine a light onto all those teams who think they have checked all the boxes of the Scrum framework, and are still not moving.

“Adherence to a framework or prescribed process does not guarantee agility.”

I prefer a process-based definition of agility. This definition answers the question: “What kind of processes typically happen in Agile teams that distinguish them from non-Agile teams?”. We worked with Prof. Daniel Russo of the University of Aalborg to answer this question with data from almost 2.000 Scrum teams. In a scientific study, we identified five core processes — or factors— that happen in and around Agile teams at varying levels of quality:

1. Teams work to be as responsive as possible through automation, refinement, and a high(er) release frequency.
2. Teams show concern for the needs of their stakeholders by collaborating with them closely and making sure that what is valuable to them is represented in their work and goals.
3. Teams engage in continuous improvement to improve where they can through frequent reflection, shared learning, and creating a psychologically safe environment.
4. Teams expand and use their autonomy to manage their work and bring their skills together in the most effective ways.
5. In the immediate environment of Scrum teams, management supports what teams do and helps them where possible.

Although we used Scrum teams for our investigation, these processes are generic enough to apply to Agile teams in general. We collected the data through our Scrum Team Survey. This tool uses a validated and scale-based questionnaire to allow teams to diagnose and improve their process in an evidence-based way. You can use the free version for individual teams.

For this post, we used a sample of 1.963 Scrum teams and 5.273 team members. The sample was cleaned of fake and careless responses and corrected for social desirability where relevant. Our sample includes teams from all sectors, sizes, and regions.

Finding #1: Teams Vary Widely In Their Actual Agility

In our sample of 1.963 Scrum teams, we observed a wide range of scores on the five core processes of agility. We calculated an overall agility score for teams and categorized them into three buckets — low, moderate, and high — for easier interpretation. The histogram below shows the differences visually:

There are two caveats to this histogram. First, the cutoffs for low, moderate, and high agility are fairly arbitrary. In reality, agility is obviously a continuum that teams traverse. Second, there is some circularity because we calculate the overall agility of a team as an aggregate of the five core processes. So it is not surprising that the scores improve as teams become more Agile.

However, the histogram illustrates nicely the size of this increase, which is both significant for all differences (p<.001) and highly substantial. It also illustrates well how all processes improve, and not just one or two. They seem to be connected. The results also affirm that calling yourself a Scrum team — as the teams in this sample mostly did — is no guarantee of actual agility. At the same time, we can clearly see that many Scrum teams are Agile.

“It is clear from these results that identifying yourself as an Agile or Scrum team is no guarantee of actual agility.”

So we know now have a better sense of the differences between the quality of the core processes at different “levels” of agility. However, the key question is whether this actually makes a difference in important business outcomes. Are more Agile teams indeed better at producing important business outcomes than less Agile teams?

“Are more Agile teams indeed better at producing important business outcomes than fewer Agile teams?”

Finding #2: Agile Teams Have More Satisfied Stakeholders

Ultimately, we believe that a strong business case lies with how effectively Agile teams can serve the needs of stakeholders, like users, customers, and other people with a clear stake. There is clearly a strong economic incentive for organizations to keep stakeholders happy, as they are the people who pay for, or use, their products.

But do Agile teams have more satisfied stakeholders than less Agile teams? Fortunately, the Scrum Team Survey allows teams to ask their stakeholders to evaluate their outcomes. We ask stakeholders to evaluate this on four dimensions: quality, responsiveness, release frequency, and team value. For this analysis, we use the evaluations of 857 stakeholders for 241 teams. 49% of these represented users, 28% customers, and 22% internal stakeholders.

We can statistically test the hypothesis that teams are increasingly able to satisfy their stakeholders as their Agility increases. For this, we used a simple statistical technique called multiple regression analysis. We entered the overall stakeholder satisfaction as the predicted value, and the five core processes as predictors. The regression model was significant (p <.001) and explained 29.2% of the observed variance in stakeholder satisfaction. This may not seem that high, but values above 20% are considered “very strong” in the social sciences. The scatterplot shown below shows the distribution of teams based on stakeholder satisfaction (vertical) and Agility (horizontal). Each dot represents a team:

A scatterplot and the results from a regression analysis may not be intuitive for many readers, especially those not familiar with statistics. So we also created a histogram by plotting the least Agile teams against the most Agile teams. It is a bit simplistic, but it illustrates the differences more dramatically.

There is one caveat to these results. The scatterplot suggests that teams are more likely to invite stakeholders when they are already quite Agile. This makes sense; non-Agile teams may interact less with their stakeholders and thus see fewer opportunities to invite them to evaluate the outcomes. But this means we lack data from teams that score very low on agility. However, it is likely that the difference would be even stronger if such stakeholders were included.

The bottom line is clear though. Agile teams have more satisfied stakeholders than less Agile teams. This effect is also very strong. So high autonomy, high responsiveness, high continuous improvement, high stakeholder concern, and high management support clearly go hand-in-hand with higher stakeholder satisfaction. Simply put; if organizations want more satisfied stakeholders, investing in the five processes of agility is a very clear evidence-based recommendation.

“The bottom line of the results is clear though. Agile teams have more satisfied stakeholders than non-Agile teams.”

Finding #3: Agile Teams Have Higher Morale

The second part of our business case can be made by looking at how agility affects team members. Employees are the “human capital” of modern-day organizations. Happy employees allow companies to save money on absenteeism, sick leave, and the hiring and onboarding of new employees to replace others who leave the company.

So it is helpful to look at the morale of teams. Team morale, or ‘esprit de corps’, reflects how motivating and purposeful the work feels to a team (Manning, 1991). Many meta-analyses — statistical aggregations of datasets from many other empirical studies — have shown strong benefits of high morale and its corollaries, like job satisfaction. It reduces turnover and absenteeism among employees (Hacket & Guion, 1989). It also increases performance (Judge et. al., 2001) and proactive behavior (LePine, Erez & Johnson, 2002).

“So there is a clear economic incentive for companies to encourage high morale in teams.”

But is morale higher in Agile teams than in less-Agile teams? To answer this question, we analyzed data from 1.976 teams and 5.273 team members from the Scrum Team Survey.

So with this data, we ran another linear multiple regression analysis with team morale as the predicted value, and the five core factors as predictors. The resulting model was significant (p <.001) and explained 44.6% of the observed variance in team morale. This is a strong result when we consider that values above 20% are already considered “very strong” in the social sciences. The scatterplot is shown below. Each dot represents a team. The line reflects the optimal regression line. It is clearly visible that as the agility of a team increases (vertical), team morale also increases (horizontal).

We also created a histogram to plot the morale of the most Agile teams against that of the least Agile teams in our database, which is more visually clear:

So what do these numbers mean in practice? Generally speaking, morale will be much higher in Agile teams compared to less Agile teams. The morale in the highest-scoring teams (on agility) is almost twice as high as in the lowest-scoring teams. So high autonomy, high responsiveness, high continuous improvement, high stakeholder concern, and high management support clearly go hand-in-hand with high team morale.

“Generally speaking, morale will be much higher in Agile teams compared to non-Agile teams.”

Intermission: A Note On Causality

Careful readers may have wondered: “But wait! Correlation doesn’t imply causation”. The observable fact that team morale and stakeholder satisfaction are higher in Agile teams does not necessarily mean that agility is the cause. And this is true. It is possible that the effect is actually the reverse; high morale or high stakeholder satisfaction drives teams to become more Agile. Or both bias teams to evaluate their processes in a more positive light than when morale or stakeholder satisfaction is lower. It is also possible that other variables explain both results. For example, an organization with a very flat hierarchical structure may both enable high agility as well as high morale or high stakeholder satisfaction.

Unfortunately, causality is very hard to establish with certainty. This requires highly controlled experiments where only the level of agility is manipulated in a team. But how can one feasibly do that? Another approach is to track many organizations over time as they adopt Agile methodologies, while also measuring anything else that could influence their results other than agility itself. It would be wonderful to have such data! However, the sheer effort involved is so gargantuan that this is close to impossible. Fortunately, we don’t need to put the bar that high. While correlation doesn’t imply causation, it can certainly make a case for it. Especially when we have strong corroborating evidence from other sources. Our data clearly shows that agility is associated with team morale and stakeholder satisfaction. So if you measure that one is going up, the others will probably go up too. Thus, it is probably easier for organizations to create environments for teams that encourage their Agility (e.g. high autonomy, responsiveness, etc) than it is to directly change the morale of team members or even the satisfaction of stakeholders.

But let's take a look at potential corroborating evidence. What do scientific studies have to say?

Finding #4: Scientific Studies Corroborate That Agility Generates Better Business Outcomes

We defined agility in terms of five processes. Agile teams are responsive, know who their stakeholders are and what they need, have high autonomy, and improve continuously. They are also supported by management. What do scientific studies have to say about the business outcomes of Agile methodologies, as well as the factors we just mentioned? So we went to Google Scholar and searched for review articles.

Cardozo et. al. (2010) reviewed 28 scientific studies that investigated how Scrum is associated with overall business outcomes. A strength of such a review is that it allows for the identification of patterns across many studies. The authors identified five core outcomes of Scrum: 1) higher productivity in teams, 2) higher customer satisfaction, 3) higher quality, 4) increased motivation in teams and 5) a general reduction in costs. While this study covers even more business outcomes, the second and fourth points match our findings.

A critical part of Agile methods is that new iterations of a product are released more frequently than in more traditional approaches. Several empirical studies have indeed found that teams with a frequent delivery strategy are more likely to deliver successful project outcomes and satisfy stakeholders than teams that do not (Chow & Cao, 2008, Jørgensen, 2016). This also matches our findings for stakeholder satisfaction.

But a high delivery strategy is of little value if what goes out to stakeholders doesn’t match their needs, or doesn’t reach the right people. So teams need to develop a good understanding of who their stakeholders are and what they need. Van Kelle et. al. (2015) collected data from 141 team members, Scrum Masters, and Product Owners from 40 projects. They found that the ability of teams to develop a shared sense of value contributed significantly and strongly to project success. They also found that the overall agility of teams increased the chance of project success. Hoda, Stuart & Marshall (2011) interviewed 30 Agile practitioners over a period of 3 years. They found that teams that collaborate closely with customers are more successful. However, they also observed that customers are rarely as involved as would be expected from Agile methodologies. This supports our finding that stakeholder satisfaction goes up as teams become more involved with their stakeholders (stakeholder concern).

“The ability of teams to develop a shared sense of value contributed significantly and strongly to project success.”

While high responsiveness and high concern for the needs of stakeholders are arguably the two pillars that distinguish Agile teams from non-Agile teams, these pillars need a good foundation. This is where team autonomy and continuous improvement come into play. Both are important characteristics of Agile teams because it allows them to remain effective in the face of complex, unpredictable work. Many studies have identified high autonomy as an important prerequisite for Agile teams (Moe, Dingsøyr & Dybå, 2010; Donmez & Gudela, 2013; Tripp & Armstrong, 2018; Melo et. al. 2013). Lee & Xia (2010) surveyed 505 Agile projects and found that high autonomy allowed teams to respond more quickly to challenges. They also note that this is particularly relevant to complex challenges, which is typically the case for the product innovation that happens in Agile teams. As for continuous improvement, this is generally more a climate in teams than a process. It is marked by high safety, shared learning, high-quality Sprint Retrospectives, and ambitious quality standards. Hoda & Noble (2017) identified learning processes as essential for teams to become Agile. So while autonomy and continuous improvement may not directly result in business outcomes, they need to be present in order for Agile teams to generate outcomes that satisfy stakeholders, and through a process that is also satisfying to team members.

“While autonomy and continuous improvement may not directly result in business outcomes, they need to be present in order for Agile teams to generate outcomes that satisfy stakeholders, and through a process that is also satisfying to team members.”

Finally, this brings us to the role that management plays. Because their support is consistently identified by scientists as the most critical success factor to make all the above possible (Van Waardenburg & Van Vliet, 2013; de Souza Bemerjo et. al., 2014; Young & Jordan, 2008; Russo, 2021). The shift that management has to go through is described by Manz et. al. (1987) as “leading others to lead (themselves)”. The authority to make work-related decisions shifts from external managers to the teams themselves. So rather than taking the lead, managers have to take a more supporting role and ask teams where they need their help. Second, management has to understand the point of Agile methodologies like Scrum and support teams by removing any obstacles they experience.

Click to enlarge

There are many other factors that can be considered. But the five processes we identified as characteristics of Agile teams provide a good, evidence-based starting point to make Agile teams more effective, and generate better business outcomes. Together with Prof. Daniel Russo We investigated a sample of 4.940 team members, aggregated into 1.978 Scrum teams, and found that these processes — or factors — together explain a very large amount of how effective Scrum teams (75.6%) are, and 34.9% of team morale and 57.9% of stakeholder satisfaction (Verwijs & Russo, 2022).

What About Other Business Outcomes?

Obviously, team morale and stakeholder satisfaction are just two business outcomes for which economic arguments can be made. But more outcomes can be considered, like business longevity, the ability to deliver value within budget, and so on. We may cover these in future posts, as this one has already become way too long. And if happier team members and happier stakeholders aren’t already convincing enough, good results on those other outcomes aren’t probably going to change minds either :).

Implications For Practitioners

So what does all this mean in practice?

1. An important lesson we’ve had to learn is that you can’t sell Agile or Scrum based on frameworks or ideals. Ultimately, the role of (top) management is to keep their business healthy and economically sustainable. Thus, the best way to convince them is to show how Scrum or Agile helps in that regard. This business case should provide some good talking points.
2. Generally speaking, it is a good idea to track important business outcomes as metrics. For example; the number of sales, the number of unhappy customers, absenteeism, etc. This provides you with an opportunity to take an evidence-based approach to improvements. You can also measure such metrics alongside an ongoing Agile adoption, and see how Agile is improving certain metrics (or not).
3. You can use the Scrum Team Survey to diagnose your Scrum or Agile team for free. We also give you tons of evidence-based feedback. The DIY Workshop: Diagnose Your Scrum Teams With The Scrum Team Survey is a great starting point.
4. Our Do-It-Yourself Workshops are a great way to start improving without the need for external facilitators. The DIY Workshop: Discover The Needs Of Your Stakeholders With UX Fishbowl or Experiment: Deepen Your Understanding Of Scrum With Real-Life Cases are great starting points. The DIY Workshop: Build Understanding Between Scrum Teams And Management is also very useful if management support is the biggest issue. Find many more here.
5. We offer a number of physical kits that are designed to start conversations with and between teams and the larger organization. We have the Scrum Team Starter Kit, the Unleash Scrum In Your Organization Kit, and the Zombie Scrum First Aid Kit. Each comes with creative exercises that we developed in our work with Scrum and Agile teams.

Conclusion

Our goal with this post was to make a stronger, evidence-based business case for Agile and Scrum. Taken together, we have strong evidence to support the belief that Agile teams deliver more value to their stakeholders than less Agile teams. They also have much higher team morale. Any company worth its salt would see the economic value in both of these outcomes. Happy stakeholders generate more revenue, spread positive word-of-mouth, and are more likely to stay. Similarly, happy team members are more productive and more likely to stay for the long term. If companies seriously invest in Agile, and in particular in the five factors we covered in this post, they are very likely to see better results.

Is this an indisputable fact? No. But the evidence to support it is very strong indeed, and growing every day.

This post took over 46 hours to research and write. Find more evidence-based posts here. We thank all the authors of the referenced papers and studies for their work.

References

Cardozo, E. S., Araújo Neto, J. B. F., Barza, A., França, A. C. C., & da Silva, F. Q. (2010, April). SCRUM and productivity in software projects: a systematic literature review. In 14th International Conference on Evaluation and Assessment in Software Engineering (EASE) (pp. 1–4).

Chow, T., & Cao, D. B. (2008). A survey study of critical success factors in agile software projects. Journal of systems and software, 81(6), 961–971.

Dönmez, D., & Grote, G. (2013, June). The practice of not knowing for sure: How agile teams manage uncertainties. In International Conference on Agile Software Development (pp. 61–75). Springer, Berlin, Heidelberg.

Jørgensen, M. (2016). A survey on the characteristics of projects with success in delivering client benefits. Information and Software Technology, 78, 83–94.

LePine, J. A., Erez, A., & Johnson, D. E. (2002). The nature and dimensionality of organizational citizenship behavior: a critical review and meta-analysis. Journal of applied psychology, 87(1), 52.

Lee, G., & Xia, W. (2010). Toward agile: an integrated analysis of quantitative and qualitative field data on software development agility. MIS quarterly, 34(1), 87–114.

Melo, C. D. O., Cruzes, D. S., Kon, F., & Conradi, R. (2013). Interpretative case studies on agile team productivity and management. Information and Software Technology, 55(2), 412–427.

Moe, N. B., Dingsøyr, T., & Dybå, T. (2010). A teamwork model for understanding an agile team: A case study of a Scrum project. Information and software technology, 52(5), 480–491.

Hacket, R. D. (1989). Work attitudes and employee absenteeism: A synthesis of the literature. Journal of occupational psychology, 62(3), 235–248.

Hoda, R., & Noble, J. (2017, May). Becoming agile: a grounded theory of agile transitions in practice. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) (pp. 141–151). IEEE.

Hoda, R., Noble, J., & Marshall, S. (2011). The impact of inadequate customer collaboration on self-organizing Agile teams. Information and software technology, 53(5), 521–534.

Judge, T. A., Thoresen, C. J., Bono, J. E., & Patton, G. K. (2001). The job satisfaction–job performance relationship: A qualitative and quantitative review. Psychological bulletin, 127(3), 376.

Tripp, J., & Armstrong, D. J. (2018). Agile methodologies: organizational adoption motives, tailoring, and performance. Journal of Computer Information Systems, 58(2), 170–179.