What Makes Agile Transformations Successful? Results From A Scientific Study

This post is part of our “in-depth” series. Each post discusses scientific research that is relevant to our work with Scrum and Agile teams. With this series, we hope to contribute to more evidence-based conversations in our community and a stronger reliance on robust research over (only) personal opinions.

What makes Agile transformations successful? Is management support as critical as we think? Are good Scrum Masters and Product Owners what makes the difference? And what about the socio-technical skills of Developers? While most practitioners in our field have a strong gut feeling about these questions, there is surprisingly little systematic scientific research to fall back on. 

In this post, I summarize the findings from an academic peer-reviewed publication by Daniel Russo (2021), a professor at the University of Aalborg. The paper presents and tests a model that identifies success factors for Agile transformations. This post is a good read if you’re involved in Agile transformations and want to learn what you should probably invest most efforts into. I summarize the core findings at the end.

What do we already know?

Good research always builds on earlier research. So what does scientific research already tell us about Agile transformations? Kim Dikert and his colleagues reviewed 52 prior scientific studies in 2016 to identify common challenges and success factors for Agile transformations. Although they presented a comprehensive list, they also recognized that existing research is mostly based on individual cases and what practitioners report in interviews. Case-based research is hardly objective, and Dikert and his colleagues urge for more objective data in future studies.

So why is that important? While there is great value in the experience reports of practitioners, they are also highly subjective. Personal opinions, preferences, and even personal interests may bias those experiences. For example, most practitioners are probably excited about Agile and emphasize those cases where it worked and ignore those where it didn’t. It is also very likely that individual practitioners participated in Agile transformations in only a handful of organizations. Whatever success factors and challenges they’ve seen there are not automatically true for any organization. This is why scientific research favors more objective data. The most objective data comes from experiments where the variables we’re interested in are changed (e.g. management support, technical skills) while everything else is kept equal. But since that kind of research is very hard to execute, and not always ethical, an alternative is to use cross-sectional data instead. Here, the experiences from a large number of practitioners from a large number of domains are combined to look for stable patterns across their experiences. This reveals what seems to be true in the majority of cases.

This is essentially what Russo did in his study. He used a case study of a complex and large-scale Agile transformation to develop (or “induce”) an initial theoretical model and then tested (or “deduced”) it with a larger sample of practitioners and advanced statistical techniques. I will explain the two phases of the study in more detail below.

Phase 1: Field Study To Develop A Theory for Successful Agile Transformation

While Dikert and his colleagues (2016) identified many potential success factors and challenges for Agile transformations from existing research, they did not offer a clear theoretical model to describe how those variables interacted. So Russo started with a field study to develop such a model. He observed a successful Agile transformation at a complex and highly regulated military agency that took place between 2014 and 2016. The transformation focused on the development of a new Command & Control platform that allows Military staff to coordinate a wide range of military resources to achieve missions. Such a platform obviously needs to be highly secure, reliable, and stable. By switching from a plan-based approach to an Agile approach, the agency was able to realize significant cost savings (40-50% per “line of code”), higher productivity, and increased code quality (substantial decrease in defects, r = -.88) compared to previous plan-based projects.

To investigate success factors and challenges, the researchers interviewed program managers, Scrum Masters, Product Owners, Developers, and other stakeholders, analyzed documents, and observed teams and management as they did their work. These observations were then coded and clustered into concepts, themes, and high-level variables. For example, interviewees mentioned the need to be able to rely on each other (“I rely on work by my colleagues”) and the inclusion of different viewpoints during meetings as contributors to successful transformations. Such observations were grouped into a variable called “Trust”. Taken together, the following core Agile Success Model emerged from the field study:

The Agile success model proposes that:

• “Success” can be defined as the delivery of desired features, high-quality results, and the ability of teams to respond to needs as they emerge.
• There are broadly three groups that influence successful Agile transformations; Developers, Product Owners & Scrum Masters, and top managers. The paper describes these as “groups of stakeholders”, but I will avoid that terminology to avoid confusion with what we normally understand as “stakeholders” in Scrum teams (e.g. customers and users).
• Developers ultimately implement the requirements needed to successfully deliver a product, so the influence of top management, Scrum Masters, and Product Owners works primarily through them. 
• Starting with developers, their technical and social skills influence the success of Agile development directly.
• In turn, the leadership of Scrum Masters positively influences the work done by developers. Leadership was defined here as creating an environment where developers know what to work on, are involved in decision-making, and bringing a sense of order to the work. I will discuss a slight limitation to this definition further down.
• Product Owners also positively influence the work done by developers when they are actively involved in the work and frequently interact with the developers to clarify the work.
• Finally, top management influences success in three complementary ways. Their support makes it easier for Scrum Masters and Product Owners to perform their roles well, and in turn, support the developers. Top management also directly influences project success by directly committing themselves to the success of the project.

I‘m quite sure that this model appeals to the common sense and experience of most practitioners. Still, the model is based on a single case. This means that it may be biased to one organization and not be generalizable to other kinds of organizations. For example, the military nature of the organization means that there is a strong chain of command where people generally only interact with those one level above or below them. This is certainly not always the case and may bias the findings.

I mention this because it highlights a limitation of most academic research into Agile transformations, which are also based on one or a few cases. Regardless of how insightful these cases are, you can’t generalize their findings to other organizations without a more rigorous test. We often fall into the same trap as practitioners when we generalize our experiences from a few cases to every organization.

Fortunately, we can use statistical techniques to test if a model can indeed be generalized across organizations. This was the focus of Phase 2.

Phase 2: Testing The Model With A Large Sample

The ultimate test of any model is to test it with every Scrum team and every organization. Since this is not practically feasible, scientists use advanced statistical techniques to draw conclusions about the population from a smaller sample of data from that population. Two things are important here. The first is that the sample must be big enough to reliably distinguish effects from the noise that always exists in data. The second is that the sample must be representative enough of the larger population in order to generalize findings to it. 

It is easy to understand why. Suppose that you’re tasked with testing the purity of the water in a lake. You can’t feasibly check every drop of water for contaminants. But you can sample some of the water and test it. This sample has to be big enough to detect contaminants and small enough to remain feasible. It's also possible that contaminants are not equally distributed across the lake. So it's a good idea to sample and test a bucket of water at various spots from the lake. This is effectively what happens here.

For this study, a so-called power analysis was used to determine that the model needed to be tested with 166 participants from different organizations to reliably measure effects (at a p-value of 0.05). So a 30-question survey instrument was developed to measure the 6 core variables in the model. This survey was sent to 201 participants that were invited to fill it in for their team or organization. After removing entries of low quality, the final sample consisted of 190 participants. 

The data was then analyzed with a statistical technique called “Structural Equation Modelling” (PLS-SEM to be precise). This allows researchers to test entire models and all proposed effects in a single test.

The full model with all the effects is shown below. The strength of the paths can statistically range between (roughly) -1 and 1. Of interest are also the values for R2, which capture how much of that variable was explained by the other variables in the model (up to a maximum of 1.0, or 100%). Values above 0.22 are generally considered “strong” in the social sciences. Why not higher? Although higher is better, a higher R2 usually requires more complex models with more variables. Science always prefers simpler models over more complex ones. This pursuit for “parsimony” — as it is called — is important because simpler models are also easier to apply in practice. If we can predict 25% of a variable from 3 other variables, this gives us more direction on what to change than with a model that predicts 30% from 8 variables.

Finding #1: The Socio-Technical Skills of Developers Are Critical To Success

A clear finding is that Developers more strongly contribute to project success than the role of Management, Product Owners, and Scrum Masters. The researchers established this with an Importance-Performance Map Analysis (IPMA) that ranks variables in terms of their effect on an outcome variable, project success in this case. To put this in perspective, for each increase of Developer Skills by one point, project success tends to increase by .358 points. For Product Owners and Scrum Masters, this is only .054 and .087 points, respectively. Top Management influences project success more strongly (.287) than Scrum Masters and Product Owners, but less so than Developers.

This study suggests that Developers are in fact the most important contributors to successful Agile transformations. This also connects well to our other study with 1.159 Scrum teams (Verwijs & Russo, 2021). With that in mind, it makes sense to direct most energy towards the development of their skills. This study also points out that these involve both technical skills and social skills. “Social skills” involve helping behavior, collaboration on shared tasks, and a sense of team spirit, whereas “Technical skills” focus on actual technical knowledge, learning of new technical skills, and technical proficiency. Teams that score high in this area are more likely to deliver successful projects.

Practical implications

Agile transformations should specifically address how Developers will be trained and supported during and after the transition. This study shows that recruiting and training good Scrum Masters and Product Owners is just a small part of that support. If organizations are serious about project success, they need to invest in the training and development of the skills that developers need.

If you don’t know where to start, we created 14 do-it-yourself workshops to create an environment in and around Scrum teams that encourage learning and the development of critical skills — both social and technical.

Finding #2: The Trickle-Down Effect Of Management Support (And Why Management Can’t Rely On It)

The researchers specifically investigated how support from Top Management influences the success of the transformation. In terms of the variables that were investigated in the Importance-Performance Map Analysis (IPMA), Top Management was the second-strongest influence (after skilled Developers).

The findings highlight a hierarchical trickle-down effect of top management support on the success of Agile transformations. This is a pattern that many practitioners will recognize. Strong support from top management makes it easier for Scrum Masters and Product Owners to perform their roles well and support their Developers more effectively. In turn, because Developers are better supported by Product Owners and Scrum Masters, project success increases strongly. But this trickle-down effect is only part of the story. The data also shows that top management has a strong direct effect on project success, outside of Scrum teams.

The study also helps us understand what this support looks like. Agile transformations are more successful when top management expresses a deep understanding of why it wants to work with Scrum and Agile and commits to the removal of impediments that make this difficult. Furthermore, they can increase success by acknowledging the role and autonomy of teams. This also matches the findings of another study (Verwijs & Russo, 2021).

“Agile transformations are more successful when top management expresses a deep understanding of why it wants to work with Scrum and Agile and commits to the removal of impediments that make this difficult.”

Practical implications

If top management wants to increase the success of Agile transformations, their role extends well beyond just hiring and training skilled Scrum teams. Their direct support is necessary to overcome impediments. One way to do this is by creating frequent opportunities for top management to meet with Scrum Teams and investigate returning impediments. Another is to create a rhythm by which the effectiveness of the transformation is periodically reviewed.

If you don’t know where to start, we created five do-it-yourself workshops to help management connect with Scrum Teams and discover where their support is needed.

Finding #3: The Impact Of Product Owners And Scrum Masters Is Limited

Of the variables that the researchers investigated, Scrum Masters and Product Owners had the smallest influence on transformation success. The effects are not nearly as large as the skills of Developers and support from management.

This may be a disappointing finding for Scrum Masters and Product Owners. But a deeper analysis of the data and the way the roles were defined in the study actually sheds a light on how not to perform the roles.

For example, the study defines Scrum Masters as leaders who clarify goals and roles, bring order to the work, and encourage participative decision-making. Product Owners are mostly defined in terms of their collaboration and co-creation with Developers. Although all of this is true in a sense, it also highlights specifically those responsibilities that are similar to what middle-management traditionally does. It also focuses mostly on their role towards teams, and not towards shaping the broader environment.

Although this study did not investigate other definitions of the roles, other research (e.g. Verwijs & Russo, 2021) shows that Product Owners strongly contribute to team effectiveness when they connect teams to stakeholders and focus conversations and decisions on what is valuable. Similarly, Scrum Masters can make more impact when they create a productive and psychologically safe environment where teams can expand their autonomy and learn continuously.

So the study shows that Scrum Masters and Product Owners certainly won’t contribute much when they (continue to) act as middle-managers. Or more broadly, that the typical responsibilities of middle-management don’t seem to contribute much to the success of Agile transformations. Perhaps Scrum Masters and Product Owners can contribute more when they fulfill their roles differently, as other research suggests. Either way, it remains true from the data that Scrum Masters and Product Owners only contribute a little to a game where Top Management and the Developers are really the star players.

“Scrum Masters and Product Owners only contribute a little to a game where top management and the teams are really the star players.”

Practical implications

As the other findings of this paper also show; skilled Developers are really at the center of what makes Agile transformations successful. The best that Scrum Masters and Product Owners can do is help teams develop the socio-technical skills they need.

If you don’t know where to start, we created 14 do-it-yourself workshops to create an environment in and around Scrum teams that encourage learning and the development of critical skills — both social and technical.

Finding #4: The Agile Success Model Applies Both To Mission-Critical And Non-Mission-Critical Products

The model presented in the paper was created from a case study that involved a mission-critical product. Since the development of mission-critical products comes with unique challenges, the model could be biased in subtle ways. 

So the researchers used a multi-group analysis to compare the effects in the model between mission-critical and non-mission-critical teams. No significant differences were founds, which means that the model applies to both domains equally well. So the core findings that I summarized above apply equally well to mission-critical products and non-mission-critical products.

Limitations

The study we report on in this post is certainly insightful. Like all studies, it also has limitations. The first is that the sample (190 participants) is statistically certainly large enough, but replications with larger datasets are certainly necessary to make the claims more robust. Furthermore, some of the definitions of the various measures (e.g. for Scrum Master Leadership and Product Owners) could be improved to take into account how the roles are also designed to shape the environments of Scrum teams. Finally, the study is ultimately based on correlational data. Better Developer Skills may simply go together with higher project success, but not cause it. However, the fact that “correlation does not imply causation” doesn’t mean that “correlation never shows causation”. This is where scientific models are so helpful. They both provide testable hypotheses as well as a narrative to explain what we observe. We use it until a better one comes along that is better able to explain what we observe with the smallest number of variables.

Conclusions

The Agile Success model emphasizes how important Developers and their socio-technical skills are to the success of Agile transformations. This is closely followed by the level of support from Top Management. Product Owners and Scrum Masters, on the other hand, contribute less to the overall success. Particularly when they fulfill their roles in ways that are reminiscent of more traditional middle-management.

The emphasis that this study places on the role that teams place isn’t always evident in practice. We reached a similar conclusion in another study. Organizations are often quick to send Product Owners and Scrum Masters to certification training. And while those roles certainly have some influence as this study shows, one can wonder if the same effort and investments are made when it comes to the socio-technical skills of Developers. Personally, I find this all the more surprising when we consider that Agile methodologies originated from Developers.

So if you want to make your Agile Transformations more successful, this study suggests that it is a great idea to invest at least in the socio-technical skills of Developers and support from Top Management. Good Scrum Masters and Product Owners can add to this by creating environments where teams have ample autonomy and can work closely with actual stakeholders.

More research into what makes Agile Transformations successful is certainly necessary. It is helpful to test this model with more Scrum teams and teams from other environments to see if it holds up to replication efforts. We often like to say that “it depends” in our profession to emphasize that every team and organization is different. At the same time, this kind of research allows us to see patterns across many teams and organizations that help us identify some of the factors it may, in fact, be depending upon.

If you are interested to read the full academic paper, you can download it here for free: Daniel Russo. 2021. The Agile Success Model: A Mixed-methods Study of a Large-scale Agile Transformation. ACM Trans. Softw. Eng. Methodol. 30, 4, Article 52 (July 2021), 46 pages. You can also reach Prof. Russo directly through his website or you can follow him on Medium at Daniel Russo, Ph.D.