Are Scrum Masters Sufficiently Focused On Valuable Outcomes?

We recently discussed a case on our community Discord that concerned a team without proper Sprint Reviews, stakeholders, and increments. The first intuition of many Scrum Masters — including myself — was to host a Sprint Retrospective. But this leaves the reflection and the initiative for change entirely with the team. But why wasn’t the strategy to show our teams how to do a proper Sprint Review, and to get as many stakeholders into the room as possible, and then reflect on what that was or wasn’t like together?

I have a hunch that this single observation alludes to a more persistent pattern across Scrum Masters. Namely, that Scrum Masters are often more focused on the quality of the process than on the value of the outcomes.

In this post, I investigate if this hunch has some basis in facts. So I entered “Scrum Master” into Google Scholar and read the few scientific studies that investigate Scrum Master activities. I also ran analyses on the data we’re collecting with the Scrum Team Survey and another dataset that was offered to me by Rishi D Markenday and Dave West from Scrum.org.

This post is part of our “in-depth” series. Each post discusses scientific research that is relevant to our work with Scrum and Agile teams. We hope this contributes to more evidence-based conversations in our community. Writing a post like this takes a lot more time than a regular post. If you appreciate it and would like to see more, we would be very grateful for your support on Patreon.

Finding #1: Scrum Masters spend most of their time on facilitation and mentoring/coaching

So what do Scrum Masters do most of their time? A recent study by Shastri, Shastri, Hoda & Amor (2021) may shed a light on this question. The authors systematically interviewed 47 agile practitioners about the typical activities of Scrum Masters. The sample included Scrum Masters and people who frequently interact with them, like Product Owners, managers, and even a few developers. The authors identified six focus areas and calculated how many of the Scrum Masters were involved in them (N=26):

1. Facilitation of the various events and removal of impediments (69%)
2. Mentoring and coaching of Scrum teams in their path towards self-organization (69%)
3. Adapting the process to suit the context in which a team operates and selecting practices to apply with this team (31%).
4. Protecting the team against interference from management (27%).
5. Negotiating with customers and Product Owners about their expectations and the capacity of teams to meet those expectations (27%).
6. Coordinating with other teams and individuals about the work that is being done (12%).

Findings from Shastri, Hoda & Amor (2021) as to what percentage of the time of typical Scrum Masters is spent on these everyday activities

They also compared the benefits of having a Scrum Master present in a team versus not having one (N=47). Teams with Scrum Masters were more likely to have a Definition of Done, a Sprint Planning, measurement of velocity, and shorter iterations. But there was no discernable difference in whether or not teams had Sprint Reviews, a Product Backlog, or a Sprint Backlog. Noll and his colleagues (2017) reviewed other studies on the Scrum roles and also concluded that Scrum Masters spend most of their time on process facilitation and hosting Scrum events.

Although the sample of this study is small, it does provide some support for the notion that Scrum Masters are primarily active as coaches and facilitators. 

Finding #2: Most Scrum Masters see the Sprint Retrospective as the best indicator of their success (and not the Sprint Review)

The Scrum Guide states that the Scrum Master is “is accountable for the Scrum Team’s effectiveness”. This makes my effectiveness as a Scrum Master a function of how effective my team is, and to what extent I am able to support my team in becoming more effective over time.

So which of the Scrum events gives me the best opportunity to assess how effective my team is, and therefore how effective I am in my role? To answer this question, I ran a poll on LinkedIn. The question I asked was “Of the various Scrum events, which one gives Scrum Masters the best opportunity to determine how effective they are?”. 58% of Scrum Masters picked the Sprint Retrospective, with the Sprint Review (25%) as a (distant) second (N=558).

The keyword here is “effective”. I infer from the results that there are broadly two perspectives on what this means to different Scrum Masters. The first perspective is process-based. Scrum Masters that take this perspective treat the Scrum framework primarily as a way to learn, grow and improve through an empirical process. Scrum teams are only as effective as their ability to learn, and Scrum Masters are there to ensure this happens. The Sprint Retrospective is the obvious indicator of how effective that learning is, and thus how effective a Scrum Master is.

The second perspective is value-based. Scrum Masters with this perspective treat the Scrum framework as a means to deliver (more) value to stakeholders through an empirical process. Scrum teams are only as effective as their ability to deliver value to stakeholders, and Scrum Masters are there to develop that ability. From this perspective, it makes sense to pick the Sprint Review as the most important indicator. After all, the Sprint Review is the only event of the four that explicitly includes stakeholders. Furthermore, the Sprint Review gives the best sense of whether or not (more) value is being delivered, and which impediments are making this difficult. 

The results suggest that the majority of Scrum Masters take a process-based perspective first. I think this aligns well with how the role of Scrum Masters is understood in writing, training, books, and even the various practices that are typically associated with Scrum Masters. It also fits well with the focus on facilitation and how Scrum Masters often describe themselves as facilitators. 

My point here is not that one perspective is better than the other. Both are useful and valid. I also recognize that the forced-choice nature of a poll leaves out a lot of nuances. But I have a hunch that many Scrum Masters favor a process-based perspective over a value-based one. I think this leads to a narrow view of success, especially when Scrum Masters never look at their team from a value-based perspective. It is not hard to imagine a Scrum team with excellent Sprint Retrospectives and a safe learning environment, but no releasable increments, no stakeholders, and no good understanding of what is valuable to stakeholders. Is such a Scrum team really effective? Is their Scrum Master?

Finding #3: Many Scrum teams struggle to understand the needs of their stakeholders

So far we’ve looked at Scrum Masters. But they are obviously part of Scrum teams that are, in turn, part of larger systems. To answer the question of what makes Scrum teams effective, we analyzed data from 1.900 Scrum teams from across the globe (Verwijs & Russo, 2021). We defined “effectiveness” from two perspectives; how satisfied are the stakeholders of a team with the outcomes of the work, and how satisfied are the team members themselves? We were able to predict a substantial part of effectiveness (between 35 and 58%) with five core factors:

1. Responsiveness, or how frequently teams can release small chunks of value.
2. Stakeholder concern, or how well teams understand the needs of their stakeholders.
3. Team autonomy, or the freedom that teams have to decide over how to perform their work and how to distribute work within their team.
4. Continuous improvement, or the presence of a climate in teams that encourages learning, experimentation, and improvement.
5. Management support, or the degree to which managers understand why teams work the way they do and support this.

The core model, along with the effects between factors, is shown below. For the full definitions and details on how we measured this, please consult Verwijs & Russo (2021).

What is important to note about our model is that the right side — Stakeholder Concern and Responsiveness — is mostly about the outcomes: how valuable is what teams deliver, and how often do they actually deliver it? The left side — Continuous Improvement and Team Autonomy — is more about the foundation to make the right side possible. When we compare how 1.900 Scrum teams perform — on average — on the five factors, we get this:

The results show that of the five core factors, Stakeholder Concern is clearly the most challenging for most Scrum teams. While the differences may seem small, the effective scoring range of all teams lies between -1 and 1 point of the reported averages. All reported differences are significant (p < .001, N=1.879).

So what does this mean in practice? We measured “Stakeholder Concern” through a number of topics, such as “product discovery”, “stakeholder collaboration”, “high-quality Sprint Reviews” and “value focus”. The picture that emerges from this is that many Scrum teams struggle to collaborate closely with stakeholders, and thus struggle to understand what is important to those stakeholders. Within the context of our model, this impedes how effective they can be.

But isn’t “Stakeholder Concern” the responsibility of the Product Owner? To some extent, yes, but it is also part of the Scrum Master's accountability to ensure that stakeholders are part of the process. Without that, the entire empirical process suffers— and that certainly is what Scrum Masters are accountable for. In practice, this means that Scrum Masters spend a lot of their time working with their Product Owners to:

1. Find the right stakeholders to invite to Sprint Reviews
2. To organize effective Sprint Reviews where actual stakeholders give actual feedback on the work that was done by a team and identify next steps.
3. Find ways to let team members effectively collaborate with their Product Owners and their stakeholders.
4. Find ways to help teams understand what is valuable to their stakeholders, and to organize this into a roadmap and Product Backlog.

Finding #4: Scrum Masters may not contribute to a better understanding of the value in teams (through coaching?)

I recently had the opportunity to analyze anonymized survey data from 67 Agile practitioners. Rishi D Markenday and Dave West kindly provided this data from their on-going survey. The survey consists of 23 multiple-choice questions. We used a statistical technique called “exploratory factor analysis” (EFA) to identify five factors:

• Understanding of Value, which consisted of questions like “A team in my organization has a clear understanding of who we are delivering value to” and “Team members in my organization are aware of how their work creates value and is aligned with what my organization cares about”.
• Empowerment, which consisted of questions like “Team members in my organization are empowered to challenge organizational goals”.
• Agile Coaching, which consisted of questions like “In my organization, a typical Scrum Master/Agile Coach helps us learn how to gain value from transparency, inspection, and adaptation”, “In my organization, a typical Scrum Master/Agile Coach spends a significant amount of time helping teams do agile the right way” and “In my organization, a typical Scrum Master/Agile Coach spends a significant amount of time helping us understand how agile can help achieve value for the organization”. 
• Status Reporting, which consisted of questions like “For teams in my organization, Sprint Reviews feel like a status report on their work”.
• Alignment, which consisted of questions like “During Sprint Reviews, team members in my organization inspect how the results of their work impact the wider organization”.

We found that Cronbach’s Alpha was above .70 for four factors, which indicates acceptable reliable measurement for such a small sample. Only Status Reporting came in at .68.

The hypothesis we wanted to test is if teams that receive coaching from Scrum Masters and Agile coaches develop a better understanding of value than teams that do not. We found that Understanding of Value was not significantly correlated with the degree of Agile Coaching (r = .21). We also performed a multiple regression with Understanding of Value as the dependant variable and the other factors as independent variables. Although the model was significant (p = .01), only Alignment significantly predicted understanding of value — regardless of the level of Agile Coaching (B=.201, p < .05). 

I should emphasize that this sample was small and all surveys were self-rated. So these results certainly do not meet scientific rigor. The data is also strictly correlational, so causal effects are impossible to determine.

But the finding is intriguing when you consider them in the context of the other findings in this post. The questionnaire specifically asked if coaching by Scrum Masters was aimed at understanding value. But whether people scored high or low on those questions, it didn’t predict the actual understanding of value observed in teams to any significant degree.

If you’re willing, you can help by also filling out the survey by Rishi D Markenday at https://www.surveys.online/jfe/form/SV_eKaA0oEubErR8cS

A summary of the findings

• Scrum Masters spend most of their time on facilitation and mentoring/coaching.
• Many Scrum Masters seem to take a process-based perspective on Scrum, where the focus is more on the quality of the process than on the value delivered to stakeholders.
• Many Scrum teams struggle to understand what is valuable to their stakeholders.
• Coaching and mentoring by Scrum Masters does not seem to build a better understanding of value in teams.

The Scrum Master as a Leader

So what does this mean? I will now engage in a speculative discussion of the data I shared in this post.

A pattern I see in the findings is that many Scrum Masters overemphasize the “servant”-part of their servant leadership. This is reflected in the strong emphasis on facilitation and coaching (findings #1 and #2). I’ve noticed this too in the hundreds of Scrum Masters I’ve met during our Professional Scrum Master II Classes. When it comes to involving stakeholders and ensuring Sprints result in valuable increments, a common thought among Scrum Masters is that Product Owners are accountable for that. And while that is true to some extent, it is also true that Scrum is just another ineffective process if none of that is happening.

The knee-jerk response of many Scrum Masters is to suggest coaching or a Sprint Retrospective in response to impediments. But what is there to reflect on — really — if teams haven’t experienced an alternative first? It feels like learning how to cook by only reflecting on a mediocre dish you just made, without tasting examples of how else that dish could be prepared.

This may also explain why Agile Coaching in itself does not contribute strongly to a better understanding of value (finding #4), especially when it isn’t supported with better examples. Regardless of how many powerful questions you ask, it remains hard for teams to see beyond the “event horizon” of their organization and see better alternatives. I suspect that many Scrum Masters are also caught in that event horizon, and also don’t see how to materialize those alternatives. But then it is also quite understandable why teams can perceive Scrum as a superficial, pointless process that doesn’t add anything. 

At the same time, we see that many teams struggle with stakeholder collaboration and understanding their needs (finding #3). This endangers the entire purpose of Scrum. So there is a lot of work for Scrum Masters to do aside from facilitation and coaching. This draws attention to the “leader”-part of servant leadership. I expect that Scrum Masters are more effective when they take initiative and lead teams into experiencing alternative ways of doing things and then to reflect on what that was like. This leadership also means that you have to sometimes put your foot down as a Scrum Master. Are teams not doing proper Sprint Reviews? Don’t wait for them to see the light, and use whatever creative trick you can conjure up to make it happen nonetheless. Are teams not delivering a done increment every Sprint? Unleash every creative trick you have to make teams realize what opportunities they lost to deliver value to stakeholders. As Scrum Masters, we should show the way. We should be leaders in empiricism and not just its facilitators. By showing the path, we can demonstrate what an empirical process looks like and how it delivers more value to their stakeholders.

“We should be leaders in empiricism and not just its facilitators. By showing the path, we can demonstrate what an empirical process looks like and how it delivers more value to their stakeholders..”

Some actionable tips to focus on value

• When you enter a new Scrum team, make sure to ask as early as possible “When and how do you collaborate with your stakeholders?”. The presence of stakeholders is a critical sign of a healthy Scrum process.
• If you are unable to get stakeholders to visit your Sprint Reviews, go and visit them to do a review there — one by one if necessary.
• Instead of process metrics, focus on customer satisfaction and quality indicators to see if the outcomes of the process are worth it.
• Start each Scrum events with a re-iteration of the purpose of your product, and what makes it important to its users and customers.
• Even when your team is completely unable to release increments every Sprint, ensure that a Sprint Review with stakeholders takes place. Use the Sprint Review to both review what was “delivered”, and to leverage the support of your stakeholders to overcome impediments to release frequency.
• Investigate techniques and methods for stakeholder collaboration and product management. Even though the Product Owner will do much of this, Product Owners often also need to see examples.
• We added a slew of do-it-yourself workshops to collaborate more closely with stakeholders and to have better Sprint Reviews. Most are free, some cost a few bucks.

Closing Words

In this post I reviewed data from four sources to test a hunch I have. I expect that Scrum Masters often focus too much on the quality of the process, and too little on the outcome of that process. Scrum is a blueprint for a process that exists to deliver actual value to actual stakeholders. That incremental and continuous delivery of value is the principal source of learning in and around Scrum teams. I believe that Scrum Masters can be more effective when they balance a process-based perspective with a value-based perspective. This means that Scrum Masters should take a leading role in designing effective Sprint Reviews, drawing in stakeholders and emphasizing the need for this clearly. I expect that Scrum Masters that stake a strong stance here are more likely to see effective Scrum teams over time. And yes, that is a hypothesis I’d love to test with some actual data!

If you want to diagnose how effective your Scrum team is, you can our free Scrum Team Survey to do so. In addition to a super detailed report, you receive tons of actionable feedback on how to improve. And subscribers get even more! You can also help by also filling out the survey by Rishi D Markenday at https://www.surveys.online/jfe/form/SV_eKaA0oEubErR8cS.

 

----------------

References

Noll, J., Razzak, M. A., Bass, J. M., & Beecham, S. (2017, November). A study of the scrum master’s role. In International Conference on Product-Focused Software Process Improvement (pp. 307–323). Springer, Cham.

Shastri, Y., Hoda, R., & Amor, R. (2021). Spearheading agile: The role of the scrum master in agile projects. Empirical Software Engineering, 26(1), 1–31.

Verwijs, Christiaan, and Daniel Russo. “A Theory of Scrum Team Effectiveness.” arXiv preprint arXiv:2105.12439 (2021).

Research Gaps

Most of the data sources and studies I shared suffer from a low sample size. This makes it harder to make robust claims. I feel there is a strong need for more data on:

• What do Scrum Masters spend their time on? How is their experience reflected in this?
• Which of the activities that Scrum Masters spend their time are most likely to positively impact the effectiveness of Scrum teams?
• Which interventions and strategies are most likely to help Scrum Masters encourage empiricism in and around their teams?

Perhaps we can work together as a community to apply an evidence-based approach to these questions?