Variability in forecasting

Are you measuring whether the Sprint Goal is met and if so, how? 

For example, if a sports team needs 2 points to win the game, and they scored 3, did they achieve the goal? Yes. In the same manner, you check whether the outcome specified by the Sprint Goal has been met. It's either a Yes or a No.

Scrum Teams commit to a Sprint Goal, not to the work that is in the Sprint Backlog.

Consider if it is important to focus on velocity or achieving the outcomes for which Sprints are undertaken.

Are you measuring whether the Sprint Goal is met and if so, how? 

A Sprint Goal should be agreed by the Development Team with the Product Owner, and being suitably valuable has the potential to become obsolete.

As a coherence which causes the team to work together, I'd suggest that the Sprint Goal is something to be assessed continually, i.e. is it being met or on course to be met.

But I'm worried about teams whose completed is drastically different from their forecast.  

It is possible to inspect the accuracy of a forecast.

Imagine at the start of the sprint, a Development Team forecasts it will get the following items done:

#110, #111, #113, #115, #118

But by the end of the sprint, what it actually got done was:

#110, #111, #120, #115, #122, #114

So it actually got 60% of its forecast items done (#113 an #118 were incomplete or maybe not even started)

It also managed to get 3 other items done (#120, #122, #114)

The team can now inspect why this was the case, and consider improvements for the future.

Perhaps the team learned more from what it released already, and was able to react early to feedback and avoid the waste of doing items that are no longer needed.

Perhaps urgent work emerged during the sprint. On the one hand, the team was able to pick this up and get it done, but what does this say about their ability to plan? Can we anticipate this work better in the future? If it comes from some kind of defect, can we reduce the number of these defects?

Perhaps the team lack focus, and weren't able to stick to their original plan, and the failure to get items #113 and #118 done was an accident.

The forecasted body of work isn't the important part.  The achievement of the Sprint Goal is.  If the team while working towards the goal identifies that they have picked the wrong items from the Product Backlog they should be able to change it.  Expanding upon @Steve Matthew's sports analogy, isn't it common for a sports team to change the line up of players and their strategies during a game in order to achieve the goal of winning the competition?  

When people hear the word forecast the first thing that comes to mind is usually weather.  But no one really expects a weather forecast to be completely accurate. They are usually more accurate when it is today's forecast than if it is a forecast for 5 days away.  So in a 2 week sprint, the forecast will become more refined as you move towards the end of the sprint.  In that manner, the makeup of the items in the Sprint Backlog will likely change in order to be more accurate.

Are you measuring whether the Sprint Goal is met and if so, how? 

The Sprint Goal should include an agreement on how to know if it is accomplished.  So the team should be able to know if the Sprint Goal is accomplished.  In fact, it should be discussed every day as part of the Daily Scrum.  

(from 2% of their forecast being done to 3900%).

Out of curiosity, where did you get these statistics? How were they measured?  How were they validated? 

Great points.  I don't want to discourage teams that are accomplishing the business' needs.  If a sprint goal is met but velocity was off then they were still successful because the sprint goal was met.  When looking across multiple dozen teams I only have some data points accessible, like velocity and what was committed to a sprint.  When a team commits 50 story points to a sprint, but only completes 1 story point - I wonder if they also met their sprint goal.  I can also see that the 1 story point delivered wasn't the highest priority of the stories in that sprint. I don't want to rely on points committed to a sprint so I'm wondering how others track if sprint goals were met or not (across multiple dozens of teams).  

I'm meeting with each team individually and there are some teams struggling to understand the need for any commitment (even commitment to a sprint goal); as such, their retrospectives are not introspective.  

So how do others measure if a sprint goal was met or not.  Is this most commonly self-reported yes/no?  Is this measurable?  Should it be measurable?  

Shouldn't it be value delivered which is measured or otherwise assessed?

This is going to get long.  Sorry but I can't answer as simply as many other of the commenters in this forum.

if you are looking at metrics across teams it sounds like you are in a position that is not defined in the Scrum Guide.  And you are trying to use old style Project Management techniques to compare teams.  My advice is stop doing that.  :)  

Velocity is a very misunderstood term and it appears that you are trying to use Story Points to indicate velocity. Story Points are estimates used to plan for work. Let me help you understand the problem with that. This definition of velocity as a noun comes from dictionary.com

• rapidity of motion or operation; swiftness; speed:a high wind velocity.
• Mechanics. the time rate of change of position of a body in a specified direction.
• the rate of speed with which something happens; rapidity of action or reaction.

 Also from dictionary.com here are definitions of estimate as a noun.

• an approximate judgment or calculation, as of the value, amount, time, size, or weight of something.
• a judgment or opinion, as of the qualities of a person or thing.
• a statement of the approximate charge for work to be done, submitted by a person or business firm ready to undertake the work.

Estimates are approximate judgerment based on the information you have available at time you make the estimate. And velocity is the rate of speed with which something happens.  So by measuring your velocity using Story Points (estimates) you are seeing how fast things can happen using estimates which are not based on complete information. I say that because how often do you start working on something and find out something you didn't know before?  Especially if this is part of an emerging complex body of work?  I suggest you read this blog post by Ron Jeffries who is given the responsibility of creating User Stories and Story Points  https://ronjeffries.com/articles/019-01ff/story-points/Index.html.  There are other ways to determine the effectiveness of teams.  I'll get to those soon. 

Another problem with what you are trying to do is that each team works differently, works on different work and you are trying to treat it all as equal.  One team may be much more conservative in their estimages and will have tend to have larger estimates while an agressive team has smaller numbers.  By trying to equate a conversative team with an agressive you are comparing apples to coconuts. Velocity should be used by the individual teams to montor their own health and ability to plan accordingly.  It should never be used to compare one team to another no matter how you choose to define velocity. 

Even trying to track whether Sprint Goals were met is not effective across teams.  What benefit does it provide and what is considered "good"?  Will you start to say that all teams must meet the Sprint Goal for 63% of their Sprints in order to be considered successful?  

Sprint Goals and velocity are tools for the individual Scrum Teams, not something that should be used to judge teams across the organization. 

So, how do you measure team effectiveness?  I suggest measuring the customer satisfaction for the work that the team delivers. Do you really care if a team can complete 20 stories or 200 story points in a 2 week sprint if they never deliver anything to the end user that improves the value of the product? Or if they are delivering every 2 weeks but their deliveries does nothing to improve the customer usage or desire to use the product? Let the team focus on how they work and you should focus on how the work they do improves the lives of the users because that is what will drive usage and ultimately profits for a company.  

Take a look at this excellent source of information provided at no cost on this site.  https://www.scrum.org/resources/evidence-based-management. It will help you understand how the company should adapt their view of success when working with agile teams. 

Ah, I'll think through the EBM internally as a team as new metrics.  We haven't evolved beyond the basic JIRA data / metrics available which aren't great measures so this is definitely something we should consider.  

Katherine, I sense you're trying to help the teams, but like Daniel pointed out, it does sound like you're in a position that's not defined in the Scrum Guide.

That's not necessarily a problem. It has the potential to benefit the teams and organization; particularly if you're able to offer servant leadership; or it can impede the Scrum Teams if your role conflicts with the accountability and self-organization expected in Scrum.

Perhaps the most important things for us to know, so that we can help, are what is your role, how much autonomy do you have in that role, and what are you accountable for?

What is your expected relationship to each Scrum Team? Is there one product (with single Product Owner and Product Backlog), or are there multiple of any of these?

Then finally what are you really trying to achieve with such metrics?

What do you want to learn, and why? What kind of conversations and behaviour are you hoping they will enable and encourage?

In a data-driven management structure, I'm tasked with defining and honing our process.  Trends take a while to establish and I want to show that our teams are functioning well or improving.   So we need a little consistency across teams and we need a little flexibility for teams to innovate and stay empowered.  I have the standard tools and can see bugs per build, velocity, and such.  We don't have a structured customer set to poll for customer satisfaction.  Looking for measurement ideas that promote a little consistency and show management all is well (or improving).  

Going to address a couple of things you said. 

 I'm tasked with defining and honing our process.

In Scrum each team is responsible for defining and honing the process that they use.  Yes, that means that there could be wide variability across teams but since each team is considered to be self-organizing teams I would see that as a success. From the Scrum Guide section that describes the Scrum Team 

Self-organizing teams choose how best to accomplish their work, rather than being directed by others outside the team.

If you are tasked with improving this, ask each team what you can do to enable them to improve.  Ask them if they have the data they need?  Ask them if they need help with any kind of authorizations or system access. Ask them what impediments they have preventing them from improving. The EBM guide provides information on how to use data to show improvements to others.  

So we need a little consistency across teams ...

Why?  The only consistency you really need is that the teams are consistently producing value to the end users and the company.  Consistency in process does not mean things are going well or improving.  It just means that everyone is potentially making the same mistakes or checking the same boxes.  



Metrics for organizations using agile principles need to focus on the actual delivery of quality and not on the details of how work is done. In addition to the EBM Guide, I suggest you also look at the DORA Metrics (https://cloud.google.com/devops/). These can be used to identify trends that show impact to quality deliverables.