Quality Metrics - Relationships between Agile Velocity and # of incidents

We measured # of incidents and tested different items to check if we can find some correlation to measure.

Have incidents been correlated in any way to the Definition of Done? If not, why not?

Has technical debt reduced with time? 

Is there a correlation between the number of Done increments (going up) and the number of bugs (going down)?

There are a number of ways to measure this but the goal should be to correlate the number of Done increments, with quality, with usage, with customer feedback, with sales/subscriptions, with return on investment. Keep your charts and reports simple. Don't try to oversell leadership.

Always include the most important metric. The customer!

One metric that we use at my company is the defect rate. We use it for bugs, but you could use it for incidents.

We cannot bind it to velocity, as teams don't use story points; but even if teams did have a velocity, we would still choose to base it on throughput. I won't explain why, but the argument against using velocity for anything other than planning is well documented, and I recommend you look into it.



For us, the defect rate for a given time period, is the number of reported bugs, divided by the number of items deployed to production.

Given the nature of our product, we can expect some noise in the metric, from legacy bugs and known issues, and obviously some bugs are not noticed immediately after the deployment, but the metric does show trends and variations, which can drive useful inspection and adaptation.

The advantage of a rate is that in general we expect a proportional increase in defects if we get more done, and a rate takes account of that. Whereas just measuring the absolute number of reported bugs would misleadingly suggest a reduced quality.

Of course absolute numbers do have an impact on the business too (e.g. time spent on customer support, or managing the backlog), so we measure them as well, but not as a quality metric.

After introducing the defect rate, we have seen it reduce dramatically. Some of the reasons were obvious, such as it driving people to look for underlying trends, and resolve the root cause; but interestingly it also led to useful discussions about what counts as a bug.

We set more sustainable expectations about what we support, and have eliminated the waste of fixing things that aren't important enough.

It also encouraged engineers and the support team to challenge bug reports that were based on a misunderstanding of the product. As bugs were often treated as business-as-usual, functionality was often being improved beyond its original design, but it hadn't been subject to the normal consideration and prioritization by the Product Owner.

How does the team define the “Quality”? How does the organization defines the “Quality”? 

Is the organization looking for quality sustainability metric for the product or the Scrum team? 

Do you think that the defects or incidents completely reflect the trajectory of quality evolution even if there is a correlation?

Answer to Ian Mitchell: The correlation has been measured between 'DONE' velocity for a print vs. # of incidents that occurred in that sprint.

Thanks Simon Bayer for the comments. I understand velocity is not a good choice for being part of a KPI, however that metric had the highest correlation within our data set. (30 sprints). 

Can you point me to an article etc. where I can assess the disadvantages of using velocity as a metric?

Hi Aditya, Quality from stakeholders' point of view is the sustainability of the application.

The correlation is positive between volume of work and # of incidents. The higher the velocity is , the higher the number of incidents.

Can you point me to an article etc. where I can assess the disadvantages of using velocity as a metric?

Sure! Here are a few:

https://www.agilelearninglabs.com/2013/08/should-management-use-velocit…

https://metrictale.com/2019/02/12/why-its-a-really-bad-idea-to-use-velo…

https://www.scrum.org/resources/blog/agile-metrics-velocity

To be clear, I'm not advising you to stop using velocity as a planning metric.

Although I personally believe there are better options for planning, my strong advice is to not use it as a performance metric.

Hi Ozgur, Thanks for the clarification. 

May I point out something from your answer that may help you in your efforts? 

You have mentioned that the definition for "Quality" in your scenario is the sustainability of the application and not the Scrum team. Does it not imply that the stakeholder wants your team to deliver on one or more of the following quality parameters?

1. Perfection
2. Consistency
3. Eliminating waste
4. Speed of delivery
5. Compliance with policies and procedures
6. Providing good, stable product
7. Doing/delivering it right the first time
8. Delighting or pleasing customers (by meeting or exceeding agreed expectations)
9. Total customer service and satisfaction

Like Ian, what I would like to draw your attention to is the fact that the velocity (a work estimation technique of the Scrum team) does not really reflect on the quality of the product even if it appears that there is a strong correlation. Also, note that all correlation metrics are unitless and have a built-in inherent risk of failure since they are functions of estimation.

At the end of the day, customers or stakeholders do not care much whether the velocity over time has improved or not, but they do demand satisfaction on at least one of the nine parameters I mentioned above. As Ian and Mark rightly mentioned the right metric should incorporate DoD plotted against other parameters that you may think are relevant to delivering one of the nine ( or perhaps more) criteria. 

Given the nature of your work have you considered measuring the time it takes to get from incident creation to resolution? I feel that is the one thing I would want to know from my DevOps team. You could also look at the number of times you have to touch the same code.  That points towards potential for defects or potential for refactor. 

Don't try to make data fit a metric or a metric fit the data.  Find a metric that makes surface to report given the reason you want to surface it and then determine what data points are best used to communicate your desired message.