What metrics (if any) would you start collecting

None.

To be clear, it's not because I don't want to have data to help support making informed decisions. Instead, there's no use in collecting data until there's a good reason to collect the data. If I was joining a new team, I would start with observing the team and talking to people (both on the team and key stakeholders outside the team) to understand the current state. If the team is already collecting data, I'd also understand how they collect it and who uses it.

Looking at your proposed list, some of those could be valid things to capture with sufficient reason, but others aren't. I can't think of a reason why, as a Scrum Master, I would care about the number of active users for the product, sales or revenue data, or customer satisfaction - these don't help me learn about the team's way of working. My only involvement with these types of metrics would be to perhaps coach the Product Owner on using them effectively in the context of managing the Product Backlog. The other metrics may be useful, but it depends on who is asking for them and why. I don't think that I, as a Scrum Master, would need any of them to start coaching the team on improving their way of working.

Regardless of what metrics, if any, I would be collecting, the one thing that I would do is try to make it as easy as possible to get the data. I wouldn't want to expend significant effort to calculate and report on metrics. The effort to collect the metric needs to be compared to the value and usefulness of having the data.

Another person that agrees with the "None" answer.  

I agree very much with @Thomas Owens.  Since you didn't mention that this is a new team that you are joining, why go in assuming that something is missing or not working?  Take time to understand how the team works now and find out what they feel is missing or needs work.  Then, and only then, start to find the data that can be used to help them.  A lot of the metrics you propose are probably available already and with a little work you can get historical as well.  But do they provide anything that will help the team become better? 

The need for a specific metric can change over time. I tend to recommend metrics that help address problems that are occurring and then phase them out as needed.  Some metrics may last for a long time to help ensure consistency but those should be few and easy to substantiate. 

what metrics (if any) would you start collecting as early as possible?

Preferably none; I would rather encourage a self-managing team to collect their own metrics and to monitor their own progress.

Before measuring any specific metric I would make sure there is a need to measure that specific metric and that it will produce some kind of value for the team and organisation. Way to many organisations measure things without clearly being able to explain why, leaving those measured to come up with explanations.

We should measure what needs to be measured, clearly explain why, what it will show us and how we can tackle it if the measurement indicates an issue.

The best thing is to involve the team in identifying and deciding on measurements to get their commitment and understanding around whatever is measured.

Thanks for your input.

To clarify, this is not a new team. From my initial conversation I believe there are currently no metrics collected and no historical data of, well anything. There are of course a gazillion other things that are more importand when joining a team but my question regards metrics specifically. Whatever metrics is collected need of course be valuable to the team/organization.

@Thomas

I definitely think that it is (should be at least) of interest for the team to be aware of things like customer satisfaction. Trivia: What is the 1st principle of the agile manifesto? 

The team is not working in isolation and a team that regularly build and ships stuff that noone wants or uses is, although common, not the goal.

Allow me to rephrase the question like this instead.

What metrics can the team/organization use to know that they are improving?

To clarify, this is not a new team. From my initial conversation I believe there are currently no metrics collected and no historical data of, well anything. There are of course a gazillion other things that are more importand when joining a team but my question regards metrics specifically. Whatever metrics is collected need of course be valuable to the team/organization.

I don't think it matters if it's a new team or not. As someone new joining an organization as a Scrum Master, or in any kind of coaching and process improvement role, I would not make any changes right away. The scope of the engagement - a single team, a small group of teams, a scaled set of teams working on a single product, an entire organization - dictates how long that observation takes. It's not just passive observation, either. Asking questions about what the team does, why they do it, and what problems they are facing is essential.

Based on the problems that the team is facing, starting to collect some metrics could be valuable. However, there could be work that can be done with metrics. There could also be a need to configure tooling to make collecting metrics easier - if it's too difficult to collect metrics, it could make the value of the metrics not worth the cost.

I definitely think that it is (should be at least) of interest for the team to be aware of things like customer satisfaction.

Yes, it could be of interest to the team, primarily the Product Owner. However, the team is the one who should determine that this metric can be useful to them in performing their responsibilities. The Scrum Master may bring it up as a possibility in coaching, but the Scrum Master is not the one who decides that the metric should be collected and used.

What metrics can the team/organization use to know that they are improving?

This is not an answerable question. It depends. Improving how? Depending on what aspect of product development and delivery is being measured, there are several metrics that could possibly be useful. Listing every possibly metric isn't that useful. The team should determine what questions they want to answer and the Scrum Master can help with suggesting some metrics, preferably with an understanding of the ease (or difficulty) of collecting and evaluating the data. The team ultimately decides how they track improvement.

I don't believe that any metrics are necessary. It's entirely context-dependent. Qualitative data can be just as useful as quantitative data.

In some cases, management will ask the Scrum Master to run a report on the amount of work completed by an individual team member. In other cases, they will do it themselves. But I think that in all this talk of "team" let's not forget that some people contribute more than others and they should be rewarded as such through an incentive plan. So that metric is important.

@Mark: I would be extremely careful to view individual team members of a self organised team. Not that it can't ever be done, but I would want the team to first identify if they consider someones contribution or lack thereof a problem and a big enough problem to not solve it themselves.



Ofc there can be members of a team that does way more than they should or way to little. But if someone does way above what they should on a team the ideal situation is for the team to share the work in a better way.

Also, fostering a team sprit where people are there for each other, commit together and as a unit are responsible for their deliveries but then incentivize them as individuals could be counter productive. Say you have a team of sales people that you want to work as a team, but then you give them each individual bonus targets. What this often leads to is short term, self centered decisions. The sales guy who keeps information to himself, closes his small deal instead of helping a team member with a HUGE one (since he would not "benifit" from it). Or the CEO who lays people off to have financial numbers that reach his targets so that he gets his bonus. How that affects the company long term is not his concern... often he has left for another company before that has to be dealt with anyway. And on and on...



I can see justification in incetives, but I would do all I can to make them on the team level. That would then encourage the team to work even better together to reach whatever incentive there was as a team. Or a profit sharing program for everyone, which forster collaboration across teams, departments and so on.



But one should keep in mind that incentives, just like KPI's, can create behaviour that we might not have wished for or intended.

I would start with the 'why' - what are the challenges/impediments the team is facing, why do they need metrics, what are the metrics that matter for the team. Thus, based on the need, identify the metrics and how can I as a SM, help the team to measure and course correct based on the metrics.

In other words, the key is knowing why a metric is needed and what the team is looking for from the measurement and monitoring of those.

If I was told that "I'm sorry but there's not way to tell that you improved based on this initiative or not" I cannot see why I would have any incentive to proceed with such an initiative.

Any initiative should be able to answer why are we doing this, what will be different compared to the current state that will tell us if we're moving in the right direction and improving.

If I was told that "I'm sorry but there's not way to tell that you improved based on this initiative or not" I cannot see why I would have any incentive to proceed with such an initiative.

Any initiative should be able to answer why are we doing this, what will be different compared to the current state that will tell us if we're moving in the right direction and improving.

This seems backward to me. You need to know what you want to change and what the desired end state is in order to start an initiative. That identification of what you want to change is how you determine what measurements or metrics you need to collect.

Since a Scrum Master is concerned with the team's way of working, an example of an initiative may be "reduce the number of defects found in production". Ideally, you would quantify that somehow, such as "reduce the number of critical defects in production to no more than 2 per quarter" or "reduce the number of defects in production by 25%". This means that you need to collect data about defects, their severity, and where they are found. Minimally, you need a count of the number of defects found in production. You may also want to count the number of defects found in-process to calculate ratios, but this would be optional. You would count the number of defects reported in production per unit of time. As you make process changes, you would expect to see a downward trend in this number if your changes are effective. It may take many weeks or months to have sufficient information to say that your process changes were effective.

This goes back to the metrics follow from the need, rather than collecting metrics for the sake of collecting metrics for future use.

Any initiative should be able to answer why are we doing this, what will be different compared to the current state that will tell us if we're moving in the right direction and improving.

I don't disagree with this statement but in order to provide the information to determine improvement means that the information is specific to that initiative. 

What metrics can the team/organization use to know that they are improving?

What everyone is trying to say is that in order to suggest a metric we would need to know the answer to these questions.  What are you trying to improve?  Quality of the deliverable? Deliver faster? Improve a specific feature within the product?  Add more features to the product? Better communication with stakeholders?  

Based on what you are trying to improve, the metric could change. Remember that agile practices are to allow an organization to quickly adapt to changes.  They promote iterative delivery that is able to adapt to feedback.  That does not only apply to product features.  It also applies to the way a team works.  Small, iterative changes to how you work will also allow teams to experiment in order to find the right changes. And just like with the Product Backlog and Sprint Goals, each "item" should be small enough to be done during a Sprint with a specific goal in mind. Yes, it may take more than one Sprint to realize a change but you should be able to implement a change in a single Sprint.  Then you can continue that way until a time that there is enough information to know if there was a difference. Consider the work you do as a "product".  

Pick metrics for a specific reason to provide information pertinent to that purpose. Picking wide ranging, ambiguous metrics are not going to help you and in some cases could harm your chances of improving. 

Another way to look at this is to think in terms of experiments of an hypothesis you want to validate. Some Scrum Teams use hypothesis driven development, and use a template such as:

We believe [doing this] for [these people] will achieve [this outcome].

We will know that this is true when we see [this measure changed].

Albert Einstein is credited with the following quote:

Not everything that counts can be counted, and not everything that can be counted, counts."

Personally, from the research I've done on agile transformations, it is advisable to decide what metrics to use to measure success and start recording them early. I've been in charge of the agile transformation in my company, which started about a year ago, and I still don't have metrics as I didn't do this research until more recently. However, I did introduce an anonymous survey that is sent out every two months, which aims to measure the Tech Team's health. I decided to do this because in my opinion, the main benefits of working in an agile way is individuals' happiness, motivation, etc. 

The people I worked with were fed up of the old ways, overworked, stressed and generally unhappy and I wanted this to change. I believe that if people are happy with how they work, are motivated and empowered, the products they deliver will benefit too so that to me is the most important metric to measure.

I have also been running small work groups to address the areas we are scoring low and get people to come up with ways of improving them, much like a retro for the entire Tech team. 

However, I have been asked by the CEO to use other measurements too and number of defects is an area I've been looking at, although I do think it's not a clear metric to use. We're constantly adding features and users, which will naturally result in more defects raised, so although I know that after each release we have a lot less issues raised that are caused by the release, overall I think we have the same amount of defects raised overtime. 

Just my two cents.