Avoiding Vanity Metrics in Agile Related Changes

Let's take your example of the avatar. If you measured success in terms of clarity about who is working on what, that could indeed be a vanity metric. More generally, there is a danger of measuring transformational progress tautologically, i.e. "we have succeeded in implementing practice X and therefore we have been successful". This is the kind of behavior in which organizations might eventually be seen to be doing agile, but without actually being agile.

However, that assessment may be useful if it is framed just as an acceptance criteria for a particular change, rather than as a metric indicating transformational progress. After all, you need to see whether or not the people you are coaching to use avatars are actually doing so. In other words, has the specific practice been adopted, regardless of any expected benefits?

The actionable metrics are to be found in whether or not those expected benefits genuinely happened. In the case of avatars for example, their use may be expected to bring better transparency to stakeholders about the totality of work in progress, improved throughput and forecasting, or perhaps simply an improved ability of the team to collaborate and create a Done increment. Where pain-points around these matters exist, there will be a rationale for coaching certain patterns and practices, such as avatar usage.

If you have a transformation backlog, describe each item clearly in terms of the pain-points which need resolving (e.g. "we need reliable data every day about the work which is genuinely in progress"). A good description will indicate the actionable metrics for determining whether or not the most appropriate practices are being coached. Use this to set expectations. This may seem like a rather trivial thing to do, but considerable thought must be invested and it is critically important.

It's starting to click.

If the problem is "we need reliable data every day about the work which is genuinely in progress" then a metric should be established to determine if that problem continues to exist after a practice has been implemented to resolve it.

For some reason my mind has wanted to show that Team A is delivering x more value each Sprint because of the implementation of X, Y, and Z practices.  This is the tricky bit because the metric described above won't measure that.  That is fair and logical... so why am I thinking that way?  I worry it's because after 100 implementations of new practices, how do I demonstrate that the new practices are actually providing value to the teams/organisation?

If the problem is "we need reliable data every day about the work which is genuinely in progress" then a metric should be established to determine if that problem continues to exist after a practice has been implemented to resolve it.

Correct.

For some reason my mind has wanted to show that Team A is delivering x more value each Sprint because of the implementation of X, Y, and Z practices.  This is the tricky bit because the metric described above won't measure that.  That is fair and logical... so why am I thinking that way?  I worry it's because after 100 implementations of new practices, how do I demonstrate that the new practices are actually providing value to the teams/organisation?

Why would *you* be accountable for that value in the first place? If you are a Scrum Master or coach, it's very unlikely you own the organization's transformation effort. It's more likely to be owned by executives who are accountable for its success to stakeholders including shareholders, the board, employees etc. They are the ones who must pony up a sponsor and a transformation backlog owner. You can help to frame and articulate useful change items, but the scope of change almost certainly belongs to others. In Scrum, the Scrum Master is not accountable for the value stakeholders obtain - the Product Owner is - and it's the same principle here.

Understood.

At risk of steering this topic off course and rambling on, I am currently the owner of that transformation backlog (by default and its current nature of being local to the IT department), but I suppose as it's scaled, which it will need to be for sufficient organisational impact, the backlog owner role may need to be reviewed. 

At that point, value of transformation will be determined at a less granular level than measuring the impact of one particular change.

Anyway, thanks a lot, Ian!