Evidence-Based Management (EBM) is a framework that helps people, teams and organizations make better-informed decisions to help them achieve their goals. If EBM is new to you, start by reading the EBM guide here: Evidence-Based Management Guide | Scrum.org. This article assumes that you have a basic understanding of EBM so please start with the guide if you do not.
I was working with a company and had the opportunity to sit and ask the CIO how I could help them best move forward with delivery. He said he felt like they were doing quite well with Agile practices on the ground. He thought it might be interesting to advise the portfolio management team on how they might take a more Agile approach as most of their practices had remained untouched for many years. As I left the room, he told me he appreciated my straightforward style and that he looked forward to my findings and recommendations.
For CIOs, it's often difficult to get real, direct information. People who report to them tell them the good news (the stuff they want to hear) and not the bad news (the fires that are currently being put out). It was refreshing to know that I had a CIO with whom I was working who would be open to hearing direct feedback and recommendations. Not everyone enjoys that kind of openness.
I worked with two key project managers who were responsible for generating “state of the portfolio” reports. Those reports were distributed to managers and executives across the organization regularly. Having seen a few traditional portfolios, I wasn’t surprised to see the reports were largely centered around what was on time, what wasn’t, and how much money was being spent. The reports largely lacked the content, from an Evidence-Based Management perspective, to make transparent what was happening in the portfolio.
We started digging into the data and the two key project managers were excellent at tolerating the large amount of questions that I was asking. We put together the “big three findings” and prepared for a meeting with the CIO and his direct reports. Below are the “big three findings” that we thought making transparent would help to trigger broader portfolio management process changes:
- Goals: Very few projects had goals other than “finish this by this date” and “only spend this much money”. Teams sorely lacked alignment with the customers who were inheriting the products they were building. In addition, nobody seemed to be measuring any form of value. At least that we could find.
- Projects In Progress: There were a total of 900+ projects that currently had work that was active in the portfolio. That was close to a one-to-one (project-to-worker) correlation. That is a lot of active work.
- Project Age: We calculated the elapsed time between an in-progress project and the current day. We were astonished to find that some projects had been being worked on for over 3 years.
We prepped for the meeting and were excited about showing our discoveries and a few suggestions on how EBM could help. During the meeting, we began walking through the data. To my surprise, the CIO had a visceral reaction to what we presented. He became quite defensive and felt very strongly that teams were aligned to the goals set by the executives. Yet there was little evidence of satisfaction gaps that were trying to be closed (Unrealized Value) or measures of existing customer value being delivered (Current Value) on any project.
What bothered the CIO the most, was the amount of work in progress. He started trying to poke holes in our numbers and became quite agitated when we objectively closed the holes he was presenting in his arguments. He did not want to accept that the large amount of work actively being worked on was greatly limiting the effectiveness of the organization (in EBM we view this through the lens of Ability to Innovate).
He became so upset that he ended the meeting early. Later that day, I received a meeting invite and found myself one-on-one in the CIO's office. He asked that I no longer work with the portfolio team and stated that I was digging up dirt that was going to do more harm than good. I was astonished.
The interesting thing about presenting objective data is that there is often an irrational reaction to it. Since this situation, irrational responses to objective data is something I greatly consider when starting with EBM.
What could I have done differently? I would have chosen to take a more cautious approach regardless of my read on the tolerance of the CIO receiving such data. It is never a good idea to surprise anyone with the results of data that may greatly impact their lens of a situation. I may have been better discussing our findings one-on-one with the CIO before taking it to a broader audience.
An important thing to understand when getting started with EBM is that there is such a thing as going too fast and too soon. EBM can create a lighting bolt of transparency, which in turn, could cause irrational responses. We humans can be emotionally triggered when we find a situation is much different than it appears.
Has this happened to you? What steps did you take to mitigate such irrational reactions?