A Discussion about Unlocking Business Agility with Evidence-Based Management

In this episode of the Scrum.org Community Podcast, Patricia Kong, Kurt Bittner, Ryan Ripley and Todd Miller discuss topics in their new book - Unlocking Business Agility with Evidence-Based Management: Satisfy Customers and Improve Organizational Effectiveness.

They discuss Evidence-Based Management, , using EBM for agile transformations, the importance of setting expectations, goal setting, prioritizing effectiveness over activity and more!

The book is available for shipping to the US and pre-order for the e-book here. You can use the code SCRUMORG for 35% off at InformIT

Transcript

Patricia Kong: 0:20
Hello, everyone, this is Patricia calm and I am your guest host today for the scrum.org community podcast I am here today with my friends and co authors actually have a new book that we started about a year ago very excited about it around the topic of evidence based management called unlocking business agility with evidence based management. So here today with Ryan Ripley, Todd Miller, Kurt Bittner Hey, guys. We started this book in January 2023. But we've been talking about this idea for a couple of years about evidence based management EBM. And what I know is that we started this not only to get the ideas of EBM and talk about empiricism and value, but really about just I think at that time, seeing a bunch of people who just felt really disconnected and unengaged with work. So, I know that while we were that was a really fancy long title, but when we were when we were throwing out names, we had some different ones.

Ryan Ripley: 1:29
We had some very cynical ones. I think we had some fun with some titles. It was a how not to be terrible at your work. Stop using Agile for just fun and profit. I don't know we had some really horrible ones too that I don't think we're allowed to say on this podcast.

Patricia Kong: 1:46
Yeah, no swear words. Put it out. There. We had the watermelon. The watermelon. Yeah, vacation.

Ryan Ripley: 1:55
Yep. That was one the watermelon being like, you know how everyone says the project's green, but on the inside, it's really red. I think it was a cool metaphor, but I don't think it would have actually made sense. Or people might not have picked it up right away. But yeah, the watermelon idea was good. Yeah, I'm glad that there were some adults in the room somewhere. They reined us in a little bit.

Patricia Kong: 2:20
So it be the editors. Yes. So I know that Kurt, for you. This was, you know, you're big on asking why all the time. And I think our struggle with agility, and customers in organizations always talking to us about oh, we want to go faster. We want to go faster? Why? And do you think that? That is still how companies are thinking about agility?

Kurt Bittner: 2:53
I hope that but I think so. You always hope for better, I still hear a lot of people, especially executives, thinking that agility is about increasing speed or increasing efficiency. And when you really peel that back, it, it's kind of about I mean, going faster is not a bad thing. But they they missed the point, because they think it's just faster execution of what they do today. And they're not asking the fundamental questions about are we building the right thing today? And how do we know? And so for me, evidence based management, a lot of it is about getting faster feedback, so that you can find out that, you know, if if a lot of the things that you were planning to build aren't valuable, you can find that out early, and then find out what would be valuable and then build that. And there are other things in EBM about improving the team's performance, improving their effectiveness, certainly, you know, improving their cycle time time to market. But for me, the most important thing is getting feedback on what you're doing to find out whether it's valuable and then and then you can act on that you can you can work on improving that. I've seen some data that's really interesting that says that two thirds of what companies end up building and delivering to customers, maybe isn't valuable, you know, that the data this is a guy named Ronnie COE Javi, who was at Microsoft, and I think did some work with Google. But anyway, that the data was that a third of the things that they built were valuable. A third basically didn't achieve what they set out to achieve. It didn't make things worse. And then a third of things actually made things worse and the thing that they were trying to improve. So that says to me two thirds of what you're building could be of no value or worse of negative value. So figure Going out with third what the right third is maybe a been improving that increasing that is, is really the most fundamental challenge. And then, you know, to do that you got to do a lot of other things to improve the team's effectiveness and, and speed up feedback and things like that. But that's, that's kind of the crux for me.

Patricia Kong: 5:18
So let's dive in to that a little bit. So, So Todd, one of the topics, when we were writing this, you were really passionate about was it was a chapter called Managing and overcoming expectations. And it was a lot about what do you say? How do you do that? How do we talk about goals? You know, let's not just build something because we, we can, what were you thinking about when, when you were talking, or when you were writing or putting those thoughts together with us about about the expectations about agile how to work together in that kind of environment?

Todd Miller: 5:58
Yeah, and as Kurt was highlighting, and organizations building features and things that people don't use, what we find often is that they don't even know that people weren't using them. They just filled them and then turned a blind eye and move on to the next thing. And so with that, I'm managing and overcoming expectations is about being in reality. And I feel like, a lot of the times this whole agile thing is maybe because the cool kids on the block are doing it, or because it worked somewhere else in a small context. And I really like when you have really good goals, and you have evidence of supporting your path towards achieving them. You're living in reality instead of some universe that is created largely by an organizational is hierarchical structure. And, and is limiting an ability ability for an organization to deliver the things that people want, because of where those decisions are made. That reminds me of Patricia, what you were talking about when we were at scrum de decision latency. And so managing and overcoming expectations is living in an objective world rather than a subjective world. And that's really, really hard. It's so easy to talk about, it's easy to say. But that reality could be something that an executive doesn't really want to know about, right. And then those decision making decision making latency that you were talking about in your talk, Patricia is is is created because of that, right. And then you could be every step in the chain of a decision, you can always just point to the other person for blame when it doesn't go away. So that's reality, if there was one word to describe about managing and overcoming expectations was was what was on my mind there. But easier said than done in a lot of instances. So and I think that really segues to what Kurt was saying about the usefulness of what we're building for our customers. And supporting that with evidence whether it was useful or not,

Ryan Ripley: 8:18
you know, the big shocking part about writing this book and then doing some of the conference talks. And maybe this isn't so shocking, or it shouldn't be. So we really don't know whether or not what we're doing is useful in most cases. And that's really just stunning to me. Todd and I have been doing a talk at conferences, and we've been doing some keynotes lately about your Agile transformation is a crime scene. And, you know, I started one of these talks at agile indie, with just a who's done a transformation and a bunch of hands go up. And I say, how who's transformation was successful? And all the hands stayed up? And then I asked, How do you know, and actually got off the stage and walked around the room. And we got answers like, well, we trained 20 different teams, and they all launched or the consultants, consultants told us, we were successful. And we were done. Or they had some very weird picked metrics on velocity, increased 10%, or team capacity, all sorts of bizarre metrics. And one of the things I pointed out was, none of these are delivery based. None of these are customer satisfaction based. None of these are ROI based. And you really don't know if what you did had an outcome, a good outcome or an impact on anyone. And you could hear a pin drop as the wheels were turning in their heads. And I said, look, the only reason your transformation stopped is because the budget ran out. It's not because you found success. And that got a laugh, but it's also kind of right, you know, it got a laugh, and people were like, Oh, he's being funny again, and then I just pause I go, Yeah, I'm glad you laughed, but that's the money you spent and you don't know. And I think that's Where EBM filled a great gap or where it potentially could. I think that's where a lot of people could get value out of the book. It's how do you know I think Kurt posed that question earlier in this discussion, and I think that is the seminal question. It's the, how do you know that what you're doing is actually the thing you should be doing? And we do not answer that in the Agile space, typically, and I think we brush it off way too often.

Kurt Bittner: 10:29
There's, there's one other thing too that, you know, a simple question that people can ask themselves. There's a measure that I like, it's really simple. And it's basically a measure of how agile Are you? And the measure is time to pivot. So first first question to ask is, have you ever changed direction? Have you ever dropped things from the product backlog? Have you ever changed the vision for the product based on feedback that you've gotten from customers? And if the answer is no, you're not agile, you could be doing all of the daily scrums you want, you could be, you know, essentially chapter and verse on the scrum guide. But you're not agile, if you're not actually changing your direction based on feedback. That's at the heart of empiricism that's at the heart of agility. And so that that time to pivot, which is really simple measure is just how long did it take between when you got feedback? And you actually change direction based on that feedback? And if the answer is we never changed direction based on feedback, well, first of all, you're not not even gathering feedback. Right? You can't change direction. And then if you get if you get feedback, there's an old joke that says, occasionally people stumble across the truth, but usually they pick themselves up and move on. And, you know, that's sort of the case here. So anyway, I think that asking that question about what are we really trying to achieve with our agile transformation is important. And if it's just go faster, if it's just deliver faster, but you're never measuring the results? You're doing something but it's not agile?

Patricia Kong: 12:12
Well, that's what just to piggyback off of both of those comments from from you, Curtin. And Ryan is one of the things that was really important in the book. And Ryan, you are good at pointing this out, especially when we're talking about the notion of signal from noise. And you are always kind of saying, basically show me the money. So if I'm a leader, if I'm an executive, why do I care about this? They probably don't necessarily heard about it, this agile thing, but the point isn't really about agile, they have different concerns. Do you think, do you think we did it? Right? We made that point strong enough that if you're not doing this, you're losing money? Potentially?

Ryan Ripley: 12:54
I think we did. I think all throughout the book, we're making it clear, you know, to Kurt's point. I mean, we've all touched on this, that if you're not looking at this stuff, you don't know. And that's I and that's why I hope people read it. And you know, I'm concerned that Kurt's Kurt's quip is actually true. People will stumble across the truth and then pick themselves up and move on. But I think this kind of this was the kind of book that if I'm a leader in an organization, most people may not know Todd, myself, Kurt Patricia, we've had experiences working in organizations, for years at executive levels at various levels. And what's gonna set you apart is, is your ability to look at the data objectively, and pivot as soon as responsibly possible when you know, there's a problem. You know, we see it all throughout organizations, this inability to just say, hey, that didn't work, let's try something different. And if you're able to be in a leadership position, embrace these EBM concepts and pivot as soon as responsibly possible, you are going to be lightyears ahead of anyone else. It is a massive competitive advantage. And then you can start answering that question, did we find the money or can you show me the money? Can you show me the value? But until people embrace this idea that evidence is important, and that the ability to pivot is what's going to define the winners and losers going forward. I don't know if much else is going to matter, right.

Patricia Kong: 14:32
The matter part, I'm wondering actually taught you had made a little bit of a throwaway comment about Agile is where the cool kids are. Hey, so we've talked about laters. And we've talked about this notion of hey, agile is not the point. Do you think that that's still true? That Agile is where the cool kids and all that notion because because, you know, based off of what we saw at scrum day at the conference, we were A couple of weeks ago, and we're both speakers there. And I just remember, they're like we're trying. And we understand Todd goals are so important. So we talked about the goals for maybe. And we talked about having strategic goals, intermediate goals, immediate goals. And in someone pitched a question to you about OKRs. And then you said something kind of funny. And then people start to kind of raise their arms and like, do hallelujah, I'm really clapping. What did you say? Again? I don't remember,

Todd Miller: 15:31
I've never seen OKRs implemented well, right, is what I said. And that's the truth, from my perspective. And maybe someone out here listening has had and seen it. But mostly, I see OKRs as another means of controlling people. And so a lot of what we discuss when we talk about goals from from our perspective and evidence based management, and by the way, I don't consider evidence based management and agile framework. I think evidence based management is universally applicable to a product situation, to a leadership situation to an organization to a portfolio, a lot of which we highlight in the book, I would not classify it as an agile framework, I think it can be useful for people that are running a department.

Patricia Kong: 16:10
Well, what what about the people? So we're, we're talking about the importance of organizational level, we were talking to the organization, essentially. But what about the people who really do believe in empiricism and these concepts and they, whether it's OKRs, whether they're calling their strategic goals, they say, Hey, I feel like this is just happening to me. I'm getting these goals. What What should I do I believe in this, I want to do better, how can how can I use EBM? And

Todd Miller: 16:43
I think you're making a good point is not a magic wand. Right? Yeah. But I think you're making a good point. Although we can be cynical, we do believe that. I mean, I speak for myself, but I know that you all feel the same way that people are not inherently doing things to cause harm, right. It's just been a side effect, I think, to some extent of it. And something that you that you said there that we hear so much, I think, right? I think Scrum is working in this context. I think we're more agile than we were before. After we spent all the budget Ryan was talking about, I think, right? How do you really know? And let's paint a picture of that reality. Let's really find out let's dig in and say, did our did agile work here, right? Is agile working here? How can we get better? What can we how can we be driven more empirically? How can we took Kurt's point, not build all of the things that people aren't going to use? Let's try to build the things that people are and if we're building something that people aren't going to use, let's stop that. Right, let's, let's let's pivot faster. And so a couple of things in there. as cynical as I can be sometimes about this stuff, I truly believe that people we're trying to do the best that they can with the information that they know, I don't think there has been an answer to answer for is agile working? I think there is I think evidence based management can answer that. I don't think that's the only thing evidence based management can answer. But I think that's one thing, if you were to want to use it for that, you could, I also think you could learn it use it for is my department getting better, I think you could use it for how's our organization, and all these other things. So

Patricia Kong: 18:25
I mean, that follows really the structure of the book, too, because we wrote it in a way that's, you know, here, here's the underlying ideas, here's how you would approach it at the product level, the portfolio level, and tons of case studies, from our experiences in there to think about that to see if you're in a similar situation. What do you have Ryan?

Unknown: 18:47
I was just gonna say that the things that we're kind of circling back to. We are as a culture as organizations rewarded for activity, not outcomes or impact. And so one of the hardest things I think that will that will come from this is that we're doing too many things. You know, to Todd's point, we're building things that nobody's using, and we're not being effective. And so how do you shift that mindset from activity to effectiveness? You know, because we'll hear time and time again, hey, we're, we're working really hard to implement Scrum, or we're working really hard to be, you know, a better delivery organization. And it's like, I'm glad you're working hard, are you effective. And there's a difference there. The busyness doesn't matter. The it's terrible that you're so busy. Why are you so busy, go be effective. And I think that's going to be the another one of those markers of success in this in this economy versus not successful. If you're optimizing for activity, you're sunk. If you're optimizing for effectiveness. I think you've got a shot at doing really, really well. And I think EBM points out the difference between In the to where it can at least help you identify, are we busy? Or are we winning? And there is a difference.

Kurt Bittner: 20:08
There's a to give an example of the activity out output outcome. So, an activity would be something like, you know, we're implementing a new CRM system. And if you have an OKR, or if you have a goal doesn't matter whether you're using OKRs, it's just a template. But if you have a goal to implement a new CRM CRM system, the fundamental question that you should be asking is, why are we doing that? And that's This, for me, that's the starting point to getting to what the real goal is. So if you ask why are we doing that? Well, maybe the maybe the why is, you know, just because, you know, somebody said, we should have a CRM system, and we don't. But then you should ask the question, what would a CRM system do for us? And it might enable us to better answer customers questions. Or it might enable us to be, you know, have a better dialogue with the customer about certain things. So so maybe there is a valid reasoning for doing that. But you have to get yet to keep asking why until you get to, what are we really going to achieve? And the outcome is usually stated in terms of how is this going to benefit the customer. And, and so there's this notion of the satisfaction gap between the customer's current experience and their desired experience. And so an outcome would would improving an outcome would close that gap. And so that those are what goals really, we think, need to be focused on. So that using that word, why, you know, whenever you get a goal, ask why, you know, if it's not focused on closing a satisfaction gap, you should keep asking why. Until you get there, and then reframe the goal in terms of well, you know, maybe one of the activities we need to do is implement the CRM system. But that isn't the goal. That's just a means to an end. And the real goal is this other thing that we we now need to figure out how do we measure that there was another point that we've sort of been touching on? And, you know, you could ask the question of why is, why is this so hard? Why is shifting to an outcome? perspective so hard? And one of the things that we found, and this comes out in a lot of our case studies in the book is that people's ego, people's egos get in the way. They don't like to look bad. So you know, some executives got some grand idea. Like, again, I'll use the example we need to implement a CRM system. And first of all, it's hard to challenge them or even ask why. Because then you get treated like you're an idiot life or thirsty, and everybody knows, we said, the CRM system. But you know, sometimes, when you know these, again, it's often an executive or a manager, but not always. But they've got this idea about about something the organization needs to do. And if you do that thing, and you get a measure back that says that didn't achieve any anything, it didn't improve customer outcomes, it didn't improve the product uptake, it didn't, it didn't do the even do the things that we wanted to do, then the typical thing for the organization to do is essentially to bury that, you know, back to the, you know, you trip over the truth, and you go on, well, you know, you bury the thing under the rug, because it makes somebody look bad. And the fundamental change for an organization to get to be to be more evidence based. And the way they manage themselves is to basically put ego to the side. And treat all this data is just interesting. You know, one of my former managers had a great phrase and said, facts are friendly, you know, Doc's are your friend, they will tell you things that you may not want to hear that you need to know. And so an organization that can get to that point and say instead of like, well, you know, who's responsible for this, or you know, that, you know, the release didn't produce the result that you want it and typically, an organization will go around, look for somebody to blame, it's the product owners fault, or it's the delivery team's fault, or somebody didn't implement this, right? Put all that to the side, just say, Yeah, that's interesting. So we found out we had a theory that this doing this would produce this result, it didn't produce that result. Interesting. Maybe we shouldn't do more of that. But did we get any information about stuff that we should do? And you form new hypotheses and you treat each release as kind of a bunch of experiments and gather data? So I think that getting putting the ego to the side and asking why to me are the two fundamental skills to learn in this

Todd Miller: 24:44
occurred? i It makes me think how many times we hear that and I love the idea behind why because you can really get to the bottom of it. And we'll hear from a lot of people and a lot of organizations but we're building stuff internally for people. Like we're building the At analytics engine for the actuarial group, for example, we don't. So that's they're our customer. And we're like, why do they need it? Why are you building so you've got to take it further, you've got to understand the real end root of why you're doing something. And I think, on top of that ego is the way that organizations tend to be structured in these like domain specific areas. So we build it for let's say, you know, an insurance company, we build it for the actuarial department who goes out and they maybe take it to sales, who talks to the customers, we're built in this awkward way where there's people that are on the downstream of all of that have no idea what they're doing and how that impacts? That's what we have to ask why. Right? We have to understand if we're building a data analytics engine is going here. And why keep asking why the whole way until it's because we want to get customers information faster. So they can make decisions faster, right? Because that could be the very root of a keep asking why find that? And to your point, Kurt, you're stepping over a lot of egos to have to do that. Right. But we need to continue to do that. And that's a powerful thing I think that anybody listening to this can do is what your current initiatives, why are you doing it? And if it is, it's to improve operational efficiency, right? Why are you trying to become more operationally efficient? But keep asking why? What impact is there going to be for the customer? If you're able to give a quote faster? What's that mean to them? Why? Right? So I just thought I'd pull that up, again, because of how important that is, and how we define our goals in a way that is the end root of the why, right, the very beginning of the plant, rather than maybe the the top of it, we need to know that.

Patricia Kong: 26:47
Let me let me take that in a different way. Obviously, I agree with you. The the notion is, though is that you can get in trouble asking why right? You want to know why I want to know why I want to know why. And someone's gonna say you don't need to know why that's not part of your job. Right. And I think that kind of ties back into what Ryan was talking about earlier is that people have choices, companies are going to come and go. And if you want to, if you want to be able to pivot, as we were talking about, and you want to sustain it, certainly something to think about that you don't need. Is it inappropriate to say a bunch of monkeys, and that's okay. But anyways, the last chapter of this book, and we'll kind of round this out. So the last chapter of this book, for me was the hardest. And it was about applying. So we talked about, again, the theory, we talk about the reasons we talked about dashboards, we give different examples. And then we talk about, okay, what does this mean, if you want to apply this at the organization? And I remember, Kurt, we were having this conversation, we were like, Wait, we just set it all? So we set it all, and especially that notion of you talked about ego, we talked about empowerment, maybe different ways that companies can look to structure themselves? Is that good or bad? We're not sure. Is there anything that you you guys think that we we didn't touch upon? That if we went went back? Is there anything that we should we should add?

Unknown: 28:17
I think, and I, we might have touched on this, it might just be subtle. But I really think this is one of those where if the leaders believe if there's like this sense of leadership is going to be empirically driven, or evidence based driven. This says a great shot of working. Right? So if you have a CIO, a CEO, a executive vice president, who's willing to say, Wow, that first draft of the idea didn't quite work, we got some good data here, let's pivot, then it's safe for everybody to do it. Right. That means that I, as a middle manager can say, Wow, we got some good data here that says it's time to switch things up or to try something different. But if you don't have that, and I and maybe we could have emphasized this a bit more that almost evidence based, managing management, evidence based leadership, data driven leader, whatever you want to call it, that that really could be the linchpin that decides whether or not these ideas can take root because if it's not okay to be wrong, if it's not okay to revise an idea if it's not okay to pivot, this data doesn't help you.

Patricia Kong: 29:34
I I like the notion of evidence based leadership so much because the opposite of that is gut based leadership. And that only lasts you so long. The gamble. What am I like that's

Ryan Ripley: 29:47
not that smart. So it's

Kurt Bittner: 29:52
one of the things that I find triggers me a little bit is in organizations when they're talking about how they're going to do, their transformation is when there's a lot of talk about. In a sense, kind of soft, squishy things, you know, we're we need to change the mindset of the organization, we need to change the values of the organization, we need to change the, you know, all of this stuff. And we need to do that all first before we can do anything else. And the problem is, is that we are all, in a sense, empirical beings, you know, we respond, you know, when we do something, you know, when when you're a kid, and you put your hand on a hot thing, and you burn it, or you heard it, you learn not to do that again. So we're all we are all driven by experience. And so my feeling is that you just need to get started, you know, pick a spot where you need to improve for him better measures, and

Patricia Kong: 31:04
on page book curve.

Kurt Bittner: 31:07
But, but, but the hard thing is that you have to put all sorts of examples behind that, because it's not. But I think the hardest thing to do, is to get started, and that you can start, you know, at a team level, you could, you could start, you know, it's harder to do top down. But eventually, you have to solve the top down part of the problem, because the organization's strategic goals have to be aligned with what the people on the teams are doing. But you know, in a sense, it's like, you just have to start somewhere and start chipping away at the problem. So and start getting that feedback loop going. I mean, that's the most important thing. And, you know, to do that, you've got to not, you have to embrace the feedback when it comes back, not ignore it and say, well, this doesn't conform to what I expected. So I'm going to reject it. So that's, for me, the thing is, is really sort of just get started. And we talked about in the last chapter about how, how to do that. But I think that, you know, there are lots of different situations, you know, people, like people you talk to at the conference. Last week, you know, they they want to get started, but they don't feel empowered by the organization or being told by people who they work for that, you know, you have to do things this way. And so I think you have to start, in a sense in the things that you can control, and then gradually show results towards better results and get people to open up and be be more open to accepting. But anyway, I'm rambling on a bit. And I know I can see Todd deeply thinking about something here.

Todd Miller: 32:52
Well, what I was, I was thinking about what we missed, is something that I think is a book in and of itself. And that is organizational structure. And we talked about that quite a bit and a lot of what we talked about in this podcast as relates to that, and I think I tangent a little bit about that. So that is not directly addressed in the book, because of the density of the topic. I think that you could apply evidence based management, I know you can apply to evidence based management at the organizational level to understand whether your structure is working or not. So there's that and every organization structure is different. But that's one thing. I think that is a book in and of itself that we don't directly address, because I think his original question that you asked Patricia, and I don't regret not addressing that, because of the density, the topic, right. There's that there's many books on that. There's a lot of thoughts on that. I think that that could be something that would be discussed.

Patricia Kong: 33:59
Yeah. I think for me, it's really about some of the programs and structures and things that we see and have going on. And let's say the Agile and Scrum industries, I think are doing harm to the focus on value. And maybe how do we call crap on that? And how do we think about those things? So maybe just removing and thinking about what is just keeping us busy there because there's a lot about that. That's a separate podcast. But thank you, everyone for joining us today for our conversation with Kurt Bittner from scrum.org, my colleague and also with Todd Miller and Ryan Ripley from agile for humans. I really appreciate it and I hope it helped everyone learn a little bit more about EBM and also the new book unlocking business agility with evidence based management