One of my favorite sayings is “Inspection without adaptation is waste”. For continuous improvement it is important not only to identify things that did not go so well but to also change something in the way you are working. If you change nothing, the inspection did not have an impact at all and is a waste of time and energy. But change is hard. To make changes not only easier and more likely to happen but also to drive change in an evidence-based manner, experiments are a great opportunity. But how can you establish a culture of continual experimentation and learning? What could you do to make more change happening in your workplace?
Why change is so hard
Everyone agrees that adapting and change is crucial to survive in the long run. But on the other hand, many teams are struggling to appreciate change in their daily work. Sometimes there is this opinion that resistance to change is part of human nature. I think this is oversimplifying the problem and the resistance for change is mostly caused by the system people are working within (work environment).
Change is usually hard because we want to get it right. With each change also comes a risk, that we are worsening the situation. To avoid this kind of deterioration of the status quo, it is natural to think about all the different possible unwanted consequences a change can also have. And this is not a bad thing per se. It only becomes a problem if the investigation and even more discussion of these consequences take up too much time. This not only slows down our ability to change quickly and frequently. Often you also see teams discussing changes in length without getting to an agreement. So, people try to avoid this hassle and even stop proposing changes as they want to avoid the time necessary for this kind of debate.
Experiments to the rescue
On the other hand, many of these concerns are just hypothetical. We do not know yet, if they will be relevant. So while we do not want to be blindsided with what happens after a change, this just shows that we do not know enough yet to come up with the “right” decision.
The purpose of an experiment is to increase our knowledge about a subject, to uncover “unknown unknowns” and to add more confidence to our decision-making process. So instead of spending much time on discussing a subject and then still just have some hypothetical assumptions, one could better spend some time to run an experiment and create real evidence. Unfortunately, many teams struggle with implementing experiments. To successfully use experiments in your learning and improvement efforts, a specific culture is needed. This culture is also part of the third way of the three ways of DevOps (https://itrevolution.com/the-three-ways-principles-underpinning-devops/) which is named “Culture of continual experimentation and learning”. But how can we create such a culture? The following tips might help to give some answers to this question.
Tip 1: Make experiments small
Remember: an experiment should be an opportunity to learn, validate assumptions and create new insights. The smaller an experiment is, the faster we can start and execute it, and will not only increase the likelihood to reach an agreement on conducting the experiment as smallness also reduces risk. If needed, we can conduct a series of experiments to cover a bigger topic. But after each experiment we can course-correct and decide which experiment will create the biggest learnings based on what we have learned so far.
Small refers to different factors like:
- Number of people affected
- Number of people involved
- Time to run the experiment and gather results
- Effort needed to conduct the experiment
- Risk of creating unwanted effects and recover from them
The smaller your experiments are, the more you can perform and the more frequently you can learn and use these learnings to shape the next steps. So, if you get some resistance to conduct an experiment, think how you can make it smaller.
Tip 2: Value learning and insights
To encourage people inside a team or an organization to conduct experiments, learning and new insights must be seen as something valuable. Performing an experiment means that some of our capacity must be shifted from feature work. If people feel – vocalized or not - that feature work is perceived as more valuable than learning, they will focus their efforts on feature work and experiments will be postponed to “when we have time” aka never.
To establish a culture of continual experimentation and learning we have to create a space for this. Some teams reserve a fixed timebox in each Sprint. This is an option that can work as a starting point but at the same time it comes with limited flexibility. Preferably a shared understanding about priorities should be established. And these priorities should never be absolute for one area or another. Experiments should not always take priority over feature work or vice versa. We might realize that we had a time where we focused too much on feature work and neglected our improvements for a while and will shift focus. Or there might be a high business demand which requires focus for a short period of time on feature work. The goal should be to balance the two over time.
But this balance can only be achieved if people feel they are equally important. And that is much related to the feedback we give to people. If a colleague, a manager, or some other person gives you positive feedback about an experiment you ran, you will be motivated to do another one. If you get positive feedback only on implementing features very fast, you will focus on getting features done as quickly as possible. The feedback we give shapes the work people are focusing on.
Tip 3: Make experiments explicit and transparent
To show the appreciation of running experiments, we must make them explicit and transparent. Experiments should be defined as such up-front and should not be misused to just fiddle around with a subject for some time. Experiments should be discussed and agreed on within the team, even if they are conducted by just a single person. This will create more opportunities to break the experiment smaller, to shape the goal clearer and to think about other options to run this experiment.
The results should also be shared within the team and probably even beyond to also share the learnings from it. Experiments should be timeboxed. There should be frequent inspection about the results we got so far and a time where we have to decide if we already have learned enough and there are other opportunities that would provide more insights or if we think, there is still value in extending the experiment. Some teams use an experiment board where they visualize all planned and ongoing experiments as well as the finished ones alongside with their results.
Tip 4: Roll back by default
As we discussed earlier, resistance is mainly driven by the fear that a change will deteriorate the situation. People are afraid, that if the change is in place, it will be hard to change it back if they do not like the new situation personally. That often results in long arguments where people try to advocate against the change.
You can cut these arguments short if you agree that the default mode will be a “roll back” after the experiment ends. That means, that we go back to the previous mode of operation as it was before the experiment started. The only way to keep the new way will then be to agree that the new mode has enough advantages, and we want to keep it. This way we can make people more open to just give it a try and see what will happen.
Tip 5: Begin with the end in mind
Each experiment should not only be clearly defined as such, but there should also be a goal defined which we want to reach with this experiment. What is the hypothesis we want to test? What are success criteria for the experiment? What are we hoping, we can learn from it? Discuss, if you can achieve this goal in one single step or if it would make sense to break it into smaller steps (see also Tip 1: Make experiments small). If you have multiple things you want to learn, maybe you can have smaller experiments focusing on one of the learnings? If you have different aspects in your hypothesis, you can break it into smaller ones.
Only if you have a clear understanding of the expected outcome, you should shape your experiment. Even if the expected outcome is not very concrete, you can now discuss what will help you to best achieve the goal. Some teams have very clear success criteria, sometimes even specific metrics that define the outcome of the experiment. Some have just a learning objective in mind, a deeper insight into a hypothesis or topic. This might depend on the people, on the situation and the question at hand. Probably a team should experiment with different ways of defining the goal of an experiment.
Tip 6: Create psychological safety
The biggest impact on a culture of continual experimentation and learning is psychological safety. Only if people feel save, if they trust that they will not be thrown under the bus by their colleagues and superiors, they are willing to take risks. If they know, they will get help when asking for it, they will be more confident in trying out new things. And the willingness to take risks is a precondition to change.
This starts with understanding that an experiment cannot fail by not delivering the expected results. Every experiment can deliver value in the form of new insights and learnings, even if the experiment was not suitable to answer the question at hand and we learn how to shape a different experiment. The only way an experiment can fail is if we do not learn something from it. And this is very much related to our own openness and not about the outcomes of an experiment.
That should not be interpreted as ignoring the risks that come with an experiment. It means we should minimize the risks with some practices that have already been discussed in this article combined with organizational and technical abilities to handle risk. Continuously monitor the progress of your experiments and adapt early. Challenge others to do so while at the same time being supportive and provide to the safety they need to succeed. Focus on deriving as much learning as possible from your experiments and not complain about poor execution or unwanted results. Inspire others to be open-minded by giving a good example yourself. And never judge people and their behavior or motives but the situation and the environment.
Tip 7: Kaizen mindset
To be successful with experiments, a deep desire to continuously improving the product as well as the current way of working is important. In Lean this desire is called “Kaizen”. Kaizen is driven by the eagerness in people to be proud of their work. The question is, how do we constrain this eagerness. Is it constrained to small details people can change in their work or do we open it up wide? Do people feel they can influence the product and the end-to-end workflow or just their personal contribution. If people want to be proud of the product they are building, they will easily identify and propose opportunities to improve it. And these is the best source for defining experiments.
It makes a huge difference if one sees an experiment as a step to improve our product or if the experiment is defined by someone else and execution of the experiment is demanded. If people want to improve, they find ways to do so. If we limit them in their options to do so, we will decrease motivation at work. Fostering a Kaizen mindset is one of the most noble obligation not only managers but also team members can serve. To do so, one first must grow their own Kaizen.
Example
There is a suggestion that investing in automated UI test can help to improve the quality of the product. There are several concerns that these tests are very labor intense not only in creating them but also in maintaining and updating them. And there are some doubts, that these tests will find most of the issues anyway. The hypothesis that should be tested is “Automated UI test help to reduce the number of issues in production with a reasonable amount of effort in creating and maintaining them”.
Instead of building a complete automated test harness, the team decides to break this down into a series of small experiments to reduce the risk. The first one will focus on learning more about how much time creation and maintenance of such tests will take. The proposal is to automate one first test case and then discuss the insights. But then someone suggests breaking this down even more. It could be separated in one experiment for the creation effort and one for the maintenance. So the first experiment would only focus on creating the test case and run it locally on a development environment. Integrating it in an automated build and see how much update it needs will then be a second step. Part of the first step will be to pick a tool for the UI testing but without spending a lot of time there. There will be follow-up experiments that can validate the hypothesis that there are tools available that can make it faster and easier to create the automated tests.
The team taped already some important learnings out of the first two experiments. They realized, that having UI tests in an area where the UI changes frequently need a lot of updating. As a conclusion out of this, they decide to create UI tests only for areas where the UI is already stable. That brings up the need to combine the UI tests with another test method for the areas that are currently under development. They agree to try using exploratory testing for this situation. But that also brings up the question on how to trigger the discovery of UI areas that became stable enough to do UI tests there. The next experiment will be to plan a recurring meeting where the team together with the Product Owner discusses this.
A team member suggests finding a tool to see which areas of the UI are already covered by tests to better identify areas where UI tests would make sense. The team agrees to timebox the search and evaluation of such a tool to 4 hours and if they cannot come up with an appropriate tool, first create the map by hand to see if it creates enough value to spend more time for finding a tool to automate the process.
Imaging how this journey might be different compared to the team trying to come to a decision about whether they should use UI tests or not in their first discussion.
Conclusion
Improvement is a complex challenge. Therefore, it is impossible to identify all cause-effect-relationships upfront. The best approach to handle complex situations is to make use of empiricism – Transparency, Inspection, Adaptation. The best approach to develop a culture of continuous experimentation and learning is by using experiments. First create some commitment about accelerating your improvement process. Then try to find a small experiment. Ask people to hold back their reservations for now and just discuss if there is the risk of significant damage from this experiment. Remember, there will be a roll back as the default. Running this first experiment can then help to build some trust in the process which should allow initiating the next experiment and over time trust will grow and you will step by step grow into this culture of continuous experimentation and learning.
If you achieve that, this might be one of the most important powers of your team for a sustainable future as this defines the ability to adapt quickly to new conditions and it will put you in a competitive advantage over other organizations.
Happy experimenting!
Afterword
And how does this now relate to Agility and even more specific to Scrum? Well, Business Agility is exactly about that. It is based on the assumption that everything we want to build or change in our product first is just a hypothesis, at best an educated guess. But we do not know for certain, which features will improve customer happiness most and which surprises will just wait around the corner. To validate our hypotheses and assumptions as quickly as possible to learn, we can use experiments.
Agile and Scrum is based on Empiricism - transparency, inspection, and adaption. Instead of just breaking a big plan into smaller junks, agility is about learning with each step and use these insights to shape the next steps. In Scrum you can see each PBI as an experiment as well as each improvement item from a retrospective. The culture of continual experimentation and learning can help greatly to do Agile and Scrum right.