People in organizations are often afraid of running experiments. I have observed that approximately 70% of people responsible and accountable for products are reluctant to validate their assumptions by experimenting. There are several reasons for this hesitancy. The most common is misunderstanding what exactly the Agile experiment means. The second one is the negative result of the experiment. I always answer that this is fine that we have such a result! It means that there is no need to develop a solution that no one wants or does not fulfill users' expectations.
Users & Customers
To understand the nature of experiments, the first thing is to know the users. Interestingly, still the majority of Scrum Teams know nothing or little about their users.
I would recommend applying a proto-persona and validation. Questions that can be asked:
- Who is our user/customer?
- What problems are we trying to solve for our users/customers? What do they need?
- What would the outcome be for our users/customers?
- What would the impact be on our product?
Outcomes not outputs
We need to differentiate outcomes from outputs. Outputs are results of our activities like a number of features/solutions, bugs, story points. These measures do not indicate about the value delivered. Organizations have been mainly measuring outputs. At least, delivery departments.
Outcomes are measurable changes in user/customer behavior. These are the results of outputs. Examples of outcomes: conversion, churn, product usage, customer usage, customer/user satisfaction, etc.
Measuring value
The topic of outcomes is related to measuring value. Without measuring value, how do we know that we actually delivered any outcomes? The most useful framework to measure value delivered and desired outcomes is called Evidence-Based Management. The framework provides you an opportunity to measure value in four different aspects – Key Value Areas. More information can be found on Scrum.org and my website.
Running experiments
The shortest answer to the question "why experiment?" is to validate your assumptions. Again, the majority of organizations make assumptions about the customers/users' usage, solving their problems or fulfilling their needs without any single trace of evidence. We assume that our proposed features, services, solutions, the product will work! In other companies, especially those who do not measure value, there is even no conversation about customers/users' needs. Lack of measuring value. This is a vicious cycle.
Spreading the agile mindset that encourages frequent inspection and adaptation is the first step. It will help to understand the need for experimentation. I am often asked how to convince business and product people to allow Scrum Teams to run an experiment. Use these questions/statements below:
- How do we know if this solution will deliver value for customers? If the answer is vague, continue.
- The experiment we mean is short and low-cost. We do not need to spend Sprints for validation. Share examples of quick, low-cost experiments: fake feature, paper/digital prototype, observations, user interviews, landing page.
- Calculate. By short experiments, I mean hours, not months.
Example:
Let’s compare:
One day experiment (provide costs) => immediate data about the value or in a short period of time.
or
Delivering the entire solution in 3 months (provide costs) => data after 3 months or later about the value.
Is it worth risking and investing such an amount of money without any validation?
- Write hypotheses. Combine them with a business problem statement, expected outcomes, user benefits and measures.
- What if the experiment brings a negative result? This is fantastic news! We do not need to spend 3 months (replace by your estimation), people effort and company money to deliver something that is not valuable.
- Who will be the subject of the experiment? Ideally, real users/customers. When you do not have direct access to them, try to find people in your environment who are users (or potential users of the solution) and match your personas. You can ask your colleagues who are not involved in the product delivery.
The next step is to design an experiment. You can combine a few Lean UX practices. Example: interviews + observations + prototypes. Useful information about Lean UX can be found here.
Ideally, you want UX people involved in this process. If you work with Scrum, a UX person should be a part of the Scrum Team. The entire Scrum Team might run experiments.
Conclusion
It is worth investing a few hours or days in validating our assumption on real users to prove if you are right or wrong with your solution. Delivering the entire solution without any validation is costly and unnecessarily risky. Think in terms of outcomes, not outputs. Involve the whole team to help you run experiments. Measure value with Evidence-Based Management and make decisions based on facts.
If you’d like to experience and know more about Scrum and experimentation, the Professional Scrum with UX class is all about this topic.