Empiricism - what is evidence?

https://medium.com/agileinsider/the-power-of-a-b-testing-in-product-development-2a783777b66

Yes, you are right. I know the concept :) So maybe PO was not the best example. The intention of my question was broather, about empiricism in general.

Let's take another example - Sprint Retrospection. The conception of feedback loops requires someone to analyze the information / inspect it. Without some rules of proceeding like scientific method people may look for the proofs that support their current believes. Do we really expect the Scrum Team members to be qualified inspectors aware of that? How can we teach it?

I wander what are your thoughts.

Do we really expect the Scrum Team members to be qualified inspectors aware of that?

Yes. The competency of being able to inspect and adapt, collaboratively, is important to Scrum professionalism.

How can we teach it?

A good Scrum Master manages people's understanding of how to best achieve empirical process control using Scrum. A good approach is to reveal rather than to resolve, and to wonder about the things you see.

The Scrum Values help empiricism, and we can revert to them even when applying the elements of the framework proves hard.

What an interesting question!

My first thought was to think of the difference between a medical trial and how a Sprint Team works, and the complexity of what is being tested.

A medical pill vs. a dynamic working environment.

The pill is inherently simpler. Each one has the same, strictly controlled, standard process and conditions for manufacture, storage, etc, whereas a team involves humans who get sick, go on holiday, work on a different thing each day, get emotional, have distinct personal and professional obligations, laugh and cry about different things etc.

Then I thought about what a pill does. It goes into a complex system. Human bodies are complex, each one acts differently, depending on a whole host of factors, both known and unknown. But the overall change may well be observable, and that's what medical trials rely on.

So perhaps a more suitable comparison to a medical trial would be a researcher studying a large number of teams or organizations.

The book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim comes to mind of a scientific approach being adopted. How well it holds up in comparison to the standards of a medical trial, I'm not qualified to say.

I suspect not as rigorous, but it also doesn't need to be.

 

But when focusing on a single Scrum Team or organization, such methods aren't so easily applicable. We do rely to an extent on good behaviour, and the Scrum Values can help with overcoming and addressing conscious and unconscious biases in ourselves and others.

I have experienced plenty of situations where individuals do indeed misuse, manipulate or perhaps just shun evidence in order to support what they believe (or worse, what they don't believe, but they know will suit them personally).

As a Scrum Master, I enjoy the challenge of trying to overcome this. Sometimes I find it frustrating and demotivating. In the past, it has been significant enough for me to look for another job.

If the behaviour within an organization and team allows it, the best method I have found is to make upfront hypotheses about what might happen. The environment must be safe enough that being wrong is OK.

Then as a team, we can proceed, looking for ways to find out whether that hypothesis is correct (not the scientific method of trying to disprove a hypothesis).

We can plan our work to maximize learning. We can talk about our learnings at the Sprint Review.

Because our hypothesis has already been made explicit, as a Scrum Team we can later inspect whether that hypothesis was correct, and based on what has been learned, further hypotheses can be introduced.

That method introduces certain kinds of necessary waste, such as time taken to identify and validate hypotheses, but is likely to eliminate an even greater amount of waste in terms of product complexity, poor user experience, missed opportunities, etc.

The method should never be so burdensome that it adds more waste than it removes.

A good approach is to reveal rather than to resolve, and to wonder about the things you see.

Fully agree on that Ian Mitchell. That's why I just wander about empiricism and a proper way of conducting experiments.

To deal with complexity we formulate a hypothesis, plan an experiment, perform an action, study the results and continue with another action. A/B testing is one of the approaches. But what if it is not the product that is inspected but the process, tools, interactions or DoD? For Scrum Retrospection:

The Scrum Team inspects how the last Sprint went with regards to individuals, interactions, processes, tools, and their Definition of Done. Inspected elements often vary with the domain of work. Assumptions that led them astray are identified and their origins explored.

I might be wrong but I feel the correct answer has to do with probability. We do not use a scientific method in business. We are not looking for an unquestionably correct answer, but an answer that will improve our results even slightly. Perhaps we should therefore clearly indicate that we accept the risk of misinterpretation of the results and the risk of cognitive bias should be clearly stated?

But when focusing on a single Scrum Team or organization, such methods aren't so easily applicable. 

I share your view Simon Mayer

The Scrum Values help empiricism, and we can revert to them even when applying the elements of the framework proves hard.

I can't agree more Ian Mitchell

I might be wrong but I feel the correct answer has to do with probability

Certainty is not ours to provide. The belief that certainty is somehow owed is a conceit of predictive thinking inherited from a waterfall culture and its hierarchies. In such a culture, when certainty proves elusive, blame is more likely to be evidenced than value.

In Scrum we offer the means of managing uncertainty, through a collaborative and emergent understanding of probabilities and forecasts. An empirical approach seeks to be predictable without being predictive.

I wonder if anyone has already described how to conduct experiments in such a way that their results are reliable?

My opinion is no one that uses empiricism has done this.  Rationalist might have but they would rely on innate knowledge for their description.  By definition an experiment is going to be unique because you are trying to discover something you don't know, test a hypothesis, or demonstrate a known fact.  Of those, only demonstrating a known fact has a predetermined result.  So how could you describe a way to do an experiment where the unknown results are reliable?  

I can easily imagine a Product Owner confusing correlation or accident with a cause-effect relationship after conducting a single experiment.

This is why you never do a single experiment. How many times do you think that a pharmaceutical company only conducts a single experiment before moving forward with producing and selling a drug?  In reality, every time an individual participating in a case study takes the drug it is a unique experiment. 

Without some rules of proceeding like scientific method people may look for the proofs that support their current believes. Do we really expect the Scrum Team members to be qualified inspectors aware of that? How can we teach it?

The rules of scientific methods are usually created uniquely for each experiment.  That is why with any scientific paper published they will include their parameters, processes and anticipated results.  Standard practices in software development will describe a problem that needs to be solved and some way of understanding if that problem has been solved.  To me, that is providing parameters for the experiment so that the results can be determined successful or not. Is that reliable?  Probably not but that is why empiricism is used.  You make decisions based on current knowledge.  

Do we expect that the Scrum Team members to be qualified inspectors aware of that?  Absolutely yes.  Empiricism is based on knowledge that is gained as a result of experience. As such the Scrum Team working on a problem space is going to be more qualified to inspect the results than anyone else.  Will they be aware that results support their current beliefs?  I hope they are because those current beliefs are the expected results. And as professionals they should be capable of understanding when their expectations are not in line with reality.

I feel that a lot of your suggestions are possible options.  But there is not going to be a one size fits all solution.  Instead all of these great ideas should be included in a "toolbox" that can be used by people.