Do or do not include UAT within the sprint

Can these users be included in the Review? This would be the best place to collect feedback from them for the next sprint planning. But this would not solve the problem that there is obviously a difference in how the Scrum Team understands the product backlog items, and how the end-users understand them. It is probably necessary for the PO to work with the stakeholders more intensively, so he can adjust their expectations and the Scrum team's understanding.

A Product Owner must be in a position to accept an increment and to determine whether or not it is releasable. Nobody else (not even end users) has these responsibilities in Scrum.

What does your PO think about the position he or she is in, and how the product ownership role can be better fulfilled?

> These users can be everyone within the organisation and most of them
> can't be included in the Development team due to their primary job.

That's fair enough, one would not typically expect end users to be developers. It would be more reasonable to expect the PO to be able to represent the interests of these and other stakeholders, and to ensure that these interests are reflected in the understanding of Done. What does the PO think about about how this representation might be improved?

Very common situation that I observe in companies all the time. That just means your DoD is not complete. If for potentially releasable increment you need a UAT it should be included in DoD.
How this can be done? It's a matter of discussion with PO first of all.

The sprint review process is to get feedback from the stakeholders on the potential increment. This is where the UAT will find its suitable place.

1) How long can the stakeholder wait without the potential release ?
2) How much authority the stakeholder has given to the PO with respect to UAT(PO cannot be an expert in legal, aerospace and medical industry at the same time).
3) What is the nature of the product ? Is UAT necessary to consider the potential product as shippable ? If No, then PO can collect their feedback on the released product and add it to the PBL ?
4) Will the organisation be able to accept the sprint review as UAT ? Some stakeholders will not agree with the possibility to timebox anything.
5) What is the nature of the UAT itself ? Some system might take 4 days to test a full flow and approve it. In this case the release planning will be done on a possible staging server , where the PO will be responsible to get the feedback into the PBL. Sprint review can still be useful to decide what is done and what can be potentially released.

Posted By Ian Mitchell on 25 Jun 2014 05:06 AM
A Product Owner must be in a position to accept an increment and to determine whether or not it is releasable. Nobody else (not even end users) has these responsibilities in Scrum.

The PO can "accept" the increments, but not approve it. Simply because the PO can't represent all the end-users of the product. The product has many different modules and each module represents a department with users specialized in this module.

What does your PO think about the position he or she is in, and how the product ownership role can be better fulfilled?

The PO understands that he can't oversee every module because he is not an overall specialised. This application is that big that there are specialists for every module.

Posted By Illya Pavlichenko on 25 Jun 2014 06:49 AM
Very common situation that I observe in companies all the time. That just means your DoD is not complete. If for potentially releasable increment you need a UAT it should be included in DoD.
How this can be done? It's a matter of discussion with PO first of all.

It is not included in the DOD because it is not possible. If we include the UAT part in the DoD then this would mean that we'll need to include end-users within the team which is not possible.

My client is building a big highly complex application which consists of many modules. Every module represents a department with specialized users on board. The PO can't oversee/ approve every item simply because he is not an overall specialist. (no one is)

Hello P.Ross,

I would recommend to do UAT (executed by the User) at the end of a Sprint but not during the Sprint.
It doesn't matter if you do Scrum or not, if you gather Feedback during the development phase it will lead to never ending stories.
Furthermore UATs should not be included in the DOD:
1. The user should only review what is done (Transparency)
2. Only the PO should decide what the product should look like. He has to find the best compromise for all the different user concerns.

When the Sprint starts all the PBIs should be clear enough to start an implementation. If there are wrong decisions made by the PO this can be fixed in the next Sprint, but for the time the requirements are set.
Even if the product is not of the value to the user as expected, it should be in a deliverable state. This does not mean that it has to be devlivered, neither that it has value to the enduser.

So regarding your concerns, if the product is tested (except the UAT) and the PO has accepted it, it is shippable!
It is done!

Find a process to handle the upcomming Feedback from the UATs and endusers (e.g. Incedent Management). Integrate this Feedback in the PB.

Kind regards

Benjamin

It's simple. You cannot change Scrum because something is "not possible" or "extrememly hard to do" in your company. On the contrary - YOUR ORGANIZATION changes to fit Scrum and its rules.
Scrum Guide clearly states that by the end of the Sprint the increment should be potentially shippable. If to 'ship' means UAT for your company then complete DoD would certainly include it.

I'm afraid Illya is right. If the PO "can't" fulfil one or more of his responsibilities, or if the DoD "can't" be improved for some reason, then this is evidence that the organization has some distance to go.

In those cases where attempts are made to change Scrum rather than the organization, the usual result, unfortunately, is very little change.

Posted By Illya Pavlichenko on 26 Jun 2014 10:43 AM
It's simple. You cannot change Scrum because something is "not possible" or "extrememly hard to do" in your company. On the contrary - YOUR ORGANIZATION changes to fit Scrum and its rules.

So if the company is telling you that it's just not possible to include the UAT within the Sprint. I bet this happens a lot. So what would you do?

Lecturing that it's stated in the Scrum guide is not going to help this case...

Posted By Ian Mitchell on 26 Jun 2014 11:34 AM
I'm afraid Illya is right. If the PO "can't" fulfil one or more of his responsibilities, or if the DoD "can't" be improved for some reason, then this is evidence that the organization has some distance to go.

In those cases where attempts are made to change Scrum rather than the organization, the usual result, unfortunately, is very little change.

I agree, but why would you do if your are in company where this (including UAT in the DOD) is just not supported. Doing UAT after the Sprint before it can go live is a mandatory step

What would you do/say to convince them

What I would do is to calculate, or at least estimate, the amount of waste being incurred through rework. It is the cost to the company of not having an adequate DoD or product ownership, and the subsequent need to repay technical debt.

You may want to check out this recent thread:

https://www.scrum.org/Forums/aft/870

So if the UAT takes five weeks to come back and is then outside of the sprint timeframe - DoD can then never be achieved?

That scenario doesn't make much sense.

Surely the UAT is entirely dependent on common sense, what is workable within your specific infrastructure and your environment.

If I were to rely on my customers to sign off on a UAT for the DoD I'd never release anything. If I thought I could reliably include it I would.

Isn't the point to produce something that CAN be released? If the User then performs an acceptance test and doesn't like it, then they don't accept that release, then the reasons why need to be understood and approached in the next increment. It's up to the product owner to ensure that this acceptance happens more often than not, as it's his/her job to make sure the users needs are met. If the ratio of rework gets too high, then your facing some serious issues.

I'm thinking that, our good friends Microsoft released Vista as a desktop environment and everyone pretty much hated it. The next increment was Windows 7, which was to some degree slightly better. :-) The point is people chose not to go live with Windows Vista and they can do the same with your increment.

Rob, if UAT takes 5 weeks - imagine how much waste they have! It's all about bureaucracy procedures and lack of agility in a company. The question every Scrum Master decides on his/her own is - would I challenge the status quo? Good Scrum implementation always causes organizational changes. It's not about "they do that dev. thing"

if UAT is taking 5 weeks , my gut feeling is , consultancy is involved. They are delivering once in 3 months with "stage boundaries" . this is also lack of Continuous XXX

Scrum master has a job to do.

> Surely the UAT is entirely dependent on common sense, what is workable
> within your specific infrastructure and your environment.

Scrum implementation is hard. It can involve changing an organization's infrastructure and environment, as well as challenging its received wisdom and cultural norms. These have an unfortunate tendency to masquerade as "common sense".

> Good Scrum implementation always causes organizational changes

Correct. Moreover these changes should be deep and pervasive across the enterprise, and persistent enough to overcome organizational gravity.

I'm trying to understand how UAT fit in Scrum. That's a great thread and it caused me to raise a question here:

How UAT are different from gathering data about users behaviour after an increment release, so that UAT must be part of DoD?

In web development you have 3 main sources for product changes: user requests, behavioural analytics data and business goals. Let's assume that business goals are constant.

Aren't UAT just a way to gather data about hypothesis that had driven the development? Therefore it's the matter of risk acceptance level whether to include them in DoD or not. Isn't it?

> How UAT are different from gathering
> data about users behaviour after an
> increment release, so that UAT must be part of DoD?

Scrum makes no prescription about how testing ought to be conducted. There is however an expectation that each Sprint Increment will be fit for potential release. That level of quality is asserted in the Definition of Done.

Any "user" involvement must be seen in the context of either work that is in progress by the Development Team and therefore undone, or work which is productionized and therefore considered Done.

> Aren't UAT just a way to gather data
> about hypothesis that had driven the development?

If UAT in this case refers to Done work that has been released, then yes. A popular term for this is Validated Learning.

If UAT refers to undone work that is in progress by the Development Team, then no. In this case it would be better to say that it gathers data about an hypothesis that *is driving* development (present tense).

> If UAT in this case refers to Done work that has been released, then yes. A popular term for this is Validated Learning.

Yes, I was referring to Done work.

Thanks.

@Pablo,
I find this article by Peter Hundermark very useful for your situation:
http://www.agile42.com/en/blog/2011/02/03/uat-on-a-scrum-team-part-1-of…

"Trying to take a too big step in this direction is unlikely to appeal to either party, so start small. If there is a team doing UAT, continue with this process but ask for a single user to join a team. The output of the team augmented by the user should continue to be tested by the UAT team. Run the exercise for three sprints and compare the quality before and after."

Pablo,

Something is telling me you ARE actually releasing your product at the end of each sprint. Please correct me, if I'm wrong, but from you posts I understand that you:

a) develop a product that is used internally in your organization
b) give it to the end users at the end of each sprint
c) probably "giving" includes a deployment that is an official deployment of a new product release

Please clarify, but if this is correct, then this is no different than just releasing what you have to production and getting feedback from real users.

Piotr

To use some expressions from SCRUM guide here:

"Release" does not mean "release to production". It could also be "release for UAT".

"potentially releasable" is different from "definitely releasable".  

In my understanding, a UAT as described here is a functional validation if the implemented, working functions meet the business needs, like a simulated release to production. It does not have to be included in SCRUM sprints, neither be part of DoD. It does not need to be covered by the SCRUM framework at all. Its just a validation before going live, in order to minimize risks for business, which might be high risks if a product is business critical and complex, as it seems to be in this case. UATs can provide feedback to the product owner like a production release.

Tilmann, I disagree strongly with your assessment that "release" in Scrum can also mean "release to UAT".

Scrum is quite clear that the goal in every sprint is to produce value that is potentially releasable to production.   If UAT is a pre-requisite to going live in prod, then it must be included in the Definition of Done and completed for each item within the sprint.

Do not allow sprints to simply represent different phases of a traditional waterfall plan.

I echo Timothy's considerations.

"potentially releasable" is different from "definitely releasable".  

Also, do not confuse the terms "potentially" and "definitely" in front of releasable. What I mean by that is, the increment produced by the team should be of production quality (per DoD), which means it should be of definite release quality, however the actual deployment to production environment is potential, for the PO (business) may decide to postpone release.

Eugene, Timothy, I cannot find anything in the SCRUM guide saying that sprint results have to be released into production, neither is there anything mentioned about UATs. For example, it is no problem to apply the scrum framework in order to create prototypes as result of a feasibility study, for example. Quaility might be a lower priority in this case. Its a matter of how DoD is defined.

I agree that if creating functions for production environments, the functionality from sprint result should work without errors and as specified. However, you might want to check further if what was specified also matches the business, or the function's intended use. That is how I understand the concept of a UAT.

For example in life critical applications such as medical diagnosis software, or control elements for airplanes, validation tests have to be conducted that mostly would take much longer than a sprint, e.g. some years of field testing. If you would want to have the feedback only from "production" in such a case, this feedback could cost lifes. However, it is possible to benefit from the SCRUM framework also in such situations. Functional validation (in contrast to verification) would just need to be done outside the framework.

Tillman, there is no express obligation to release software to production because, we can reasonably assume, the thinkers didn't want to be rigid/obtuse (and impose a rule of always releasing!). But reflect a bit on these:

• Scrum Teams deliver products iteratively and incrementally, maximizing opportunities for feedback. Incremental deliveries of “Done” product ensure a potentially useful version of working product is always available.

• The Development Team consists of professionals who do the work of delivering a potentially releasable Increment of “Done” product at the end of each Sprint. A “Done” increment is required at the Sprint Review

• Each Sprint may be considered a project with no more than a one-month horizon. Like projects, Sprints are used to accomplish something. Each Sprint has a goal of what is to be built, a design and flexible plan that will guide building it, the work, and the resultant product increment

Also note these:

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Deliver working software frequently, from a  couple of weeks to a couple of months, with a preference to the shorter timescale.
• Working software is the primary measure of progress.

In certain industries/divisions, things are considerably more difficult. And you gave the perfect examples (healthcare, aviation).

I agree that if creating functions for production environments, the functionality from sprint result should work without errors and as specified. However, you might want to check further if what was specified also matches the business, or the function's intended use. That is how I understand the concept of a UAT.

For example in life critical applications such as medical diagnosis software, or control elements for airplanes, validation tests have to be conducted that mostly would take much longer than a sprint, e.g. some years of field testing. If you would want to have the feedback only from "production" in such a case, this feedback could cost lifes. However, it is possible to benefit from the SCRUM framework also in such situations. Functional validation (in contrast to verification) would just need to be done outside the framework.

These are extreme examples because of the impact and potential circumstances. Trials here could last for months, if not years! So while they constitute some form of UAT, they are far more than that.

This is a very interesting conversation. May I suggest that we not try to force an Agile best program practice into the Scrum framework? Here's what I mean...

It is true that the PO must represent the interest of the customer sufficiently to allow a Scrum team to deliver something that is potentially shippable and meets customer needs. 

The PO role was designed to be a proxy for the customer. The customer only enters Scrum directly at the Sprint Review, but it is unrealistic to expect that thousands or millions of customers will, can, or should attend a Sprint Review. So there is a step, after release, where the organization depends on market feedback outside of the Scrum framework. For some companies, that step may involve a limited release, a beta program, phased release, etc. One can argue that UAT fits in that bucket, especially UAT for a complex product or a product where customers insist on doing extensive testing to mitigate risk.

If the organization is being Agile, it will strive to shrink the release cadence and reduce the delay between release and market adoption. That part of the production sequence is outside of Scrum, but it is certainly a legitimate concern of Agile practice. I would call it part of Agile program management. Lean startup and other Lean practices have tools to address it.

So I am arguing that some part of UAT is essentially market adoption. And, though delay in market adoption is not resolvable by the Scrum process, it is an important concern of the product's program if the organization expects to leverage the Scrum process most effectively.

To the anonymous author... have the PO do the most representative job possible, try to expand the audience of the Sprint review to include some customers, and work with your program manager to accelerate market adoption.