Challanges to perform UAT in Scrum

The following is my own practice, for your reference. 

When writing or discussing User Stories, the Product Owner invites people from the business department to participate.

Each User Story will indicate the acceptance criteria and how to demo.

During the development process, if the development team has any questions about the requirements, the Products Owner will invite people from the business department to help clarify them.

Finally, during the Sprint Review, the Product Owner will invite people from the business department to participate, and refer to the requirements description and acceptance criteria on User Stories for functional demonstration.

People in the business department will also provide feedback or request amendments in due course.

The Product Owner will add User Stories as a work item for the next Sprint depending on the situation.

Thanks Ching.

So, the definition of done should include all the test cases that if passed, would mean the product increment is complete (releasable). Sure, the PO is the bridge between business and development team but can he/she alone sign-off the completeness of the increment on behalf of the business for the Sprint? 

Since it would sound unreasonable to propose the actual business users of the product to be involved as part of development team to perform the UAT, would this be performed without them (based on the criteria captured in the acceptance criteria and definition of done)? If yes, I see another constraint here as mentioned in my earlier post. The development team wouldn't have the broader access in most organization to perform broader tests. How is this constraint dealt?

I know there might be different strategies followed to deal with it. I heard from some agile coaches that in few projects that they worked, UAT was not considered as a separate gate to declare product increment as done. While others stated, the Sprint was split to accommodate that review viz. a 4 week Sprint with 3 weeks of development + 1 week of QA (including UAT). A few other prefer to have a separate time after each Sprint to cover UAT.



The reason for my question is to learn what best worked in the real world staying closer to the principles of agile development.

Have you found the term UAT in the Scrum Guide?

Scrum is a framework, and you can establish your own development process under the framework of Scrum.

For PRODUCT development, the Product Owner is just like an entrepreneur, taking full responsibility for the success or failure of the product and the return on investment.

For PROJECTS, the Product Owner can play the role of communication and QA/QC, but s/he may not suitable for signing UAT documents on behalf of the business department, if signing UAT is necessary.

Special reminder, do not regard Sprint Review as an acceptance activity. Sprint Review is used to confirm functional requirements and collect feedback. When the function is confirmed to meet the requirements, the Product Owner can record it on User Stories. If the company requires UAT procedures, this can be performed in the stage environments. If there are DevOps environments, don't worry about this issue.

Scrum Guide provides a development framework, but for ALM, there may be Product Plan, Release Plan before the development phase. After the Product Owner decides to release the latest increment, there may be a final test plan and deployment plan.

I know that many people will say that the Scrum team does not need UAT, and the DoD is defined by the development team. We must trust the Scrum team and the Product Owner.

If this theory is correct, is it true that everyone who participates in the Scrum course does not need to take an exam to obtain certification? We should trust the lecturer and all the students. Why do we need a certification exam?

Is it possible that after the Product Owner decides to release, it fails to pass the UAT of the business department?

Of course it is possible. What to do then? The Product Owner may have to communicate with the business department to understand why they did not accept it.

Just like when a product enters the market, customers use money to vote whether the product passes UAT. What can Product Owner do? Therefore, the business department may also reject the products provided to them by the Product Owner.

Basically, don't equate UAT with software quality or DoD.

In the development process of V-Model, UAT's acceptance criteria comes from requirements analysis.

In Scrum, you can list the UAT acceptance criteria in User Stories as a reference for development and Sprint Review. Finally, these acceptance criteria can be used as reference verification materials for the final UAT.

Definition of Done can only respect work the Dev Team is doing themselves. They are responsible to reach the DoD of the SBIs by the end of the Sprint. Therefore integrating UAT into the DoD you would make it unpredictable if those items can really be done at the end of the Sprint.

At the end of the Sprint you then have the Review. We do invite customers and user representatives to the Review and demo what has been done. Those increments may then be released to a model office where the UAT can be done and if there are findings they will end on the PBL as new items.

One logical step could be for PO to pitch in on behalf of the business users and perform the UAT (by becoming part of the development team).

There's nothing in the Scrum Framework which forbids a PO from also being a Development Team member. The team should possess all of the skills needed to craft a Done increment, including UAT. 

The challenges you describe can indicate a need for organizational change, if the way the organization delivers value is actually to be improved.

Therefore integrating UAT into the DoD you would make it unpredictable if those items can really be done at the end of the Sprint.

100% agree.

We can add the UAT acceptance criteria to the DoD.

This action only ensures that the UAT test can be passed, but it cannot replace the UAT acceptance procedure.

Perhaps during the Sprint Review, the business department agreed to all the functions and was also willing to sign the UAT document.

However, the Scrum Team cannot force the business department or its representatives to sign UAT documents.

The team should possess all of the skills needed to craft a Done increment, including UAT. 

The owner of DoD is DT.

But... who owns UAT.

UAT: User acceptance testing

Scrum Guide - Sprint Review

This is an informal meeting, not a status meeting, and the presentation of the Increment is intended to elicit feedback and foster collaboration.

But... who owns UAT.

UAT: User acceptance testing

In Scrum the Development Team should have all of the skills needed for work to be "Done", which means to be of immediate release quality.

+1 to Ian.

In my point of view, "User Acceptance Test" doesn't mean "tests run by user", but "test from the user point of view".

Same for "Performance Test", "Vulnerability Test", "Usability Test"...

Around me, I see collaboration in the elaboration of the UAT (see 3 amigos). Who actually run the tests is of very low importance.

In my experience, it does matter whether you develop a sales funnel accelerator or a core banking system. I just don't think one UAT approach fits all. Especially not one which boldly rejects even the idea of expert testing.

Thank you everyone. Very good feedback and perspectives.

Although I didn't exactly get the answer to my question but maybe I don't have to. There is not "one point" answer to this challenge cos it looks layered. Depending on what layer the problem is hurting the organization or the adaptability of Scrum framework (and how mature the Scrum team is), different teams may apply different strategies.

The definition of done is an indicator the increment is complete and if the resulting increment meets that definition, it should be ready for release. How that definition is defined is unique to each context. And whether the PO or business actually releases the increment after the Sprint or ask for correction/enhancement to it triggers the next cycle of grooming, planning, work and review. Retrospective meeting would a good place to discuss why that occurred and how the definition of done can be adjusted to overcome that gap.

-- Mohit

Same for "Performance Test", "Vulnerability Test", "Usability Test"...

I cannot agree.

Of course, the development team can perform the same tests by themselves according to UAT’s acceptance criteria and add these tasks to DoD,

User acceptance testing (UAT) is the LAST PHASE of the software testing process. During UAT, actual software users test the software to make sure it can handle required tasks in real-world scenarios, according to specifications.

I still want to emphasize again, who owns UAT?

If the development team can judge the results of UAT by themselves, or even sign UAT documents, then, if you pay my team to help you develop software. I will invite you to participate in every Sprint Review. When the software development is completed, we can tell you that we have completed UAT, and the Product Owner thinks the software is suitable for delivery to you, please accept it. You don’t need to test the software anymore.

 Depending on what layer the problem is hurting the organization or the adaptability of Scrum framework (and how mature the Scrum team is), different teams may apply different strategies.

Based on my own experience, I briefly summarized my own opinions.

If we are developing a packaged software, even if the Product Owner thinks it can be released, we will still invite people in the product marketing department and training department, and even invite some power users to represent real users to perform pre-market UAT tests, or Beta test.

If it is an outsourcing development project, even if stakeholders have been invited to participate in the development process, and closely cooperate and communicate. Even if the development team has completed all UAT testing before Sprint Review. Before the release, we will provide them with the test report for reference, but it is up to them to decide whether to test again or accept the test results.

If it is a system used by an internal business department, the situation is much simpler. Because in the process, you can invite actual users to participate in the test.

If it is a system used by an internal business department, the situation is much simpler. Because in the process, you can invite actual users to participate in the test.

Not necessarily. E.g. when your solution is subject to a third-party audit. Or, when your solution needs to be approved by the accounting department. Or, when your data-driven solution can only be efficiently tested in an environment you will never have access to. Or, when you have a staging environment simply for the sake of emulating the production ecosystem. Or, when you are interfaced with a couple of third-party services offering no test interfaces (though that is rather a pilot than a UAT). There are lots of reasons why the final checks can be far beyond the control and competency of the development team.

There are lots of reasons why the final checks can be far beyond the control and competency of the development team.

Agree. I think we are from the same world.

Scrum is not a standard process, Scrum is a framework.

Scrum is not a silver bullet, it is not possible to define all activities in the product life cycle.

Before the development process, we may use DDD to collect the user's functional requirements.

After the development process, in addition to commercial policies, the most important thing is feedback from real users.

Unit testing, functional testing, functional integration testing, system integration testing, these tests related to the quality of the software are under the control of the development team.

The development team can certainly do the testing of UAT, but it cannot represent the opinions of actual users. Especially in the case of outsourcing development, do you think the stakeholders will agree to accept it on your behalf?

The development team can certainly do the testing of UAT, but it cannot represent the opinions of actual users. Especially in the case of outsourcing development, do you think the stakeholders will agree to accept it on your behalf?

Perhaps there is a degree of uncertainty there, and it would be best to inspect and adapt empirically, releasing small increments to minimize the leap-of-faith being taken.

Being a tester for all my professional career (total 26 years) and a tester in pure Agile environments, mostly under a Scrum framework set-up (last 14 years) you could say that I have some experience with testing (UT, CT, SIT, ST, UAT…).



A Development Team is responsible for several types of testing, going from Unit Testing by the developers to Component Testing, System Integration Testing and System Testing by the developers (testers) and if the competences are present in the team also the Performance Testing and Load Testing.

Security Testing is mostly done by external parties, because it’s a niche market with very specialized skills.



User Acceptance Testing should always be done by USERS and the focus lies not on Quality Control, but on software/application ACCEPTANCE, else this testing would have been called User Testing and not User Acceptance Testing.

 

Al the Critical, High and Medium defects and most of the Low and Cosmetic defects should have been removed during the UT, CT, SIT and ST tests.

In general the UAT will find some Low and Cosmetic defects, from time to time User Misusage of Software (not an ISTQB defined term) and some Nice To Haves, yet the main focus of UAT’s is that the users become familiar with the new functionalities and improvements, start gaining trust in the application/software and accept these changes, which will result in an user commitment to use the software.

What use has great software if no one will use it?

Now there are hundreds of ways of involving users in this process, I will give the two extremes and you can probably think of the other hundred ways between those boundaries.



Boundary 1) Power Users: After each new added coded that has been tested by the Dev Team (from UT to SIT), that increment will be deployed on an Acceptance Test Environment (Test Environment that is identical to the Production Environment) and a selection of earlier defined Power Users is let loose on it. In the beginning following UAT Test Scenarios to get acquainted with the changes/improvements and after those UAT TS, they can have a ball. Feedback and defects are captured.

Boundary 2) Early Adapter Clients (Companies): Can only be done when there is a limited amounts of releases per year. The releasable increment is given to a small subset of client companies a month to six weeks before releasing it to all your clients. These companies use your software and give feedback and in case of defects, also the defects. Defects that need to be fixed before an overall release are found within the first two weeks giving the Dev Team to correct them and release a new bug-fixed increment to the early adapter group for retesting. This requires off course a very good knowledge of versioning and for a certain amount of time (between early adapters release and overall release) all bug-fix code needs to be implemented and tested in the releasable increments and in the most recent increment on which the Dev Team is working and that won’t be released to the public, since time has passed and new functionality has been added to that non-releasable increment that isn’t present in the releasable increment.



Like I said, two extremes with several other (hundreds) approaches between them, the selected UAT approach depends from the company, the type of software (public customers or private clients), budget, time to market, customers, etc.….

RENÉ ,

Thank you for providing such valuable and detailed experience sharing.

Hi Rene,

The information provided by you is indeed very helpful. Quick question though on the boundary 1:

Boundary 1) Power Users: After each new added coded that has been tested by the Dev Team (from UT to SIT), that increment will be deployed on an Acceptance Test Environment (Test Environment that is identical to the Production Environment) and a selection of earlier defined Power Users is let loose on it. In the beginning following UAT Test Scenarios to get acquainted with the changes/improvements and after those UAT TS, they can have a ball. Feedback and defects are captured.

In this case, does the step of moving the newly added code (i.e the increment) and UAT performed by the power users occur within the Sprint time-box? Or outside of it? Would you consider the Sprint as over at the handover to power user for UAT stage? And I assume the identified defects are evaluated (between must fixes and good to haves) and discussed with PO to put into the product backlog for prioritization for subsequent Sprint. Ideally the no of must fix defects should be very low at this point.

-- Mohit

Hello Ching-Pei (I hope that I've selected your first name, else my sincerest apologies),



Testing is my stick and if I can help someone, I'll gladly do it.

In case you or someone else has further questions, I'm always willing to answer them.



With kind regards,

René

In this case, does the step of moving the newly added code (i.e the increment) and UAT performed by the power users occur within the Sprint time-box? Or outside of it? Would you consider the Sprint as over at the handover to power user for UAT stage? And I assume the identified defects are evaluated (between must fixes and good to haves) and discussed with PO to put into the product backlog for prioritization for subsequent Sprint. Ideally the no of must fix defects should be very low at this point.

Hello Mohit,



The above technique works if you create several increments during the Sprint, each build captaring just 1 to several tested improvements. This means that UAT's are performed constantly during the Sprint keeping the workload for them in small batches. Never build 1 increment at the end of a Sprint because then you are doing some form of Waterfall and UAT Testing becomes a bottleneck.

So if the dev/test workload has been planned in a good way, the UAT will occur for most new code during the Sprint, only the last finished code becomes a small bottleneck at the end of the Sprint, so try to keep that last batch of new improvements as small as possible.

The team is working on some flow management of the delivered value.



In my case, the Sprint is over when the last piece of improvement (in the last increment) is handed over to the Power User because normally all Critical, High and Medium defects are already found by the UT, CT, SIT and ST testing, so defects that are found by Power Users are prioritized by the Product Owner and Tester and most of the time added to the Product Backlog.

In very rare cases that a high defect is found at the end of a Sprint, it's prioritized by PO, Yesyer and Developer and put in the Sprint Backlog, to be picked up immediatly when the next Sprint starts, followed by arelease of a "bug-fix" increment to the Power Users.



That's the high level approach, most important is to trust your common sense and keep your head cool and all will work out.

Thanks Rene. Follow up question:

The above technique works if you create several increments during the Sprint, each build capturing just 1 to several tested improvements. This means that UAT's are performed constantly during the Sprint keeping the workload for them in small batches. Never build 1 increment at the end of a Sprint because then you are doing some form of Waterfall and UAT Testing becomes a bottleneck.

So if the dev/test workload has been planned in a good way, the UAT will occur for most new code during the Sprint, only the last finished code becomes a small bottleneck at the end of the Sprint, so try to keep that last batch of new improvements as small as possible.

The team is working on some flow management of the delivered value.

Usually in one Sprint the goal is to build an releasable increment of the overall product being built. So, with the above approach you have mentioned, looks like the subset pieces are being tested within the Sprint as development team builds the features. For example: if in a Sprint the development team is working on 5 user stories to build a releasable feature, as the team finishes story 1, it is handed over for UAT after unit test etc. are done. Correct? Or did I misread it? If yes, sometimes it may not be possible to follow this approach...cause only once all the 5 user stories are complete (development wise), can the increment/feature be tested. So in this case, I would assume we may have to follow a little form of waterfall within Sprint to include the UAT so that the increment can be tested by users of the product/software. I agree it can create bottleneck though and also, it kind of creates waste when the development team is just waiting for the test results from UAT.

-- Mohit

Usually in one Sprint the goal is to build an releasable increment of the overall product being built. So, with the above approach you have mentioned, looks like the subset pieces are being tested within the Sprint as development team builds the features.

From the Kanban Guide for Scrum Teams:

Increment:

Scrum requires the team to create (at minimum) a potentially releasable Increment of “Done” product every Sprint. Scrum’s empiricism encourages the creation of multiple releasable increments during the Sprint to enable fast inspect and adapt feedback loops. Kanban helps manage the flow of these feedback loops more explicitly and allows the Scrum Team to identify bottlenecks, constraints, and impediments for faster, more continuous delivery of value.

 

For example: if in a Sprint the development team is working on 5 user stories to build a releasable feature, as the team finishes story 1, it is handed over for UAT after unit test etc. are done. Correct? Or did I misread it?

No, you did not misread it. Create a releasable increment after each new improvement in combination with managing the flow of value/user stories, so that the user stories aren't ready all (or most) at the same time.

sometimes it may not be possible to follow this approach...cause only once all the 5 user stories are complete (development wise), can the increment/feature be tested. So in this case, I would assume we may have to follow a little form of waterfall within Sprint to include the UAT so that the increment can be tested by users of the product/software. I agree it can create bottleneck though and also, it kind of creates waste when the development team is just waiting for the test results from UAT.

This is the difficult part, like the slogan says, Scrum is easy to learn, but hard to master.

Sometimes you will come into the above scenario with all its disadvantages.  It gives you and the Dev Team and the PO an opportunity to learn lessons from it and see how to manage the Product Backlog and Sprint Backlog in such a way that the above situation happens rarely while still providing the most suitable value for the customer.



The problem in your above scenario is that the CT, ST and SIT can than also only be tested at the end, creating there also a bottleneck and this is what we try to avoid (keyword: "try", because sometimes we can't avoid it, but we want to minimize such instancies).

Thanks Rene.

Ironically whenever a complex scenario arises and tend to think "how would you do that staying complaint to the rules of Scrum", we kind of tilt towards bending the rules a little (note, "bend" not "break") in the form of adaptability and tailoring. Don't get me wrong though. Sure thing, it's needed. Maybe that's the beauty of it being a Framework. It can be adapted for what works best for an organization's context. There is no single right way to solve a complex problem. We have to use sensibility, trust and sometimes courage to get the best value out of the process. And it should start from the top so that the transition gets easier and stays fruitful.

Appreciate your valuable inputs on this topic.

Create a releasable increment after each new improvement

A wise thing to do, and anyway, technically this is happening in the development environment. But the Scrum Guide does not imply this, I believe this is rather Kanban. In the guide, the increment is "the sum of all the Product Backlog items completed during a Sprint and the value of the increments of all previous Sprints". The guide does not even tell when to show it to the Product Owner. What is more, the guide only once mentions that the Product Owner may accept something potentially releasable, ironically in the Cancelling a Sprint part.

It is even funnier that we used to work this way in the pre-Scrum era:)

While the definition of the Increment in the Scrum Guide doesn't state explicit that several increments can be delivered during a Sprint:

Increment

The Increment is the sum of all the Product Backlog items completed during a Sprint and the value of the increments of all previous Sprints. At the end of a Sprint, the new Increment must be “Done,” which means it must be in useable condition and meet the Scrum Team’s definition of “Done.” An increment is a body of inspectable, done work that supports empiricism at the end of the Sprint. The increment is a step toward a vision or goal. The increment must be in useable condition regardless of whether the Product Owner decides to release it.

The definition of Done in the Scrum Guide has a sentence that implies that several increments can and may be delivered during 1 Sprint:

The purpose of each Sprint is to deliver IncrementS of potentially releasable functionality that adhere to the Scrum Team’s current definition of “Done.”

Unfortunate, I've read over that "s" in Increments in the above sentence like probably everyone else, but in the Scrum Open assessment here on Scrum.org there are sample exam questions that will put your attention on releasing several increments during a Sprint, so long that you also have at the end of a Sprint a potentially releasable increment.

The Kanban Guide for Scrum Teams finally explicit defines the practice of several increments during a Sprint and although the focus of the guide is on adding Kanban techniques in Scrum, it states that "according to the rule that Scrum can only exists in its entirety", the Kanban practices do not change the Scrum practices, ergo the definition of the Increment in the KGFST is 100% compatible withe the definition of the Increment in the SG.

Increment

Scrum requires the team to create (at minimum) a potentially releasable Increment of “Done” product every Sprint. Scrum’s empiricism encourages the creation of multiple releasable increments during the Sprint to enable fast inspect and adapt feedback loops. Kanban helps manage the flow of these feedback loops more explicitly and allows the Scrum Team to identify bottlenecks, constraints, and impediments for faster, more continuous delivery of value.

While the definition of the Increment in the Scrum Guide doesn't state explicit that several increments can be delivered during a Sprint

... it does not state the opposite either. This is a technical detail not regulated by Scrum.

Unfortunate, I've read over that "s" in Increments in the above sentence like probably everyone else,

For me, that sentence sounds like a generalisation. On the other hand, out of the 47 occurrences of increment at least 20 cannot be understood but as 1 increment at the end of the Sprint. But again, in practice, there is nothing against considering every PBI an increment - just the Scrum Guide does not use the word 'increment' this way.

The Kanban Guide for Scrum Teams finally explicit defines the practice of several increments during a Sprint

This is intrinsic to Kanban, but not to Scrum. The Kanban Guide certainly does not define anything in Scrum. This is a solution for implementing Kanban rules within Scrum. It would be more than astonishing to learn such a decisive Scrum rule from the Kanban guide.

the Kanban practices do not change the Scrum practices

Most XP practices are fully compatible with Scrum either, many even encouraged. Just not mandatory. There is a reason why Scrum is not Kanban and not XP but Scrum.

I guess it depends on case to case and what product being developed (and what value means to the customer). Don't think it is possible to deliver increment with each sprint backlog item or story every time. It has to be combination of few stories.

For example, if an app is being developed where the first page is login, second to add some shopping list to cart and the third to place order, each page can be broken into various backlog items but delivering just the working login page in a Sprint won't give any value to the customer yet. Only once the entire cycle (login->select.>place order) is complete and delivered, would the end product be useful (automating the order placement experience). Of course this is just my opinion and a vague example but I guess you get the point.

-- Mohit

@Sean, there is one Increment per Sprint which can be made up of many releases during the Sprint.  As per the Scrum Guide: "Release products and enhancements, as frequently as many times per day;" and then the Increment would be all of the releases combined.

@Eric, release usually means distributing the solution to the consumer, and I believe, in René's post that was not the main point. On the other hand, I have just realised that strictly based on the Scrum Guide, we may not have the necessary vocabulary for this discussion.

I guess, the "Release products and enhancements, as frequently as many times per day;" was added to the Uses of Scrum because there were complaints in the early years of agile: if you have to wait until the end of the Sprint to release something, that is just not suitable for teams working on many small enhancements. Thus, this statement clarifies that regardless of the end date of the Sprint, software can go live.

In the end, the release and the increment are not necessarily related to each other. In practice, POs certainly prefer to be in the position to go live with any combination of PBIs regardless of the end of the Sprint and our definition of Increment.

@Mohit, I agree, releasing a bare login functionality would make little sense. However, when the team builds the increment, it can be implemented, verified, inspected, approved on its own. So, from this point of view, it is a good candidate for a Kanban-style 'increment'. There is no need to wait for implementing payment and whatsoever to make the login "Done". Going live is a different question.

Thanks Sean,

@Mohit, I agree, releasing a bare login functionality would make little sense. However, when the team builds the increment, it can be implemented, verified, inspected, approved on its own. So, from this point of view, it is a good candidate for a Kanban-style 'increment'. There is no need to wait for implementing payment and whatsoever to make the login "Done". Going live is a different question.

I understand and agree. Increment can be anything in "done" state as long as it meets the Sprint goal. However, honestly, if the increment is done but adds no value since it can't be released before the other pieces are done, will not make sense either. I guess that's why the PO must work (or try the best) to prioritize the backlog in a way that the outcome of every Sprint should be something that adds value and potentially releasable. The example I gave may not a suitable candidate to be build using Scrum in that case. Because Scrum aims to deliver value early. And, in this case until the 3 pages are ready, it will not deliver much value to the customer (although they maybe done individually). Would you still call it a potentially releasable increment??

-- Mohit

@Mohit, I think it is true in general for potentially releasable increments that it may not make sense to release them without further added functionality. That is why they are potentially releasable and not to-be-released.

They also may be potentially releasable because the PO has decided not to put it out in public yet for various reasons. Could be support training, could be a bigger announce launch, could be wants more before putting it out or whatever other reason.  

I guess that's why the PO must work (or try the best) to prioritize the backlog in a way that the outcome of every Sprint should be something that adds value and potentially releasable.

I would express this in rather stronger terms: the Scrum Team must genuinely intend, in Sprint Planning, to release a Done increment by the end of the Sprint which satisfies the Sprint Goal. Each act of release ought to be a timely one which is based on the latest emergent conditions. No team is obliged to release the wrong thing. As such, the developed increment is potentially releasable.