Is UAT part of the Sprint?

This goes to Definition of Done.  Is the story complete if it hasn't gone through UAT yet?  What happens to the story if UAT finds problems? Is it considered done then?  Admittedly, this will require you to think more carefully about the sizing of your stories, but that's not necessarily a bad thing.

I highly doubt that the item be delivered to production if it has not completed UAT.   In addition, it sounds like you just divided the sprint by phase, creating a mini-waterfall in each sprint.   This is neither Scrum nor Agile.

In my opinion, you need to change the "Focus" of your sprints, from trying to keep everyone busy (premature optimization), to actual completion of items regardless of who is busy or not.

It sounds as though the work in a Sprint is being moved between stations as a batch, and with reference to a schedule. There’s probably no sense in doing so if you have well-formed items that are independently valuable.

It is usually better to make schedule orthogonal to the workflow, which means limiting Work In Progress and having the team focus on one item at a time in order to complete it. In other words, think about how you can smooth the flow of value throughout a Sprint so as to deliver it earlier and more often.

In addition to the points made above, I'd like to note that your current company has the wrong understanding of the term sprint.

At my organization we follow a schedule so there's a release to production every 4 weeks.  So the Sprint schedule is setup the following way:  Two weeks development then we go into one week QA. After QA signs off, we go into UAT and at that time the development for the next sprint starts.  So the UAT is not part of the sprint.  Is it ok to move with the above schedule if we promised our client a new release every 4 weeks?  

So you have

• 2 weeks development, followed by
• 1 week QA, follwed by
• 1 week UAT

Your company seems to believe there are 3 week sprints, releases to production every 4 weeks, but the reality is quite different. 

Development, QA and UAT should go hand in hand, and daily cooperation is a must. As soon as a story is dev complete, it's QA-d and pushed for final UAT. So rather than following the traditional stages of design > development > testing > acceptance, you should put efforts into integrating all aspects of the work into a certain time frame, with a daily completion rate, and inspection the following morning.

You are doing waterfall or possibly Kanban but certainly not Scrum.  From the Scrum Guide

The Development Team consists of professionals who do the work of delivering a potentially releasable Increment of "Done" product at the end of each Sprint.

It sounds like your Development Team is delivering a potentially testable Increment.  

But a lot of this depends on how you have written your Definition of Done and whether your Development Teams starts working on the next iteration while the testing is occurring on the previous.  If that is the case, then you could argue that they are doing Scrum.  But I would expect that during the testing phases they are working on fixing bugs found.  That would lead me to deduce that the Increment was almost potentially releasable. 

I know this is a scrum.org forum but I suggest you take a good hard look at Kanban because it is an Agile practice that would fit better into your organizational structure. (Hope that last statement doesn't get me banned.)  If you really want to do Scrum, then your organization is going to have to change.  Remember that you can still do a 4 week release schedule as long as the teams are able to build potentially releasable increments each sprint. But you will have to incorporate testing into your sprint activities.  

It seems like mini waterfall inside the sprint and at the end of each sprint there should be incremental release. Also I don't see sprint review or retrospective happening here.

As "potentially releaseable" is different from "definitely releaseable", and "release" can be different from "release to production", a UAT can be done outside the SCRUM framework.

Given a complex business software, if a feature is implemented and verified to be working without errors at the end of a scrum sprint, it does not say anything about if the sum of all features are suitable and sufficient for business operations. A UAT could be seen as a simulated production release for validation of the product's suitability for the business, in order to minimize risks when developing and releasing complex and business critical applications.

Tilmann, this may be true only if UAT is not a requirement for releasing to production.   Perhaps there can be a separate effort to vet the new functionality with a subset of users, to gauge suitability and sufficiency for business operations, but this only works in Scrum if this effort is not a pre-requisite for going live.



Scrum is quite clear that the goal of each sprint is to produce items of value that are suitable for production release.   Arguing that UAT can also be an end-goal for a sprint simply muddies the waters.

If there is any testing to be done then the work is not Done, and it remains work In progress. It doesn’t matter how close it is to production, such as a UAT environment.

This does not imply that work ceases to be tested once it is Done and released into production, rather from that point it will be in a fit state for evaluation under empirical conditions.

As "potentially releaseable" is different from "definitely releaseable", and "release" can be different from "release to production", a UAT can be done outside the SCRUM framework.

Tilmann, your position is not really accurate. I'll copypaste from another thread where I said: "do not confuse the terms "potentially" and "definitely" in front of releasable. What I mean by that is, the increment produced by the team should be of production quality (per DoD), which means it should be of definite release quality, however the actual deployment to production environment is potential, for the PO (business) may decide to postpone release."

Timothy, re "this may be true only if UAT is not a requirement for releasing to production" have you (or anyone else for that matter) ever experienced a scenario where user accepted testing was not such a requirement? While I've got enough experience behind me, I don't know it all, hence my question.

What is user acceptance?  Sometimes it's better to deliver something to the user and learn from their real-life, marketplace behaviour, than to give them the opportunity to not-accept something when they've only 'tested' it.

Alex, I'd argue that sounds more like market feedback, not acceptance. Wouldn't you agree?

Most teams use acceptance criteria (regardless of whether these are written in a ticket, in Gherkin, etc) and these acceptance criteria have to be met in order for the story to be considered done. UAT means internal validation (not production one) and can be performed manually, by automated tools, or a combination of both. We surely don't want un-checked stories to be shipped to production and have the users UAT them, do we?

In some cases UAT is performed by the PO themselves in order to gain confidence and add an extra layer of quality security. 

Let me rephrase my original question then.  What is the purpose of User Acceptance Testing?  Is it for the users to determine what is production ready?  Or is it to understand how the users' behaviour and validate/invalidate assumptions? 

If it's the former, does that sound fair and in line with Scrum?

If it's the latter, then yes, it does sound like market feedback.

Not sure I fully understand where you're coming from.

You do UAT internally to check whether the work is in agreement with the intended development (any variations/changes to requirements would have been handled one way or another, and result in an appropriate solution). So, from this perspective, the purpose of User Acceptance Testing is to verify whether the story, as thought of and as understood by the team, meets the criteria set forth for it to be deemed complete.

This is surely very different than having the very same story in production, subject to users' consumption - utilization - operation - call it what you want. This is the real market feedback (whether from external or internal customers).

So, from this perspective, the purpose of User Acceptance Testing is to verify whether the story, as thought of and as understood by the team, meets the criteria set forth for it to be deemed complete.

What separates this user acceptance to any other form of testing?  

My understanding of UAT has always been that it includes the user in some capacity.

"What separates this user acceptance to any other form of testing?"  

Perhaps the fact it is supposed to be a rather exhaustive one, performed on behalf of (or by, in some cases) the stakeholders in order to gain confidence the development satisfies the wanted behaviour/requirements, there are (to the best of their understanding and efforts) no bugs (or if there are, they are so minor there's a chance they won't even be noticed by end-users), etc? In UAT you want to make sure that was was developed works flawlessly and the user will be able to use it (no bugs, no errors, no crashes). Therefore, User Acceptance Testing is performed, on behalf of the business, for the end-users.

As mentioned above, this is way different than end-users' feedback - hence the value that is yet to be proven.

Here's where a DoD (and a review/demo) come in handy. Would the business (PO) be confident of releasing to production a story that, although it passed dev checks and QAs, was not looked into by the stakeholders' representative (PO)? Or a BA if there is one? Or have a wide group of interested people (review) see it in action (albeit not in depth, cause UAT should be done before the review).

My understanding of UAT has always been that it includes the user in some capacity.

If by user you mean end-user (external customer) then the answer is that it usually doesn't. UAT is performed internally, by the business, from a end-user perspective. In performing the UAT, the business wants to gain confidence on whether the story is fit for the end-user's utilization.

____

The above represent my knowledge and understanding, and therefore constitute my practice. If someone has different views/understanding, please share your thoughts and arguments.

Here's where a DoD (and a review/demo) come in handy. Would the business (PO) be confident of releasing to production a story that, although it passed dev checks and QAs, was not looked into by the stakeholders' representative (PO)? Or a BA if there is one? Or have a wide group of interested people (review) see it in action (albeit not in depth, cause UAT should be done before the review).

Are you not risking relinquishing the control of the Development Team to external parties by doing this? As in, if a PO and a BA or a wider group (any/all of which might fall outside of the Development Team) are needed to determine what is or what isn't Done, is it then fair to say the Dev Team has all the skill it needs to bring the Increment to a potentially releasable state?

The above represent my knowledge and understanding, and therefore constitute my practice. If someone has different views/understanding, please share your thoughts and arguments.

I might be wrong too! I guess this is why transparency over terminology is so important.  I wonder if by leaving out UAT entirely, the Scrum Guide is suggesting that such a process should be viewed from a different perspective.

Are you not risking relinquishing the control of the Development Team to external parties by doing this? As in, if a PO and a BA or a wider group (any/all of which might fall outside of the Development Team) are needed to determine what is or what isn't Done, is it then fair to say the Dev Team has all the skill it needs to bring the Increment to a potentially releasable state?

I don't think so, Alex. Here's why (note I'm referring striclty to Scrum):

• PO is part of the Scrum Team (and a BA can be, too, as long as they actively contribute to the current increment). 
• There is no control over the DT, but an ongoing collaboration.
• No outsiders would determine whether it's done or not. If there's a DoD in place, then that's all the guidance needed.
• The DT can surely bring the increment to a potentially releasable state, however the PO, even if the increment satisfies the DoD and is potentially releasable, may still not be convinced of some stories in it, in which case they may delay release, ask for some stories to be rolled back from release and so on.
• I've seen cases where everything went well up to the review, where, post demo, a senior stakeholder advised that the weekly release be put on hold due to their concerns over functionality (everything worked/looked great from our team's perspective, but said stakeholder was concerned over the functionality - financial industry btw and recommended we don't present it to customers)
• Bottom line: UAT should be part of DoD, imo

• I've seen cases where everything went well up to the review, where, post demo, a senior stakeholder advised that the weekly release be put on hold due to their concerns over functionality (everything worked/looked great from our team's perspective, but said stakeholder was concerned over the functionality - financial industry btw and recommended we don't present it to customers)

This is exactly the problem I think introducing the concept of traditional UAT to a Definition of Done brings.  

By making acceptance (in this case by a senior stakeholder) something beyond the Development Team's control, they cannot fairly be held responsible for bringing something to Done, a potentially releasable state.  

I understand the collaboration has its benefits and would more than likely be useful for the Development Team in reducing the risk of creating an Increment which is not Done. If collaboration goes beyond that and becomes a formal step of approval, then the Development Team no longer has control over what's Done.

This is exactly the problem I think introducing the concept of traditional UAT to a Definition of Done brings.  

By making acceptance (in this case by a senior stakeholder) something beyond the Development Team's control, they cannot fairly be held responsible for bringing something to Done, a potentially releasable state.  

I actually didn't say that, Alex. If I lead you to interpret my words that way, apologies. UAT was part of a DoD, but there was no acceptance from anyone else in the above example. The functionality was done in acordance with the DoD, everything was right in every way possible. My point was that, even if a story passes UAT and is ready for release, it can still be parked (not released) for reasons outside the team's knowledge, hence the importance of not only UAT being done internally, but also of gaining as much internal feedback (through demo ideally) as possible.

Let me use another example to better illustrate some of the points above.

In that finance company I mentioned above, we weren't operating in a pure Scrum setup. We were very close to it, but also very far if you know what I mean. But some things we did do right. One of these was focusing on value and deliver in small bits, week after week. There was a very good collaboration between all team members and also with most outsiders. In our setup, the PO would always perform UAT against the work created by DT and reviewed by QA. The reason, in that case, was that because of the circumstances and industry specifics, we had to ensure the functionalities delivered to end-users were as neat as possible. Sure, there were bugs, there were live (production) issues, but that's not the point.

The point was that, the PO, a former trader, had to ensure the feature is fit for other traders' use, and it was as close as possible to "real-world" baptism. It surely would've been far from wise to skip UAT, and have customers do the so called alpha- or beta-testing. While some companies, in some industries, can afford to engage in a- or b-, it wasn't our case.

the PO, a former trader, had to ensure the feature is fit for other traders' use, 

And if it wasn't fit? What would occur?

There are many real-world examples, where SCRUM can be beneficial, but something like a UAT would not match into a sprint. 

As explained in another thread, life critical applications are a good example for this, like medical diagnosis software or airplane control systems. For a good reason, users of such systems would not rely on a few days of testing such a software. They would not give acceptance to go into production before it has been tested intensively in many simulated real-life situations, which in most cases would take much longer than a single sprint.

Getting the empirical feedback only after setting such systems into production could cost lifes in such cases.

So "potentially releaseable" of a done backlog item could mean "if released into production, it would work without errors and as specified", which in many cases is insufficient information for judging the risk of really releasing it into production.

the PO, a former trader, had to ensure the feature is fit for other traders' use, 

And if it wasn't fit? What would occur?

The story wasn't done per DoD, would go back to dev, testing, or looked into further.

There are many real-world examples, where SCRUM can be beneficial, but something like a UAT would not match into a sprint. 

I wrote my counter arguments on the same thread, noting there that trials in such delicate areas ("life critical applications are a good example for this, like medical diagnosis software or airplane control systems") could last for months, even years. Those are extreme cases. The standard is quite different, as the business needs to test (UAT) internally; only a few companies in a limited number of industries engage in alpha (beta) testing where they engage a small number of the intended audience in private testing sessions. Many of them (or perhaps all) volunteer to beta test for example (especially in gaming!). 

"The difference between quality assurance (QA) testing and user acceptance testing (UAT) varies across organizations and often within organizations. In general, UAT is usually the final testing process prior to deployment and is performed by a business tester who ultimately signs off on the product."

So "potentially releaseable" of a done backlog item could mean "if released into production, it would work without errors and as specified", which in many cases is insufficient information for judging the risk of really releasing it into production.

First of all, I argue potentially releasable refers to the whole increment, not a single done backlog item. It means the sprint backlog, if it satisfies the DoD, can be shipped to production at any point (in fact, the vast majority of increments are released to production at the end of the cycle, if there is a single release per sprint - for there are many cases where there are multiple releases each sprint, that's continuous delivery).

Potentially releasable means that it (the increment) is perfectly fit for release, with no or very few risks (that's why you have QA-UAT), and the business (through PO) is free to decide whether to release or not. Usually, business chooses to release (again if there's a single release per sprint).

If there are risks - concerns on various topics (usability, security, stability, financial implications, regulatory impact, etc), then a decision must be made, by the business, before the release, whether to go full speed or not. They'd like choose "not", and try to mitigate the risks by employing various strategies, ie opening the platform for use by a limited number of users (say 100) for a limited amount of time (say 2 weeks), or releasing the app (although intended to global audience) only on a small market (Say Portugal), or adding geolocation rules to block most external access, and so forth. 

First of all Scrum (and it's derivatives like LeSS, SAFe) is not suitable for all SW projects and all industries. Scrum is to go fast, develop and do not care strictly about software quality. Seriously what else except mostly happy path testing can be done on the last day of sprint, testing. Usually there is no time during sprint for regression testing (what's the impact of changes to already existing features), for in depth security or stress testing for 24/7 software operation.

For all those that say that UAT is not important before releasing and deploying in production I would ask a question: would you embark on a plane that you know that the Development Team said all the stories during sprint #666 met the DoD on Friday, and on Monday the software was released and deployed to production - yes, the plane that you would fly in a minute. Let's disregard the fact that in real world no one would take that risk. But you would definitely get off that plane immediately.

So why does Scrum has such a problem with UAT? In my opinion that is because most of the projects using Scrum are web based, non-essential, non-critical projects. The projects where liability is low. The risk of health damage, serious accident, financial loss or legal liability is low with SaaS software in most cases. The end user usually accepts Terms of Use with software provided AS IS. No liability. No risk. Let's accept stories to meed DoD. No stress testing, no strict requirements tracking - only business user stories.

In all projects where there is risk, where there are external specifications (IEEE, other standards), where interoperability between various vendors must be provided, where there are requirements for SW to operate 24/7, where there are external certification parties and many many more cases where prior to release and deployment there must be much more tested than user storied developed during sprint.

Mainly it's automotive, transportation, telecommunication, aerospace, a lot of projects combining SW&HW, core banking and trading platforms.

Read that one: https://techbeacon.com/app-dev-testing/5-effective-powerful-ways-test-tech-giants. It's good read, especially about FB and MSFT cases. The only issue about this article is lack of example from the industries I mentioned.

The bottom line: Scrum is not for ever project, the UAT is not part of Scrum but in some cases required. Releasing SW is both risky and costly, both in terms of money and time. 

Jack,

I disagree strongly with some of your assertions about Scrum:

Scrum is to go fast, develop and do not care strictly about software quality.

what else except mostly happy path testing can be done on the last day of sprint

most of the projects using Scrum are web based, non-essential, non-critical projects

I believe you either have an imperfect understanding of Scrum, or you have had experience(s) working under a poorly-executed Scrum framework.

@Fahad if this schedule works and makes sense for your Organization then it is just fine. However, in my humble opinion I believe there is room for improvement.

If you really want to improve your value stream then I suggest you try to use Scrum as intended and that is:

- find a sole Product Owner than can decide at any time to release Done work;

- find ways to produce Done work at the end of every Sprint to support the Product Owner's decision to possibly release.

This may include taking less work at start, perform all needed actions throughout the sprint. Inspection and adaptation with all the stakeholder can and should happen during the Sprint Review.

Success.

Timofy,

Any specific beyond buzzwords like "you either have an imperfect understanding of Scrum, or you have had experience(s) working under a poorly-executed Scrum framework"? Please be specific. I can discuss details not  buzzwords. Please share examples.

What kind of projects you've been working on? Any serious ones or just web based SaaS? Have you been working on a projects requiring 24/7 operation like flight control? telecommunication? etc. And answer the question: "would you embark on a plane that you know that the Development Team said all the stories during sprint #666 met the DoD on Friday, and on Monday the software was released and deployed to production - yes, the plane that you would fly in a minute", please.

The problem with Scrum I see is that time-boxing leaves no time for some activities in many cases: more in-depth testing (more than just manual verification of given user story), no time for user feedback (outside of Dev Team) during Sprint Review/Demo. What kind of feedback can you get during 2-4h Sprint Review? 

The point I'd like to make - Scrum is not a silver bullet and just does not fit every project, every organization. UAT must be there if we work on a very long term project where sprint is not only development of new product increments but also refactoring/redesign and 2 weeks sprint is very insignificant to the whole project duration and the code base that is already there. Have you heard about regression, when even small change in code may result in unforeseen consequences in other parts/features?

Jack, I would encourage you to look further into automated testing, continuous integration, and continuous deployment. Having a suite of automated tests set up will cover your concerns about regression testing and if tests are automated it often means tests are run throughout the sprint and not on the last day for only happy path. It ensures the high quality of software - higher than manual testing which is time consuming and error prone.

If you're working on an important system (e.g. flight control), I imagine your testing would involve much more than unit tests and happy path tests - you would probably want to include all kinds of security tests and ensure the infrastructure has a lot of redundancy to ensure uptime.

I agree that Scrum is not for every project, but I think a lot of your concerns can be addressed by Scrum. (I also didn't see any "buzzwords" in Timothy's answer).

I'd be interested to hear other opinions on this as this is just my opinion.

I am of test automation origin and when I mean test automation I mean higher test levels than just unit testing - which tests code not functionality. Let's take BDD which is poor for test automation. I mean it's better than nothing but in most cases does not provide enough coverage, mostly some happy paths with Given, When, Then but not corner case testing. Have you guys heard about model based testing with strict test coverage tracking from software model, ideal for extensive UAT mostly used in the industries like automotive, aerospace, telecommunication, healthcare, etc? Companies like CA produce software for that: CA Agile Requirements Designer - it's not product placement, just an example.

Regarding my question about the plane and sprint increment deployment to a plane I bet that recent Boeing 737 crashes are software related:

https://www.theguardian.com/world/2019/mar/17/ethiopian-airlines-crash-thousands-turn-out-for-funeral-service

This is the outcome of just poor UAT. UAT for software that have to be reliable.

What I mean in those posts is that there are industries that cannot follow sprint methodology as it's even not plausible to fit UAT in 2-3 weeks increment. There are industries out there that any issue within software may cause more serious liability than just not being able to post your sweet cat picture to Facebook.

And what worries me, that a lot of companies try to blindly follow some Scrum Guide where they definitely should not - because of the industry they software is used in. Thus I'm fully backing opinion expressed by: 

 Tilmann Zuper

 04:12 pm November 16, 2018

"life critical applications are a good example for this, like medical diagnosis software or airplane control systems. For a good reason, users of such systems would not rely on a few days of testing such a software. They would not give acceptance to go into production before it has been tested intensively in many simulated real-life situations, which in most cases would take much longer than a single sprint."

Hi Jack,

The Increment is the sum of all the Product Backlog items completed during a Sprint and the

value of the increments of all previous Sprints. At the end of a Sprint, the new Increment must

be “Done,” which means it must be in useable condition and meet the Scrum Team’s definition

of “Done.” An increment is a body of inspectable, done work that supports empiricism at the

end of the Sprint. The increment is a step toward a vision or goal. The increment must be in

useable condition regardless of whether the Product Owner decides to release it.

What do you infer from the this part of the Scrum Guide ?

Firstly i agree that Scrum is not the only solution for every industries. It majorly depends upon the organization to organization whether to use agile scrum or waterfall or any other framework. Some industries do use waterfall and it works best for them , does not mean that they are not progressing.

Mainly it's automotive, transportation, telecommunication, aerospace, a lot of projects combining SW&HW, core banking and trading platforms.

>> Your remark drew my attention about the big high risk industries where scrum cannot be used eg - 'Telecom' . I worked in a scrum team for the Telecom Giant 'Telefonica O2' at the time when it was getting merged with another big company 'Eplus'.

So project was to migrate all the data-like tariffs,customers,third party companies data etc from Eplus to Telefonica.

It was a huge project and involved high risk plus had a deadline. 

So, entire merger was done using Scrum. We had multiple teams working together on separate components and delivering increments each sprint. After that we had an integration team integrating all the components together and ofcourse it involve the integration testing. Next phase was to test the migration end to end from Eplus to Telefonica.

Once it passed the end to end phase we released it to production environment but the feature was still not made live. On production we had team to make the live calls and test the network.

Please note the moment after delivering the increment to integration (at the end of sprint) , scrum team starts the new sprint and works on new PBIs. So scrum team was always ahead.Also we had both manual and automation testing at each phase.

So entire merger was done by delivering the increments iteratively and incremently and we did it before the deadline.

Once everything was successfully migrated and tested, the merger went Live. Success story !! 

If you get my point - Increment must be in usable condition and releasable state which does not mean that we need to ship to Live as the sprint ends. On what i have seen & researched many industries you mentioned uses scrum at some or the other stages. 

Let's take BDD which is poor for test automation.

Funny, that isn't my experience at all.   BDD is actually a very effective way to not only identify all possible conditions (both "happy path" and otherwise), but feeds very well into automated testing tools such as Cucumber.

I can’t believe myself!!!   Am I in scrum.org forum!!! Why two of my question topics were dropped?

So how can we learn scrum?

Scrum is the right of all people

No one can prevent somebody from learning it

I must leave this forum…but I decided to register my word!!!!

Harshal,

Thank you for that comment and the statement:

"We had multiple teams working together on separate components and delivering increments each sprint. After that we had an integration team integrating all the components together and ofcourse it involve the integration testing. Next phase was to test the migration end to end from Eplus to Telefonica.

Once it passed the end to end phase we released it to production environment but the feature was still not made live. On production we had team to make the live calls and test the network."

So in my opinion the increment was "usable" as you put it, but definitely not released - what does usable mean is another question and how is usable related to potentially shippable increment? It underwent integration testing, then probably after some fixes released to production, but not made live (not sure what you mean by that - production means go live usually). But the point is that the process that you are describing is really UAT + integration of multiple components, and as you describe it - its OUTof the sprint. The sprint development team is already working on the next increment, but the SW has not been released yet. So I'm confused about your statement as it brings back original question from the thread "is UAT part of the Sprint?" - in my opinion and as you described it in your past project the answer is NOT.

Regarding my comment about BDD, maybe poor was to strong. It's of course good to some extent but when project gets huge it may be difficult to manage and maintain hundreds of human-readable BDD format tests. For those interested in other solutions for business-critical software please watch that video: https://youtu.be/pTYj2tAbJHI?t=493 or read any publicly available documents on the subject.

There are industries out there that any issue within software may cause more serious liability than just not being able to post your sweet cat picture to Facebook.

I'm going to do something that will probably get this post removed but it helps to prove a point. I live in the central part of Texas in the USA. I did a quick search of Indeed for jobs in the San Antonio area.  That area is home to 2 large military bases.  I search for Software and found these job postings.  All of them are for agile specialists, mostly asking for SAFe experience.  Scrum, and Scrum practices, is used extensively in SAFe.  While this may not be pure Scrum it does use the same incremental/iterative delivery process without introducing specific UAT cycles. 

• https://gdit.com/careers/rq30511-project-manager  (Job at General Dynamics - military contractor)
• https://gdit.com/careers/2019-55430-project-manager-software-development (Another General Dynamics position)
• https://www.indeed.com/viewjob?jk=6987088653169219&tk=1d6bcn9u6hlgl803&… (Lockheed Martin development manager position that specifically states agile development experience is needed)

There a many more that have the specific keywords of "agile" and "SAFe".  I can also do the same kind of searches for many financial, insurance jobs in this area.  In fact, I worked for Charles Schwab on Scrum projects that were used for monitoring stock markets all over the world and had 24/7/365 up-time requirements. 

I will agree that Scrum is not right for every project that could possibly be justified. But Scrum can and is used on many projects related to critical and "never down" systems.  And many of these have no specific UAT cycles built in.  You have asked for specific examples. I gave you a some circumstantial evidence.  I am also going to pull a few things from the Scrum Guide and provide interpretations that fit your somewhat warped interpretation. 

From Scrum Guide section on the Definition of Done:

When a Product Backlog item or an Increment is described as "Done", everyone must understand what "Done" means. Although this may vary significantly per Scrum Team, members must have a shared understanding of what it means for work to be complete, to ensure transparency. This is the definition of "Done" for the Scrum Team and is used to assess when work is complete on the product Increment.

 If the definition of "Done" for an increment is part of the conventions, standards or guidelines of the development organization, all Scrum Teams must follow it as a minimum.

...the Development Team of the Scrum Team must define a definition of "Done" appropriate for the product.

This Increment is useable, so a Product Owner may choose to immediately release it.

If UAT is important to the company/organization for which you work, what is to stop the introduction of statements such as "UAT must be finished and all issues found included in Product Backlog for inclusion in future consideration."  There could even be an development organization wide DoD that puts this condition on all teams. 

Another point is that the increment is considered useable but it is not required to be released. That is entirely up to the Product Owner to decide and there can be a lot of conditions associated to it.  Scrum never says that the result of each Sprint is a fully releasable product.  In the section explaining the Sprint this is stated.

The heart of Scrum is a Sprint, a time-box of one month or less during which a "Done", useable, and potentially releasable product Increment is created.

Useable and potentially releasable is clearly stated. Again, there is are a lot of conditions that affect the reasons for releasing. 

A quote from one of your earlier posts.

Seriously what else except mostly happy path testing can be done on the last day of sprint, testing. 

If you are experiencing this then I will willing say that your developers do not know how to manage their development time and are overestimating their skills.  Every developer that I have worked with, Agile, Scrum or waterfall, has always included the time it will take them to build include testing (unit, integration, system or manual) into their work.  They all take pride in what they produce and do not want to provide something that is low quality. The ones I am working with currently in Scrum have actually stated that they like it better now than when they had a QA department to hand off to.  They feel like the quality of their work has increased as they have become more responsible for the entire work. 

Regarding my question about the plane and sprint increment deployment to a plane I bet that recent Boeing 737 crashes are software related:

https://www.theguardian.com/world/2019/mar/17/ethiopian-airlines-crash-thousands-turn-out-for-funeral-service

This is the outcome of just poor UAT. UAT for software that have to be reliable.

While I don't argue that these issues are probably software related because in this day and age, everything in a plane has software involved.  But I will argue that this had nothing to do with lack of UAT. I actually blame this on a lack of adequate testing by the development organization. While I don't know for sure, I am also going to venture that this could have been caused by pressure to release a product on a schedule rather when it was ready. My knowledge of the airline industry (garnered from 2 very close friends and my brother's father-in-law who work as pilots and a couple of people I know that work at government contractors for the US Air Force) airplane manufacturers will release details/specs about new airplanes far in advance of their availability in order to determine viability and start taking purchase orders to help finance completion of the development. After those orders are placed, pressure to deliver is applied by the purchasers which then gets applied internally.  That pressure will often influence decisions made and often cause wrong decisions to be made. 

I have worked on projects in many industries where UAT was done for months only to have some significant bugs shortly after the product is released for general use.  UAT is only as good as the people that do it.  Often UAT is done by individuals that are inexperienced in testing methodologies.  It often is made up of happy path testing which you have stated is not adequate.  Only individuals with true knowledge of software development are likely to test edge/fringe cases. I have tested products that passed months of UAT and found numerous bugs that were missed by our first internal testing and all of the UAT scenarios. I state this as a person that has been a Software Quality Assurance professional for over 20 years before I became an agile coach. I challenge you to go back over your experience and find one instance where a product was released that contained no bugs. 

I did a Google search for "User Acceptance Testing" and read 3 pages of articles found in that search.  The general description of UAT, and I am paraphrasing,  found was that it was testing by end user to validate that it meets the requirements and can the general needs of the real world.  Here is a good example of the general definition I just stated. (https://www.techopedia.com/definition/3887/user-acceptance-testing-uat)  That article does say it is very important but it also states that it is to be done based on the requirements.  However I would argue that any type of testing should be done based on the requirements.  In your case of the recent Boeing 737 crashes, do you know for a fact that the conditions that caused the crashes were covered by a specific requirement?  Do you know for sure that anyone doing UAT would have thought to cover the actual conditions that caused these 2 crashes?  Given that there are a large number of those airplanes that have had many successful flights (including the two planes that crashed) I challenge you to explain how UAT would have uncovered the exact conditions/situation that caused these 2 planes to suddenly crash.  Sure those conditions should have been considered and covered by testing but I still argue that UAT would not have found them and the only real way to have caught the edge case(s) causing them would require in-depth knowledge of the code base. 

I will admit that you have made some valid points and arguments for weaknesses in Scrum. But I am going to say that those are weaknesses in the process adopted by the companies using Scrum. Scrum contains absolutely no process statements and it is clearly stated to be a framework.  

Scrum is not a process, technique, or definitive method. Rather, it is a framework within which you can employ various processes and techniques. 

It is up to the organization adopting Scrum to determine the processes that will be used.  To say that Scrum does not endorse UAT is wrong. It is more that organizations trying to do Scrum are using it as an excuse for not doing UAT.  UAT can and is used in many Scrum organizations.  It however is adapted to fit into the framework and is usually broken into incremental testing efforts done on each increment in order to make apparent information (transparency) that is considered as needed (inspect) and included in future iterations of work (adapt). Large time and efforts investments in regression and UAT efforts are being investigated/reviewed (transparency) and being found to be part of the problem in delivering products (inspect) and are being included in the development practices at more frequent intervals on individual requirements as they are satisfied (adapt).  I have seen this practice actually help to uncover many missed, misunderstood or misstated requirements that in the end resulted in applications that actually fit the customers needs better than the original expectations. 

I am going to finish this tirade (which became much longer that I expected) with a statement that everything I have written above is my opinion and my opinion only. It is based upon my interpretation of agile practices including Scrum. I acknowledge that everyone is entitled to their own opinion and gladly welcome differing views.  But I do take offense to people that are not willing to recognize that their own opinions are not absolutely correct and will not consider that others also have valid opinions and knowledge. All of us base our opinions based on our knowledge (Empiricism at work even if you don't realize it) and these will be different for everyone. I appreciate @PJ's and all contributors to this discussion passion's and encourage us to continue it.  But please realize that none of us are going to be right. 

@Ripple Man,

We're sorry to learn you're upset. 

We agree that it's everyone's right to learn Scrum. 

We removed one of your posts yesterday because you were posting questions from our professional-level assessments. This is in violation of our Terms of Use. We sent you a notification about that yesterday, and we're sorry if you didn't receive the message. We emailed you again today, but just in case it landed in your spam folder, we'd like to repeat some of that content here.

It looks like your account was then flagged for inappropriate content, which means your posts will temporarily require approval before they will display. Once your trust rating is high enough, you will again be able to freely post on our forums.

I hope we were able to clarify any confusion with regards to your posts on our forum. 

The purpose of the forums is to cultivate understanding of the implementation of Scrum through the sharing of insight, knowledge and experience. They are not a source for specific answers to our Professional-level assessment questions. We do encourage you to use our forums as they were intended, and wish you the best in your continued learning of Scrum.

Daniel Wilhite - what is to stop the introduction of statements such as "UAT must be finished and all issues found included in Product Backlog for inclusion in future consideration."  

Again, this might be my own ignorance playing a part here, but it's the word 'Acceptance' in UAT that I find difficult.  User Testing is even a better way of saying it because, what does Acceptance mean? What if something is not Accepted? Is it not Done then? If so, is that fair to put that dependency on the Development Team?

What does Acceptance mean? What if something is not Accepted? Is it not Done then? If so, is that fair to put that dependency on the Development Team?

Will try to learn from Ian and use questions (afraid it will take me a decade or more to be as good as he is, assuming he stalls and stops learning): Would it really fall under "Done" if it's not Accepted?

No, if you ask me.

Acceptance means that, given a "current" understanding (requirements'll never be perfect), the team is confident/satisfied the work produced meets the requirements and returns the expected results. And no, I don't read (interpret) it to be a dependency on the Development Team - rather, I see it as a leap of faith: with the x amount of information, and z amount of effort, a piece turns out either done or not done.

From a wider perspective though, many people don't seem to appreciate the purpose of sprint UAT, the purpose of overall quality, and the difference between sprint UAT and reliability & quality. Once UAT passes, there are many other steps undertaken to ensure overall quality. Just because a user story passed UAT, just because a group of stories (by stories I mean software bugs, too!) passed UAT, that doesn't automatically mean they won't be verified, either independently, or as a whole (as applicable), in later stages before production deployment, or after production deployment (alpha/beta testing, etc).

@Alex, fair point and thanks for pointing that out. It is something I have overlooked. Dully noted and lesson learned.  

Why not 4 week sprint? (?)

I think this issue comes when you have UAT done by people outside the scrum team. Suggest in this case what would be the best approach.

Challenge with UAT Testing within the Sprint

Context :

1. During a Sprint 1
   1. Developer performs coding and Unit Testing.
   2. Change the ticket status to QA Testing.
   3. QA Team performs QA testing.
   4. Bugs identified will be resolved by the Developer.
   5. QA team then change the ticket status to UAT Testing.
   6. So the ticket is not moved to done status and stories are not burned.
      1. This will show the burndown chart like a straight line.
   7. Last day of the sprint, we show sprint demo.
   8. Last day Product Owner starts the UAT Testing.
   9. The Product Owner cannot move the story to done status on the last day of the sprint.
   10. So the ticket is not moved to done status and stories are not burned.
       1. This will show the burndown chart like a straight line.
   11. So sprint cannot be closed.
   12. We have to wait for a week to close the sprint
2. During Sprint 2
   1. Bugs identified in Sprint 1, needed to be closed in Sprint 2 "Active Sprint"
   2. We will add the bugs to the active Sprint

Ask:

1. Is UAT Testing should be done as a part of the Sprint, as UAT testing will start after the Sprint Demo which is the last day of the Sprint?
2. How to ensure that we can close the Sprint on the last day of the Sprint if UAT is a part of the Sprint?
3. Can we set the Definition of Done as QA testing completed and close the Story so that we can close the Sprint on the last day of the Sprint after the Demo and Retrospective?
4. If the Answer to Ask3 is "yes".  
   1. UAT Testing will be done after the Sprint is closed. IF 'UAT Testing' identifies the bugs after we close the Story and after we close the Sprint THEN how can we move the story to the next sprint for fixing the code. 
   2. How can we remove the story points awarded to the previous sprint as UAT Testing failed
5. If we keep the Sprint open till UAT testing is completed, the Sprint will get closed after the last day of the Sprint, and the Burn Down Chart will not be useful. Am I right?

A Sprint will end when the time-box for it ends, and not when someone "closes" it. Moreover no credit is ever "awarded" to teams for undone work, since the value is not available for release.

What is actually stopping your team from deciding how much work it can genuinely finish in a Sprint, and from planning and committing to a Sprint Goal that has an achievable forecast of work?

"A Sprint will end when the time-box for it ends, and not when someone "closes" it. Moreover no credit is ever "awarded" to teams for undone work, since the value is not available for release." - Correct

So the challenge is if we have UAT to be a part of the Sprint, and the UAT is done after Sprint Demo, then we have less than 1 day (the last day of the sprint)  to complete UAT to get the story points awarded to the Sprint.

So is UAT should be a part of the Sprint?, and can we consider that Development and QA and Sprint Review is enough for Definition of Done, so that we can award the story points to that Sprint.

Thus if UAT is not a part of the Sprint, then any blocker bugs identified during the UAT phase, will result in moving the story to the next sprint and we have to remove the story points awarded to the previous Sprint. Thus Jira or any tool will not help us to calculate the velocity properly and we need to do the same manually. Am I right?

The "sprint" you describe sounds very much like a waterfall Gantt chart to me.  There are multiple handoffs, multiple gating events, multiple separation of duties. "Closing" of events that have a fixed timebox. So other than using terms from the Scrum framework, what actually makes you feel that you are using Scrum?

I'll start with the "closing"of a Sprint.  As @Ian Mitchell points out a Sprint ends when it's timebox ends.  For example, consider a sports event.  Most sports events have mechanisms to determine the ending and almost all of them have some type of time element to them.  When the time limit is reached the event ends.  This is often referred to as "time has expired".  So it might be easier to understand by saying the Sprint has expired rather than the Sprint is closed. 

QA team then change the ticket status to UAT Testing.

So the ticket is not moved to done status and stories are not burned.

This will show the burndown chart like a straight line.

Last day of the sprint, we show sprint demo.

Last day Product Owner starts the UAT Testing.

I am going to provide a passage from the current revision of the Scrum Guide.  This comes from the section that describes the Increment.

Multiple Increments may be created within a Sprint. The sum of the Increments is presented at the Sprint Review thus supporting empiricism. However, an Increment may be delivered to stakeholders prior to the end of the Sprint. The Sprint Review should never be considered a gate to releasing value.

When something is put in front of users, especially external users, you are in effect releasing.  You are using the Sprint Review to gate the release of your increment to the USER Acceptance Testing.  What is the reason you can't release something to UAT when it is ready? 

I'm also going to ask what the difference is between a change being requested or bug being found during UAT and the same happening from Production?  If there is a difference in the behavior, that might be a good place for you to focus and identify why.  All are coming via stakeholder feedback and all are going to impact the Product Backlog in some way. 

Thank Daniel. One of the solution I worked out is that UAT should be outside the sprint and any UAT bug should be addressed in the next active sprint where we have a task named UAT bug fix. Thus development and QA testing is sufficient to close the sprint with proper sprint review and retrospective. 

Always best to have product owner team to participate in QA testing, so that they will try to identify core issues during the sprint. The sprint review will focus on external audience who will be part of the UAT cycle. 

One of the solution I worked out is that UAT should be outside the sprint and any UAT bug should be addressed in the next active sprint where we have a task named UAT bug fix

The work undertaken in a Sprint is not finished, it has defects uncovered by UAT and is unusable. What solution is that? What problem have you actually solved?

Hi Ian Mitchell

In the Sprint, we have mentioned that QA is performed by QA and Product Owner/Product Team and they have given the go-ahead for the ticket to be demoed and closed. Thus the probability of getting a blocker bug during UAT Testing (Testing performed by external Stakeholders/ End Users) should be minimal.

Having said that, in our IT world there is always room for any unknown issue to pop up during any stage of testing. Thus the feature delivered in current Sprint should be "usable" as it is crossed QA testing by the Product Owner and QA team, and any additional issue identified in UAT should be categorized as Blocker, Critical, Major and Minor. Based on the severity of the issue, the Product Owner will reprioritize the Bug for the current sprint and that will be handled via the "UAT Bug Fix" Task. Handling UAT Bug Fix will surely have an impact on the velocity of the Sprint, so the team needs to retrospective on how to minimize UAT bugs.

For the analogy, I will refer to the same line quoted by Daniel Wilhite

"I'm also going to ask what the difference is between a change being requested or bug being found during UAT and the same happening from Production? If there is a difference in the behaviour, that might be a good place for you to focus and identify why. All are coming via stakeholder feedback and all are going to impact the Product Backlog in some way."

Thus the feature delivered in current Sprint should be "usable" as it is crossed QA testing by the Product Owner and QA team

Is it usable though? Have the Developers of the Scrum Team assured its immediate usability no later than the end of the Sprint? Who uses this feature without the Developers -- or anyone else -- having to complete additional undone work?

I think you missed what I was saying. 

I advocate that there is no such thing as a bug.  I encourage my teams not to log bugs.  To me everything is an opportunity to improve the product.  When you view items as different classifications you always end up with processes that try to accommodate it with different ways of ordering, prioritizing, addressing.  This just leads to more complexity and waste in your process.  Everything in your Product Backlog should be viewed as a Product Backlog Item.  Write them in the same format. Provide the same information. Discuss and handle them all equally.  This will ensure that the Scrum Team is maximizing the value of the work they do.  When you stop seeing things as bugs and see them as opportunities, everything has a value.  

Any item in the Product Backlog has intrinsic value.  There is a negative value as long as it is still waiting to be addressed and positive value once it is "done".  Even something that is considered a defect matches that statement. For example, something labeled a defect might be so terrible that it prevents the users from being able to work in the product.  Fixing that has value unlike what your business has said.  But the situation where a word is misspelled or a wrong word is used does not have such a high impact and the value to fixing it is less. Stop viewing it as a defect and start viewing it as an opportunity to make the product more usable and that value starts to be easier to see. 

Your proposed solution does not seem like it addresses your problem at all.  As @Ian Mitchell points out, your thoughts are focusing on the wrong problem. 

If you are working in a Scrum framework,

UAT is continuous on a PBI level basis, not on an Increment basis.

Otherwise you're getting the "increment" to "95% done" - none of it is "done" at all - until after some milestone event where all PBI's do their final 5%. And if one fails, you hold up the whole increment.

That is the flaw here. Finish PBI's continuously. Don't release PBI's to a "ready for one more activity" then move on to the next PBI: finish each PBI.

When the Sprint Goal is met, the Increment is "done" - which does not preclude working on stretch candidates, value adds, ...

Make your sprint goals more achievable until you get to a point where you are completing the UAT before moving from one PBI to the next. Get to a point where you're exceeding the Sprint Goal and if some of your "stretch" PBI's (which help the team exceed the sprint goal... not only does it function, it functions more smoothly, with better performance, with a prettier interface, with better (not just "adequate") descriptive text...) don't make it nobody is upset by that.



Review is the opportunity to review not only the PBI's that achieve the sprint goal but also the ones that take the team beyond that goal. Moving past the goal is a good thing. It demonstrates there was inbuilt slack.