Is it necessary to maintain screenshots during the test execution?

Agile should avoid unnecessary complexity, but it doesn't mean "without documentation."  You mention that you don't feel this goes against the agile principles, but doesn't documentation support transparency and accountability for the development team?

Perhaps there is a better Definition of Done your team could use that doesn't involve tons of screenshots?

How does the team currently ensure that all of the acceptance criteria have been met, and why isn't the team's process thought to be justified for the purposes of auditing?

Good topic :)

As a practice, our company has required screen shots as evidence of testing as we have been doing manual functional testing.   And I am just guessing that has come as input from our auditing firm since they carry a fair amount of weight here.

The recent question that has come up is the same required if we are doing automated functional testing.   Real quick google search on the topic today does not provide a clear response for best practices.   I did find someone suggest logs from the testing if not screen shots.

As everything related to auditing it depends on what the auditors expect. According to what standard/system/... are you being audited?

In my experience automated testing is usually evidenced through testing logs. Most automated testing frameworks generate those by default anyway. Depending on the standard your being audited under, this may require the Development Team to review that the automated test really does test the acceptance criteria in question. This can usually be done informally.

If the auditors have concerns you need to talk about it with them in the interest of openness (which is as we all know a Scrum value). Find out what it is they really need and find a way that works for both sides. This will involve your team making suggestions, because sometimes auditors are unaware of the possibilities at your disposal. If they're used to screen shots they may ask for those because they're used to it. If you show them how test logs will also give them evidence of testing, they might agree with you.

I would have thought a copy of the logs would suffice to prove that the tests have passed. Or if they really want screenshots, can you not run all tests at once and then screenshot the final page showing they all passed?

It isn't the responsibility of auditors to prescribe how an agile team should evidence its work. Auditors should instead seek to assure that the method a team uses is being applied consistently and to the level of quality being claimed. The wider organization may set certain standards, but it is not up to auditors to define what is done.

- For manual testing (such as exploratory tests) the team may decide to record screen-shots, or to describe the scenario being tested and check it off, or to author an automated script which is added to the regression pack. Auditors would assess the clarity of whatever policy they had and the consistency with which it is applied.

- For automated testing, logging ought to be in effect and sufficient for the purposes of auditing. If it isn't then the level of automation is inadequate.

It's not required to maintain screenshots for test execution but we also have this process in our scrum projects. All scenarios are executed when new functionality is added to product. This makes sure that old functionality still works when new functionality is added. However it's tiresome to take screenshots repeatedly. We are planning to start automated UI testing to find regressions easily.

Thanks for your comments all, am passing along internally as part of the conversation here.