In my current role I for the first time worked with user stories and had to think about how to test them.
“Main goal of testing during the scrum is not to find bugs but eliminate them early and collaboratively.”
Acceptance Testing User Stories
Acceptance criteria are used define when a story is done.This criteria will be used to define acceptance tests which proof the story was coded and implemented as defined by the acceptance criteria.
Ideally the acceptance criteria is added to a user story before coding of a story starts, as acceptance criteria can provide vital information and guidance regarding the scope of the functionality. But this doesn’t mean the criteria is static. Acceptance criteria can also evolve during a sprint.
Typically it is written/updated whenever a customer/stakeholder and member of the team talk about the story and may capture further details.
Discussed details will be reflected/implemented as tests.
Whenever new tests are discovered during the development process, acceptance criteria may be adjusted.
Good questions to help define acceptance criteria:
What am I assuming about the implementation of the story? ie. Get the stakeholder to outline what they envision (do not show them ideas or examples as this may lead them, leave those for discussing the acceptance criteria).
Are there circumstances that affect this story meaning it may behave differently?
What are the risks/potential challenges with this story?
Acceptance criteria and tests need to be specified by the customer/stakeholder. This can be in collaboration with a developer/tester but initially the customer/stakeholder should define when the story is regarded as done.
Testing of the acceptance criteria needs to happen alongside the development process and not when it is done, as testing can provide valuable information if the story is being implemented how the customer/stakeholder envisioned it.
Can we have too many tests?
A customer/stakeholder should continue to write tests during a sprint as long as they add value and clarity, with the focus on clarity. Once the tests do not add any clarity or value in defining/verifying when the story is done, the writing of tests should stop.
The developers should focus on having a good set of unit tests to verify the code functions as described in the acceptance criteria, and if possible also test some of the acceptance tests using unit tests.
Ideally the customer/stakeholder should execute the acceptance tests as these were defined by them/in collaboration with them and they need to sign off the implementation of the story. When using scrum methodologies however, often the customer/stakeholder will not actually tests the user story but the implementation will be reviewed in a sprint review meeting with the whole team.
Furthermore customers/stakeholders are often busy and as the software grows this task of manually testing the user stories becomes unmanageable. This means the team needs to put effort into automating the acceptance tests where possible.
This starts with unit tests, moves to business tests and perhaps also web tests, depending on the story. The focus should be on unit tests, however as these provide the fastest feedback to the team and also means maintainability of code is improved as more features are added.
So we can have too many tests when they start to convolute what a story should be doing. The focus needs to be on proving the acceptance criteria is being met and high risk areas are being covered.
Where should the testing focus lie when acceptance testing user stories?
First of all the focus will be functional testing as this will aid to confirm the acceptance criteria laid out. Ideally these tests are automated. But there are other types of testing that may need to be considered that are not necessarily automate-able.
1. User interface testing – does the user interface look and behave as expected?
2. Usability Testing – can the software be easily used?
3. Performance testing – how does the application perform under different workloads?
4. Stress testing – how does the application perform under extreme situations?
Due to time constraints of a sprint and to be able to support fast development of new features, there cannot be a 100% test coverage, but areas of risk should be defined to focus the testing effort on, especially if that area cannot be automated at a unit test level.