Testing Talk – Notes from "Testing in a Continuous Delivery World"

Working as a lone manual tester with an agile team that consists of 2-3 development streams, plus potentially other side projects, I have recently looked towards other testers in similar situations to see how our processes could be streamlined.

I found this awesome talk by Amy Phillips on how Songkick works with Continuous Integration.

To be able to consider continuous integration we have to address our bottlenecks and see if this could be solved.

Pointers I have taken are the following from the talk but do watch it yourself! (I have the odd reflective thought here and there but for now this is just what I took from the talk as I am still learning.)

1. Testing provides answers to questions
So testing is part of the whole project as you gather information the whole time from the first mock up to conversations over having tea.

2. In testing you think of the whole picture
All everyday activities are testing as you are confirming or not confirming expectations of the project.

3. Consider who is invested in the project and what they are expecting
This is not the user necessarily but others are involved too, lots of people have an interest in the system.  

4. When automating tests make sure they do fail as well as pass
Always consider the context also and sanity check the state of the system.

5. Checking vs testing the result
Only humans can judge contexts, user expectations and needs, machines cannot think.

6. Automation myths – 100% can be automated
This for one is not possible and for the other we need to be testing and not just checking the system.

7. Not everyone experiences bugs in the same way
Bugs could become obvious later such as scalability issues.

8. To move to continuous integration or not we would have to evaluate our values as a company and development team
  • what are our values?
  • ship new features asap?
  • developers responsible for their code in production?
  • if you are writing code it needs to be good enough for production quality?
  • what would the test strategy be? A team approach?

9. Test strategy would need refining – maybe follow these pointers
  • remove overlap
  • make sure testing is not time wasting – efficient automation
  • team agrees – whole team approach – define how to split the testing effort
  • how much testing needs a human aspect and what is “just” checking?
  • tests should be risk based
  • team is the key word – have everyone test – designers test designs – PO their acceptance criteria – would designers be too involved in their designs to test efficiently? Where is the whole picture approach that a tester would contribute?
  • freedom to decide how to cover the testing for each project
  • testers responsible for quality – helping everyone test – more of a consultancy role perhaps?

10. Fast feedback

  • question things continuously
  • shared ownership of testing means it goes beyond one person or team and is constantly a focal point meaning quality is a focal point from every angle
  • every team member has a say and an influence on testing and therefore development
  • remove duplication of automation test
  • benefits in cutting tests down include faster more reliable feedback – by cutting down on the automation tests everyone can have an overview about what each single test does
  • everyone knows what each test does and how important the test is – why is it being and tested and what it tested

11. Acceptance tests become journeys defined by the business

  • user journeys
  • realistic user journey defined by the business
  • limit the journeys and all tests have to finish in a certain time – 10 mins? Realistic for us?

12. Perform a risk assessment
  • what are we trying to achieve
  • identify risks – too risky? what is the potential return?
  • agree on how to mitigate risks – more manual testing?

13. Flexible release pipeline
  • testing the right things for each feature – how are they defined?
  • feature toggles to do exploratory testing on production – we already do this but does it add too much overhead?
  • test data in production environment – benefits include realistic test scenarios
  • feature toggles need to be accessible from UI  – is an admin tool enough?
  • can control type of user to view the new code and use it – involve specific user group or just turn features on for users that sort of fit the desired user?
  • trust the test results – let jenkins make the decision – do we trust our automation?
  • be aware and honest about flakey tests such as integration tests with other systems
Execute the right tests at the right time!

14 . Tools – Use what you have
  • Brain! Think, explore, question!
  • use people from different aspects of the business – everyone has expertise and knowledge
  • Be brave and have the awkward conversations
  • Define a “what if it goes wrong” strategy – role forwards/backwards? who will respond? more monitoring
  • Talk! A Lot!
  • Team work – continuous integration cannot be individuals

15. Benefits

  • everyone cares about testing and quality
  • motivated to maintain automated tests
  • understand the value of the pipeline
  • new area of code – need to ensure coverage becomes a natural thought process
  • bug fixes just take minutes
  • respond to tweets – “Thanks fixed!”

Automation doesn’t make you safe!

When things go wrong use it to make a positive change!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s