Thoughts: Test automation strategy

I am a bit behind the curve on this I am sure but the free platform Test Automation University by Angie Jones is pretty great. There even is a more high level short course on test automation strategy. I was super grateful for this, because of a few reasons which I will cover in a moment.

Why was I looking at the platform in the first place? Well we have objectives around test automation. (A discussion on objective setting and personal development plans could actually be a whole series. I still very much refer back to Stephen Janaway and what he wrote a while back on the subject and 1:1s. Read the whole series, it is great!)

Don’t assign objectives or tasks if the person doesn’t have time or isn’t motivated

But back to the objectives. Now it is pointless assigning someone an objective or personal development task when they are not motivated to act on it. If someone does not want to learn to code or cannot put in the time for whatever reason, then there is no point in them needing to achieve actually writing some code. At the end of the day we want our teams to succeed and feel like they are growing in their role. Now for some that is simply being a valued team member on their team. But we do like pushing people and letting them explore their role from different angles.

As a company approach we want to aim for continuous delivery where feasible (regulations), which means more automation. But not many of us can or want to learn to code.

So we decided to split our automation focus and take on the approach of not just doing but also understanding the whole approach. This is something I really liked about the automation in testing approach by Richard and Mark. It gives test automation another dimension because it no longer is about automating away testing tasks such as regression but also understand why you are doing it and what other things it enables you to do and how it supports testing activities.

Test automation is a whole team approach

Sometimes though these sort of words need to be in a course or some sort of material for people to accept it and listen to it. So finding Angie’s course on test automation strategy makes the whole subject of test automation way more approachable. Most of us testers lead scrum ceremonies or meetings. We have to be good communicators, so armed with some great questions from that course, anyone can help lead a discussion on a test automation approach to make sure it becomes a whole team approach.

Not being a coder can be such an advantage when designing a test automation strategy

Not being a coder can be such an advantage when designing a test automation strategy, not a disadvantage. I have seen people feel intimidated and scared but all the technical details, but being a functional tester often means you have a whole picture of the application and can really help the team automate the more risky parts of the application and the most valuable to the business.

I found that functional testers can easily feel less valued than the ones who can write automated tests and by also focusing on understanding test automation it made this objective a lot more approachable in my mind, and shows that you can still sit at the automation testing table, even if you cannot code the test. It might even free your mind. You don’t have a predetermined level of “automating that is hard so I won’t even go there” mind block. You are free to ask all the questions around what is possible and then is it feasible given the estimated effort.

So high level what do you need for a test automation strategy? Well watch Angie’s course. ūüėÄ

My summary of the first video with some of my own thoughts below.

WHAT?

A goal! What is your goal? This made so much sense to me. This can help you identify where are automation efforts are most value-able.

The most common goal I have seen is to reduce regression testing time. But here it can be important to then not jump to conclusions. Just because regression testing happens by manually clicking through the UI, it does not mean your tests need to do the right thing. Be clever about which layer is gets the automated tests and what your regression tests actually check and think about the least costly design that is most effective and build on that.
WHO?

Brigh#Testactually and terrible UI tests

It’s been a big quiet here recently and this had to do with me changing job and trying to get used to a commute. Luckily our meet up didn’t suffer and thanks to the co hosts we had another great evening.

This time we had a talk on terrible UI browse tests in selenium and how to not write them in a terrible way.

 Rich the speaker, in my opinion made a great point that got me thinking a little bit further.

He mentioned the automation test pyramid and how at the top it always states UI tests. However what it actually means is end to end tests run through the UI that test the full stack of code.

You could even argue that end to end tests are a sub form of UI tests because really UI test do not need to be run against the full stack of code if you mocked the API calls and other third party integrations.

So the UI could be written with unit type tests from day one run on the CI server if it was a separated layer.

Do we distort the message of what types of tests we want at the top of the automation pyramid by letting us be boxed in by its shape?

I think we shouldn’t necessarily just put UI tests at the top. It should be end to end tests run through the browser. Is it just me that also thought UI tests have to take a long time, be brittle and use the full stack?

It hadn’t occurred to me that if we separate the UI layer and apply a few other tricks we could test just the UI fast and free up manual tester time even more to explore those edge cases.

Do you think we boxed our definition and understanding of what UI tests are in due to the shape of the pyramid?
The slides from Richs talk

Thoughts: Testing Strategies in a microservice Architecture – takeaways and reflections

“The Causeway”



This post based on some research we did about hypothetically moving to a micro service architecture/soa. 

This first post is a reflection on an article by Martin Fowler.
My personal problem with these testing approaches to a micro service architecture  is that a lot of it is automated and in isolation. Automation is good as it allows us to make sure we are building the code right but we need manual testing a lot earlier to make sure we are building the right code.
There are many benefits with this approach such as the ability to independently deploy, scale and maintain each component and parallelize development across multiple teams. However, once these additional network partitions have been introduced, the testing strategies that applied for monolithic in process applications need to be reconsidered.

Testing Pyramid РHow to focus the testing effort 

This is actually not that different from other automation test strategies I have read.


Aim to provide coverage at each layer and between layers with the challenge being to stay lightweight.
The first three layers are best tested in isolation where possible. Component and integration tests may need to call on other services but these can be stubbed to keep tests fast and maintainable. 
Any external service that is out of the team’s control should be stubbed/mocked in order to test failure scenarios.
One key characteristic in this type of architecture is graceful failure! The system must fail gracefully as it is prone to fail in many places.

Testing micro services in isolation:
Benefits:
Fast feedback as no end to end test.
Controlled environment – minimizes moving parts
Unit tests can help to identify areas which are too coupled as the unit under test needs to be isolated from its collaborators.

Resources for testing:
Expose internal resources such as logs, database commands and metrics to be able to test the system from another layer

Problem:
Business value not achieved unless it is proven that many services work together to fulfill a larger business process, so testing in isolation will not suffice.

End to end tests:
Automate end to end tests using the GUI and API as well as perform manual end to end tests. But external services may still need to be stubbed/mocked. Due to more moving parts the system is likely to fail, so write less end to end tests at GUI level to avoid expensive maintenance and feedback times. Are these story acceptance tests?
Need to define a time we are happy to wait for these to give feedback – 5 mins?
Best tests would be data independent – not relying on any data to have been set up as this makes the tests more brittle
In which sort of environment do we run these tests? Is in production with a feature toggle feasible? Do we keep a production like environment running with the downside of double the overhead to technically have 2 live environments?

Exploratory manual testing:
These are manual tests that are performed with a mission statement in mind, they are session based ideally; time boxed from 45 to 135 minutes. What sort of environment is feasible for exploratory tests? Production environment with a feature toggle?

Questions:
Have you solved the conundrum of testing micro services sufficiently in a continuous integration world? Please send me any resources or thoughts! Would be very interested to read them! ūüôā