Thoughts: Test automation strategy

I am a bit behind the curve on this I am sure but the free platform Test Automation University by Angie Jones is pretty great. There even is a more high level short course on test automation strategy. I was super grateful for this, because of a few reasons which I will cover in a moment.

Why was I looking at the platform in the first place? Well we have objectives around test automation. (A discussion on objective setting and personal development plans could actually be a whole series. I still very much refer back to Stephen Janaway and what he wrote a while back on the subject and 1:1s. Read the whole series, it is great!)

Don’t assign objectives or tasks if the person doesn’t have time or isn’t motivated

But back to the objectives. Now it is pointless assigning someone an objective or personal development task when they are not motivated to act on it. If someone does not want to learn to code or cannot put in the time for whatever reason, then there is no point in them needing to achieve actually writing some code. At the end of the day we want our teams to succeed and feel like they are growing in their role. Now for some that is simply being a valued team member on their team. But we do like pushing people and letting them explore their role from different angles.

As a company approach we want to aim for continuous delivery where feasible (regulations), which means more automation. But not many of us can or want to learn to code.

So we decided to split our automation focus and take on the approach of not just doing but also understanding the whole approach. This is something I really liked about the automation in testing approach by Richard and Mark. It gives test automation another dimension because it no longer is about automating away testing tasks such as regression but also understand why you are doing it and what other things it enables you to do and how it supports testing activities.

Test automation is a whole team approach

Sometimes though these sort of words need to be in a course or some sort of material for people to accept it and listen to it. So finding Angie’s course on test automation strategy makes the whole subject of test automation way more approachable. Most of us testers lead scrum ceremonies or meetings. We have to be good communicators, so armed with some great questions from that course, anyone can help lead a discussion on a test automation approach to make sure it becomes a whole team approach.

Not being a coder can be such an advantage when designing a test automation strategy

Not being a coder can be such an advantage when designing a test automation strategy, not a disadvantage. I have seen people feel intimidated and scared but all the technical details, but being a functional tester often means you have a whole picture of the application and can really help the team automate the more risky parts of the application and the most valuable to the business.

I found that functional testers can easily feel less valued than the ones who can write automated tests and by also focusing on understanding test automation it made this objective a lot more approachable in my mind, and shows that you can still sit at the automation testing table, even if you cannot code the test. It might even free your mind. You don’t have a predetermined level of “automating that is hard so I won’t even go there” mind block. You are free to ask all the questions around what is possible and then is it feasible given the estimated effort.

So high level what do you need for a test automation strategy? Well watch Angie’s course. 😀

My summary of the first video with some of my own thoughts below.

WHAT?

A goal! What is your goal? This made so much sense to me. This can help you identify where are automation efforts are most value-able.

The most common goal I have seen is to reduce regression testing time. But here it can be important to then not jump to conclusions. Just because regression testing happens by manually clicking through the UI, it does not mean your tests need to do the right thing. Be clever about which layer is gets the automated tests and what your regression tests actually check and think about the least costly design that is most effective and build on that.
WHO?

Real Life: How to survive commuting?

 

Picture taken on the commute home


I watched a presentation by @medickinson on presenting/public speaking for introverts. She makes a great point on how introverts get over stimulated more easily than extroverts.

There is actually some science behind it in terms of how we actually have different tolerances to the stress hormone dopamine.

During the presentation of this data I realised that maybe I am not imagining that going to big cities makes me really tired. Big cities can be really overstimulating.

We are presented with tonnes of adverts, it is noisy, there are tonnes of people, there is less space than in the countryside. Just now I realised that me being an introvert means that it is no wonder that this tires me out.

This gave me the idea of the introverts survival guide to commuting.

To put into perspective I have commuted locally in all sorts of ways, by bus, walking whenever I can, cycling, driving etc.

But what I had never done before was travel really far and do that by train.

This changed 3 months ago, when I joined Songkick in London.

With no intention of moving to London I knew this meant tackling the commuter train.

 

My platform at 6:30am


The commuter train to and from London is sort of special. You get different types of people at different times.

The earliest I have managed is 6:31 to London and it is full of snoring people and zombies.

Today someone was asleep opposite me with his eyes open! Scary!

I am also a sleeper even at 7am, but I wasn’t initially. I think the actual commuting and being in the city has made me so tired that I need the morning to myself sleeping.

In case sleeping in carriage full of people worries you, here are some other survival tips:

1. Noise cancelling headphones – Great for those noisy snorers, loud eaters, or blocking out the ones that actually talk to each other!

2. Kindle – light, easy to hold and read. A book will do as well but can be bulky to carry around.

3. Tablet/Phablet – download videos onto your tablet or phablet. This can be some great distraction, or even learning experience. I download Ted talks on a regular basis. 🙂

4. Podcasts –  my new favourite! Podcasts are a must for at least the morning if not evening as well. My general favourites are: Criminal, Your are not so smart, Serial, This American Life and Welcome to Nightvale.

5. Music – maybe even try new music! You will hardly ever get an hour to yourself to just listen to something new. Then when you like them and want to see them live near you, why not download the songkick app.!? 😛

6. Good backpack – What do I mean by good!? Well it depends on your use cases but I need one that is cushioned, has a laptop compartment that is cushioned, big general compartment for running gear, lunch, water bottle and other bits and pieces. Waterproof is a major win as well, especially if you walk around London a lot like myself. I also like wide and well cushioned straps that you can also fasten across your body. I know the look sucks but if you are late and need to run you don’t want the bag to jump all across your back.

7. Water – I have broken out water separately here because recently I have been feeling faint on the train, especially when I have had to stand up. It is very important to keep hydrated during your commute, kids!

8. Snacks – In similar vain as water, you never know how long your commute may be. In three months I have had 4 or 5 trains be on time. Carry some snacks with you in case you don’t make it home in time for dinner.

9. Notepad and pens – The commuter train can be a great place to unwind and let your thoughts wander! You never know what great ideas you will have. Make sure to have pen and paper to hand to write them down!

10. Knitting – Now I unwind with crafting. I can’t really sew on the train (I don’t do hand sewing) so I knit. I have so far made a cowl, a hat and working on a jumper. It can be really nice to not feel like you are wasting your time by just waiting.

 

This has been knitted 80% on a train


11. Coffee – Now a good coffee can make the day! Make sure you are friendly to your coffee provider so they start recognising you and make you your usual! This saves time in those moments where you got out of bed a few mins later than usual and you can still catch your normal train!

 

Coffee! (although I have since given up – no coffee for a while now)


Also keep your stamp cards. Commuter train coffee can really drain your pockets!

12. Apps and Social Media – make sure your trainline’s twitter account and news page is easily accessible! You never know when there may be too many leaves on the tracks causing a delayed or cancelled train! You will be thankful for making checking these part of your morning routine, because they may enable you to have another half hour snooze!

These are some ideas to get you started. I left out actual work here, but do feel free to do some work on the train or write your next blog post up, as I am doing. 🙂

But let me tell you it was not easy! Now after three months of commuting it is getting better, but I struggled with exhaustion, travel sickness, leg cramps (suddenly walking 8-10km a day) and I even passed out on the train (this showed me that commuters can be nice and helpful).

Brigh#Testactually and terrible UI tests

It’s been a big quiet here recently and this had to do with me changing job and trying to get used to a commute. Luckily our meet up didn’t suffer and thanks to the co hosts we had another great evening.

This time we had a talk on terrible UI browse tests in selenium and how to not write them in a terrible way.

 Rich the speaker, in my opinion made a great point that got me thinking a little bit further.

He mentioned the automation test pyramid and how at the top it always states UI tests. However what it actually means is end to end tests run through the UI that test the full stack of code.

You could even argue that end to end tests are a sub form of UI tests because really UI test do not need to be run against the full stack of code if you mocked the API calls and other third party integrations.

So the UI could be written with unit type tests from day one run on the CI server if it was a separated layer.

Do we distort the message of what types of tests we want at the top of the automation pyramid by letting us be boxed in by its shape?

I think we shouldn’t necessarily just put UI tests at the top. It should be end to end tests run through the browser. Is it just me that also thought UI tests have to take a long time, be brittle and use the full stack?

It hadn’t occurred to me that if we separate the UI layer and apply a few other tricks we could test just the UI fast and free up manual tester time even more to explore those edge cases.

Do you think we boxed our definition and understanding of what UI tests are in due to the shape of the pyramid?
The slides from Richs talk

Thoughts: Testing Strategies in a microservice Architecture – takeaways and reflections

“The Causeway”



This post based on some research we did about hypothetically moving to a micro service architecture/soa. 

This first post is a reflection on an article by Martin Fowler.
My personal problem with these testing approaches to a micro service architecture  is that a lot of it is automated and in isolation. Automation is good as it allows us to make sure we are building the code right but we need manual testing a lot earlier to make sure we are building the right code.
There are many benefits with this approach such as the ability to independently deploy, scale and maintain each component and parallelize development across multiple teams. However, once these additional network partitions have been introduced, the testing strategies that applied for monolithic in process applications need to be reconsidered.

Testing Pyramid – How to focus the testing effort 

This is actually not that different from other automation test strategies I have read.


Aim to provide coverage at each layer and between layers with the challenge being to stay lightweight.
The first three layers are best tested in isolation where possible. Component and integration tests may need to call on other services but these can be stubbed to keep tests fast and maintainable. 
Any external service that is out of the team’s control should be stubbed/mocked in order to test failure scenarios.
One key characteristic in this type of architecture is graceful failure! The system must fail gracefully as it is prone to fail in many places.

Testing micro services in isolation:
Benefits:
Fast feedback as no end to end test.
Controlled environment – minimizes moving parts
Unit tests can help to identify areas which are too coupled as the unit under test needs to be isolated from its collaborators.

Resources for testing:
Expose internal resources such as logs, database commands and metrics to be able to test the system from another layer

Problem:
Business value not achieved unless it is proven that many services work together to fulfill a larger business process, so testing in isolation will not suffice.

End to end tests:
Automate end to end tests using the GUI and API as well as perform manual end to end tests. But external services may still need to be stubbed/mocked. Due to more moving parts the system is likely to fail, so write less end to end tests at GUI level to avoid expensive maintenance and feedback times. Are these story acceptance tests?
Need to define a time we are happy to wait for these to give feedback – 5 mins?
Best tests would be data independent – not relying on any data to have been set up as this makes the tests more brittle
In which sort of environment do we run these tests? Is in production with a feature toggle feasible? Do we keep a production like environment running with the downside of double the overhead to technically have 2 live environments?

Exploratory manual testing:
These are manual tests that are performed with a mission statement in mind, they are session based ideally; time boxed from 45 to 135 minutes. What sort of environment is feasible for exploratory tests? Production environment with a feature toggle?

Questions:
Have you solved the conundrum of testing micro services sufficiently in a continuous integration world? Please send me any resources or thoughts! Would be very interested to read them! 🙂

Thursday@EuroSTAR – My notes and some pictures

These notes are a bit delayed. Sorry. And also a lot more ramble-like. I hate it when the real world just swamps you and you feel like you cannot do things as well as you may want to. It is not all bad though. 🙂
Hope you get some value from this!

Shmuel Gershon, Intel, Israel


Do we understand what the real value of software is?
Civilisation runs on software. Dependent on software!
Everyone stores, transfers and applies knowledge in their life.


Civilisation itself depends on testers now. We look for the most faithful representation of knowledge possible in software.
Testers are therefore researchers of knowledge.
Software really is endangering civilisation as we know it – big responsibility of testers!


We need to care about users and what they are looking for.


We come to conferences to save civilisation and for the beer! (Or in my case the food).



Testing Traps to Avoid in Agile Teams

Janet Gregory, DragonFire Inc., Canada
traps and risks due to them
waiting for the build – why arent we getting stories to done – next iteration testing
mini waterfall – ketchup effect
tests not complete – dod not reached
testers lose credibility
tech debt
plan for test infrastructure – build pipeline
testing taks in velocity
devs used to instant feedback
test where it makes sense
Testers not part of the team – be active
risks – wrong assumptions, team becomes divided, skills missing
become useful, power of three
testing is a team responsibility
quality police – do not want the power to stop a release into production
communication needs to be face to face
devs use testers as a safety net
testing as a service – consulting mindset
everyone needs to pick up testing tasks
technically aware
make testing visible
help deliver software successfully
manual testing – agile not sustainable without automation – minimum of run once a day
include automation time in estimates
encourage collaboration
understand power and risk of what you are automating
business logic back in the UI
big picture
devs add extra code after they are finished
integration too late
mind map of the big feature
more examples from product owners
story done and feature/release done



Diversity in your team – embrace it or lose the best thing you have!

Julie Gardiner, Redmind AB, Sweden
strength lies in differences not in similarities
get the balance right
Dreyfus model for skills aquisition
people at different skill levels need different management
excercise
friendly
approachable
business like
open
organised
social
logical
random
warm
perceptive
7


to the point
accepting
quick
4
pragmatists – efficiency – direct
facilitator – togetherness
the analyst – accurate
the pioneers – enthusiastic
belbin role types


Zeger van Hese, Z-Sharp, Belgium


interconnectedness – triangles
impossible objects
comfortable clone syndrome – avoid as it squashes diversity – testing needs diversity to find different types of problems
control dilemmas
randomness increases variety and serendipity – finding something valuable while looking for something else
creative vs critical thinking
we copy to gain understanding and knowledge which we can then transform and apply
everything is a remix
leadership makes the diverse mix work
empower people to do their best work
leave people in control
recognise good ideas



Programming for Testers – It is Easy!

Graham Thomas, Independent Software Testing Consultant, United Kingdom
don’t mix data types – str()
single or double quote
python 3 uses ()for strings


first line called the shebang – comment # ignored
first line is special #! = shriek
shows where python environment is

Some pics:


The TestLab!
I also want to give a shout out to the TestLab team! They were fun, engaged individuals working as a team to provide puzzles and exercises and brought a whole new level of engagement to the event!


What do we do after the conference?
Now comes the difficult part! When you are home how do you implement any of the great thing you learned without just suffering the come down from the conference?


I started with just a question:
Are we giving our customers the right information for the task?


Did you ask a question? How are you applying your newly gained knowledge?

Webinar: Integrate Test Activities in Agile Projects

Integrate Test Activities in Agile Projects


I attended the webinar from Rik Marselis about integrating test activities in agile projects (slides).
The topic caught my attention because I am currently solving this issue in my job especially around manual test activities.
The existing process used to focus mainly on web tests but has evolved to include unit tests and now also includes manual tests executed by a professional tester instead of just the PO.

The talk started with the acronym QD.

I had never heard “Quality development” as a term before when referring to agile development. Maybe it was implicit but I could relate to QD being described as “Quick Development”, especially when the emphasis was on just getting features out there quickly, to be revisited at a later date to improve usability.

In my experience this does not happen unless there is a blocker as in the new features becomes unusable for some reason or causes too much of a maintenance headache. 

I recently did a short talk at my company about Agile testing and how we are trying to integrate test activities (especially manual testing) into the scrum teams, and like Rik I also referred to the Agile Manifesto.
He, unlike myself, argued that it does not have strong footholds for testing in an agile development project. I would argue that it includes the main parts for testing to be successfully integrated namely, people and interactions, collaboration and responding to change. if these ingredients are not in your scrum team than testing cannot be successfully integrated in my opinion.
You need the team to be interacting and sharing to make testing possible and to avoid duplication of the testing effort. This goes hand in hand with collaboration and responding to change. As issues are uncovered the team needs to work together to fix these/make changes if new requirements are uncovered.


Rik went on to mention the Scrum guide which I have not read in detail and hence will not add anything to right now, but this did yield a couple of valuable pointers.
Testing is described as inspections performed by skilled inspectors with adjustments made asap and each increment being thoroughly tested.
Thoroughly is a word which actually rubs me up the wrong way and Rik seemed to agree. “What does thoroughly mean?” 100%? 100% test coverage is not possible, who decided what is thorough and adequate?


The development teams I have been involved in recently talked about a definition of done in terms of if all development streams are on the same page. We mentioned code/features/stories to be adequately commented/documented/tested (automation and manual).
I understand that in scrum methodologies these sort of guidelines for a definition of done need to be flexible to an extent but using words such as adequate and thoroughly when these are subjective seemed incorrect to me. Maybe more on that in another post.

Add caption



Testing during agile development should be about providing an insight into quality and risks. Defects should be identified as soon as possible, with exploration into root causes to avoid them in the future.
Through exploratory testing we  should yield re-usable test cases as each increment should be thoroughly tested and will therefore need a regression testing phase.
Keep testing transparent!
This is feedback we have had in my current teams that the developers were unsure about what I actually do. We try to combat this by writing test charters and acceptance tests and circulating them for anyone who wants to read them/use them as broad guidance to ensure code test coverage.
This follows onto the next point that was made that testing is a skill that should be integrated into the team, preferably in the form of a skilled/professional tester. Long gone ae the days of sub teams. We need to make the transition of an us vs them mentality and become a valuable asset of the dev team. By thinking outside of the box and like a user, testers can integrate into agile development teams to provide another set of skills.
However testing is not just a developer/tester activity. Test integration also needs the PO to review/test the stories and features, ideally before the actual end of sprint review. This can ensure they accept the result of the sprint as at the end of the day they will make the decision if a sprint was successful.

During planning the Scrum Master can act as a tester helping to define acceptance criteria and making sure the proposed designs are fit for purpose and will create a product of high quality. This process of questioning can minimise rework for the whole team and ensure they are focused on the same outcome.

This small list from the talk highlights the whole team approach that is so important for success.
To be fully integrated testing needs to be part of each step of the way, during planning to identify risks and start planning which tests and test tools may be needed. then while the features are developed, the tester creates and maintains their tools which ideally can be used by the whole team.
The test strategy will not just depend on the project but also on the sprint and tasks associated with this. To make testing more transparent Rik mentions adding tasks around testing to your sprint/scrum boards. I may well do this and try it as part of our next project as the team is growing and it would be nice to offer some guidance on what is needed to perform the tests required for that sprint and let other testers take over the creation of test data/environments etc. Tester in the context here also includes developers, Scrum master and POs.

The pitfall of a test strategy according to Rik, is to focus solely on unit testing or functional tesing only and how to automate them. Manual testing can expose certain scenarios that machine simply cannot and hence we need to consider the user perspective.

It may be useful to think of different testing objectives instead. Objectives of each level of testing need to be considered when defining and executing testing. What should the unit tests cover/ensure? What should the web tests cover? What risky areas should the manual tests look at? Is manual testing focused on usability?
We try to manage unit and business tests during the sprint and automate them accordingly. Also each story or feature has a web test added if appropriate and is tested manually.
We then perform full end to end tests when code is about to go live on our most like production environment (however this still could be closer to the actual live environment).
Throughout this process we run automated regression tests also, because manually regression testing is simply not possible anymore.

Like development we need to look at testing when integrating it into an agile development cycle to be sustainable. integrate activities as early as possible and as many layers of testing as possible. Assign tasks for the different layers to skilled team members, ie. unit tests are written by developers. testing needs to be considered to be a task of the whole team.

With that in mind consider reviews as a form of testing that can ensure quality from day 1.

I took a lot from this talk and hope to put forward the idea of also tasking up testing tasks to make them more visible to the team. I think this could be valuable as it can also be a good training tool and won’t be micro management. Certain tasks can be split across team members (not just testers) and reviewed in a peer review before being considered to be done.
This would offer a platform for learning and sharing in my opinion which is a big part in successfully integrating the testing effort into the team.