Thoughts: Performance testing types

3 years ago I spoke about my experience with performance testing at Songkick, and to be honest I loved this specialism and discipline. It combines two of may favourite things, how users behave and how the system behaves in different conditions. At songkick it was fascinating to flick the switch for a an artist’s ticket sale and seeing how different fans would use the system differently, not just if they were using mobiles or certain browsers but what did they do? Do they keep refreshing, going back and forth through the journey? Are they happy to just buy 1 ticket or always going for multiples?

I just love combining the data of what actual users do and how they behave to model different types of performance tests.

And cover-19 has actually meant for a few reasons that I get to think about this again. The reasons are:

  1. I no longer commute so have a bit more time (my commute was very small though).
  2. Our users used our systems more for a period of time due to several reasons
  3. I have changed teams for a short while -giving me a tiny bit of breathing and much needed thinking space.

While we do not do much performance testing in my team, at least there isn’t a proper strategy (it has been my retro item for a few weeks to create a strategy), we have got external teams with the ability to, and it is their role to test all the systems together.

What excites me is that performance testing let’s you get under the skin of the application. You need monitoring and analytics in place to watch what happens and then you get to look at all the layers of your architecture. Being from a non technical background, I find this fascinating and I learn a lot each time. As I don’t have actual tales yet I thought I would describe the 4 performance testing types I have experienced and talk about in my talk of 2017. Enjoy.

Performance testing types

Performance testing in my mind is not a type of test but a collective description for many types of tests that provide information about how your system performs under different conditions.

The ones I’d like to mention are these below. (Thanks to Mark Tomlinson for the drawings 3 years ago, as part of my presentation mentoring).

  • Load Testing
  • Stress Testing
  • Soak Testing/Endurance Testing
  • Spike Testing

Load Testing

Screenshot 2020-04-19 at 21.01.37

Load Testing is type of performance test to check how a system deals with a sustained load. It can inform your system’s threshold before it starts to degrade.

When we say we create load we mean the number of concurrent users going through the system and its transactions behaviour.

Load testing is used to inform and determine:

  • Heavy load conditions – informs other tests you can run to determine bottlenecks
  • Throughput rates
  • Response times – especially do they degrade?
  • Resource utilisation

Stress Testing

Screenshot 2020-04-19 at 21.01.49

Stress Testing is often performed to check how a system deals with an ever increasing load (number of concurrent users). This can give information on the stability of software when hardware resources are not sufficient like CPU, memory, disk space etc.

This performance test type is used to determine or validate an application’s behavior when it is pushed beyond normal or peak load conditions. This sounds similar to load testing but the idea here is to actually push the system beyond its limit so it is “stressing”. You want to find out when the system fails, how it fails and keep an eye on how the system recovers (if it can/is meant to).

By running a stress test you can discover things about:

  • The limits of the system
  • How does the system fail?
  • Is the failover strategy adequate?

It may also uncover:

  • Race conditions
  • Memory leaks

Soak Test/Endurance Test

Screenshot 2020-04-19 at 21.01.56

Soak testing or endurance testing involves testing a system with a normal amount of load over a long period of time. In some companies this isn’t possible, bt we have found memory leaks using this technique, even after just 2 hours. However some companies that can run these tests for 2 days. But remember generally load is not high but the test runs for a long time.

This approach will help find:

  • Any problems that only manifest themselves after long periods of use.
  • Memory leaks
  • How long will it take for my system to break?
  • What are the request error rates over time?

And now the most fun!

Spike Test

Screenshot 2020-04-19 at 21.02.03

A spike test is carried out to validate the performance characteristics when the system under test is subjected to high load volumes repeatedly. A common model is to test the load increasing and decreasing several times over a period of time. This can be really useful if you want to test the scaling of your system and also how it scales back down.

Spike tests can uncover:

  • Issues with concurrent transactions at the same time – race conditions
  • Does the application scale quickly?

Why do I find this the most fun? The sort of projects and products I have worked on, this type of test found the most issues. Hitting a system hard suddenly, can quickly show you bottlenecks and other strange issues that you may not find otherwise, more often than not caused by race conditions.

These are the types of performance tests I have experienced and used in the past. What I will do next, I am not sure about yet but I will try to share my journey. Are you a seasoned performance tester, or new to it? What strategy do you have at your work? Does anyone do performance driven development?

P.S. Thanks Mark Tomlinson from @perfbytes http://www.perfbytes.com

Thoughts: Tamagotchi design

I was talking to a colleague the other day and we were talking about some proposed tool and architecture that had been talked about, which seemed to need a lot of human intervention in its weekly, or monthly existence and it may even just randomly shit on everything. Figuratively and this is when we coined the term “tamagotchi design”.

Tamagotchi design is an anti pattern. So not one to follow. If you are not familiar with what a tamagotchi is:

“The Tamagotchi (たまごっち Tamagocchi?) is a handheld digital pet, created in Japan by Akihiro Yokoi of WiZ and Aki Maita of Bandai. It was first sold by Bandai in 1996 in Japan.”

The idea is that you have to look after it or it dies, or gets in a  mess. So you regularly feed it, clean up after it, play with it and cuddle it, maybe?

Do you regularly have to restart servers or micro-services?

What we were saying was, if you find yourself in a world where you have to regularly restart your servers or micro-services, need to check they haven’t lost connection to plug ins or maybe you have implemented a queuing system that needs a manual kick once in a while to keep on processing, then you may have tamagotchi design on your hands.

To avoid tamagotchi design, I would always try and have a plan form an architectural point of view and talk about points of failure a lot and how you will recover from those. Of course some playbooks may include stuff like restarting servers and sometimes that is inevitable but you don’t want this to be in your general run book.

Use an architectural diagram

Drawing or asking your team member to draw an architectural design is so useful. Then just take away the lines and ask what happens. Are there transactions in the system that can still happen if that line is missing? Do we know what the customer sees when this happens? If you don’t know, start testing and providing that information.

Can you also test your playbooks, if they will actually work?

If your system does require frequent human intervention, can these be automated? Can you create a feeding schedule so to speak? Or a cleaning up schedule? How about alerts so you can make sure the system does not get into a right mess?

Monitoring and alerting

How well are you monitoring your system? Do you really know what it needs to be happy and what happy and unhappy really looks like? Maybe you are playing with the system when really it just wants to be cleaned up? Monitoring can tell you a lot about what is going on and learning what happy looks like versus different unhappy states can be invaluable when working on a system.

The worst bit about a tamagotchi was always having to react to it. If you could prepare and feed it a bit earlier or play with it before it got bored, you would have a much happier pet. Alerts could do the same for you in your tamagotchi architecture. They could warn you that queues are getting full for example and you could try and remedy this early.

Do you work with a tamagotchi design? I hope no-one does but maybe you do and can share a bit about your experience?

EDIT: When I told my other half about this anti pattern he mentioned pets and cattle analogy which is very similar to this and maybe I am just muddying the waters with my tamagotchi design message? You can read the history of pets vs cattle on the linked blog. Here a quick explanation from the same blog.

Pets

Servers or server pairs that are treated as indispensable or unique systems that can never be down. Typically they are manually built, managed, and “hand fed”. Examples include mainframes, solitary servers, HA loadbalancers/firewalls (active/active or active/passive), database systems designed as master/slave (active/passive), and so on.

Cattle

Arrays of more than two servers, that are built using automated tools, and are designed for failure, where no one, two, or even three servers are irreplaceable. Typically, during failure events no human intervention is required as the array exhibits attributes of “routing around failures” by restarting failed servers or replicating data through strategies like triple replication or erasure coding. Examples include web server arrays, multi-master datastores such as Cassandra clusters, multiple racks of gear put together in clusters, and just about anything that is load-balanced and multi-master.

 

 

 

 

Thoughts: Test automation strategy

I am a bit behind the curve on this I am sure but the free platform Test Automation University by Angie Jones is pretty great. There even is a more high level short course on test automation strategy. I was super grateful for this, because of a few reasons which I will cover in a moment.

Why was I looking at the platform in the first place? Well we have objectives around test automation. (A discussion on objective setting and personal development plans could actually be a whole series. I still very much refer back to Stephen Janaway and what he wrote a while back on the subject and 1:1s. Read the whole series, it is great!)

Don’t assign objectives or tasks if the person doesn’t have time or isn’t motivated

But back to the objectives. Now it is pointless assigning someone an objective or personal development task when they are not motivated to act on it. If someone does not want to learn to code or cannot put in the time for whatever reason, then there is no point in them needing to achieve actually writing some code. At the end of the day we want our teams to succeed and feel like they are growing in their role. Now for some that is simply being a valued team member on their team. But we do like pushing people and letting them explore their role from different angles.

As a company approach we want to aim for continuous delivery where feasible (regulations), which means more automation. But not many of us can or want to learn to code.

So we decided to split our automation focus and take on the approach of not just doing but also understanding the whole approach. This is something I really liked about the automation in testing approach by Richard and Mark. It gives test automation another dimension because it no longer is about automating away testing tasks such as regression but also understand why you are doing it and what other things it enables you to do and how it supports testing activities.

Test automation is a whole team approach

Sometimes though these sort of words need to be in a course or some sort of material for people to accept it and listen to it. So finding Angie’s course on test automation strategy makes the whole subject of test automation way more approachable. Most of us testers lead scrum ceremonies or meetings. We have to be good communicators, so armed with some great questions from that course, anyone can help lead a discussion on a test automation approach to make sure it becomes a whole team approach.

Not being a coder can be such an advantage when designing a test automation strategy

Not being a coder can be such an advantage when designing a test automation strategy, not a disadvantage. I have seen people feel intimidated and scared but all the technical details, but being a functional tester often means you have a whole picture of the application and can really help the team automate the more risky parts of the application and the most valuable to the business.

I found that functional testers can easily feel less valued than the ones who can write automated tests and by also focusing on understanding test automation it made this objective a lot more approachable in my mind, and shows that you can still sit at the automation testing table, even if you cannot code the test. It might even free your mind. You don’t have a predetermined level of “automating that is hard so I won’t even go there” mind block. You are free to ask all the questions around what is possible and then is it feasible given the estimated effort.

So high level what do you need for a test automation strategy? Well watch Angie’s course. 😀

My summary of the first video with some of my own thoughts below.

WHAT?

A goal! What is your goal? This made so much sense to me. This can help you identify where are automation efforts are most value-able.

The most common goal I have seen is to reduce regression testing time. But here it can be important to then not jump to conclusions. Just because regression testing happens by manually clicking through the UI, it does not mean your tests need to do the right thing. Be clever about which layer is gets the automated tests and what your regression tests actually check and think about the least costly design that is most effective and build on that.
WHO?

Thoughts: It has been 3 years!?

Yesterday I was inspired to write about something and explore my thoughts about a couple of topics. This hasn’t happened in nearly 3 years it seems!

I know I’ve been going on and off social media for the last few years, even deleting other blogging activities and finding it hard to cope with the seeming pressure of social media but I do like putting my thoughts into writing to try and explore a topic for me.

Where did those years go?

Well my last blog was written while I was at an agency. This was a great opportunity. I guess it took away my fear of new tech and not knowing something as every other week you could immersing yourself in a new tech or way of working, from mostly mobile apps, to voice devices, they did a lot and it taught me a lot.

“Use experts to help you focus on your role”

One of the main things I learned was collaborating with the experts. I used to pride myself in being able to spot pixels being out of line, but actually it would take a lot of looking at a front end app. Show it to the designer whose baby it is and they tell you in seconds, that a shadow is missing, something is left aligned instead of centred, the font is too big or small. It opened my eyes to the fact that I don’t need to be good at everything but the expert will see things way faster allowing you to do your job better, you can really focus in using your own expertise of testing then.

“New tech is just new tech”

Considering I really had to get to grips with iOS testing as well as voice testing, I sometimes felt overwhelmed. I even had a mini breakdown in front of everyone, which helped as we changed my role and allowed me to really help with testing the different projects. And at the end of the day new tech and platforms don’t have to be scary. All the things we talk about at conferences around heuristics and oracles apply regardless of which platform or technology you are looking at. At the end of the day you need to ask questions of you application under test an it doesn’t matter what it is for.

I tried to talk about this in my talk on voice testing at Testbash San Francisco in 2018 which you can watch on the dojo.

 

What now?

It has been nearly two years that I joined my current company. Here I have been a hands on tester in a scrum team, as well as managing a couple testers as well. It has felt like a good opportunity to upskill again in a specific technique or domain (although I still don’t code) and put myself outside my comfort zone. Working in a big corporate it can be hard to be honest all the time but it is something that since 2017 I vowed to myself I would always be and address problems I would have early, so I don’t stew on them and then leave. I guess it is right that if your people suddenly go quiet on you they have made their peace and mentally have moved on.

So what I do now, is mostly process improvement, trying to lead testing and thinking about quality. Mentor and teach, (although covid-19 shocked us all a bit and we have put this on hold since we started to all work from home). And then I also do hands on functional testing. It was 75% api and micro service testing using postman and functional testing of front ends until recently. It might go back to that sooner rather than later.

While I still don’t code and still suffer from imposter syndrome, I do feel that I have grown in confidence a lot and that is mostly due to the skills I learned at Songkick, which seems an age away. I do still follow them closely from afar and am routing for them as a company and ex work family. Lauren actually put out an awesome tester career progression document recently. 🙂

Outside of work I put a lot of stuff on hold. But we did run a successful software testing clinic in Brighton which was a lot of hard work but also great fun and I met so many great people. ❤ Emma is continuing the meet ups, or was before covid-19.

And we also ran a great (if I do say so myself) Testbash Brighton in 2019 which you can also find on the dojo!

So have a I really progressed much in 3 years? Yes and maybe not in the way I wanted to but I am excited to see where things go from here.

Yes, developers should verify their bug fixes, but …

First of all, thank you to everyone who has spoken to me about this on twitter, on the blog, in person and even in the aisle of Tesco! 😉

This is part 2 of the conversations about my blog post “Should developers verify their now bug fixes?”. Thank you to all participants!

The main consensus is that “Yes developers should verify their bug fixes.”. And now come the many “buts” (is that even a word?).

Yes developers should verify their bug fixes, but…

  1. … still use the 4 eyes principle or two heads are better than one. It does not have to be a tester necessarily who helps with the bug verification, it could be another developer, a product owner/manager, scrum master or designer. -Basically, most have stated that another pair of eyes makes the team feel comfortable that the bugs were fixed correctly.
  2.  should this be a discussion about quality? I expect devs to replicate the bug and test it’s fixed as part of fixing it , regardless of if there’s another step in the process or not. –I would like to add here that I find the pointer towards quality interesting and want to explore that more. And I think both point 1 and 2 also point towards the fact that you should be owning your work and make sure it is getting the attention it needs.
  3. clear bug reports help build trust. – I totally agree. If you can rely on your team members to communicate issues effectively and clearly this builds trust and you know that if you follow the steps indicated you can then verify if the bug is now fixed.
  4. if we fall back to “aren’t the AC enough?” then why test at all? Aren’t the AC enough for a Dev to verify all their work? –I think this went in a slightly different direction. I still feel that testers’ time is well spent exploring stories and testing implementation of features. In the process they may find bugs and report these (hopefully clearly). And the steps and expected results in those bugs should be enough for anyone to verify the bug is fixed.
  5. is verifying a fix, not more than that? Is it not also retesting a full feature and doing a regression of the system’s area? Hence it should be a tester doing the verification. -Well, I think they are different tasks and exercises. At least in my current context. If the fixes to bugs are so large that the whole feature or area needs regression testing than maybe there are other process issues to think about.
  6. there should be testing expertise available if needed. -Yes!
  7. in certain context the risk is too high to not involve testers in bug verification. -This was generally the consensus from people working in financial industries. I totally agree and think this makes sense. These are often large legacy systems and you never know what you may find as a result of a bug fix. I mostly deal with newish code (less than 2 years old)

 

So this is my second post on this subject. I think a lot of my thinking comes down to team sizes and speed of working. I am trying to work out if we have enough testers across the projects and how we may have to change the way we work if more projects drop on us in the near future. One of these changes may involve the way we think about testing and what a tester’s role is on the team. Will their role still involve verifying all  bug fixes? I think i’d like to push for No and see what happens. More on this if I get somewhere, or not. 🙂

So far this has been great in getting an idea of how people may react and how it may affect the projects we are working on. Thank you all!

Should developers verify their own bug fixes?

This is my attempt at starting to document this discussion I am having in my life about if developers should check their own bug fixes?

In my previous job, we worked very differently to how the teams I work in now work. Not only because my previous teams worked in Continuous deployment and delivery  compared to a more rigid 2 weekly release cycle, it is also partially due to the fact that I now work on projects rather than a product the company owns and develops and maintains through and through. The projects can be anything from 6 weeks to 2 years long with the long term projects being a newer thing.

In the old job we were 2 testers across around 2 product teams and our time had to be used in the most useful and productive way. One task traditionally executed by testers is to verify bugs are fixed. I had forgotten about this because I hardly checked bug fixes in the last 2 years. Why did I not verify bug fixes for 2 years? Well, one argument is that in general a bug report has a well defined steps to reproduce and an actual and expected result. Hence anyone should be able to check if the bug is fixed. Why not the developer? It hardly takes testing expertise to check a bug is no longer happening. With verifying bug fixes off of my plate, I could use my time doing exploratory testing of new features and user journeys through the application. I would of course still log bugs and discuss them with developers, but there was less of the bug fix ping pong going on.

Now with me changing my job, I saw an opportunity because a big time sink of the few testers I now work with is doing bug verification. I tried to argue that maybe they should not be testing the bug fixes but it should be the developers’ responsibility to test their own fixes. I heard some interesting pros and cons for this internally and through twitter as well. So here we go:

Pros:

  • Frees up testers’ time to do actual exploratory testing. Bug fixes could be classed as “checking” and the bug reports should be clear enough for anyone to check the fix.
  • Trust – no-one wants todo bad work and a bit of trust goes a long way for good team dynamics.
  • If developers do more testing, they may find ways of making it easier/faster by writing scripts or little tools, that the test team may not have the skills for doing on the fly.
  • It may lead to more pairing to avoid bugs in the first place.

Cons:

  • Lack of trust – historically bugs were not fixed properly and there may not be trust that the developer completely fixed the bug.
  • Would you check your own homework? Devs testing their own code is frowned upon, does this also apply to bug fixes?
  • Is it really fixed if no-one else has seen the fix?
  • What about knock on bugs that appear as a result of the fix?

 

What are your pros and cons for verifying bug fixes? Do you check bug fixes? Does it take away from your time testing the application?

I do think there is an element of “it depends” here as well. And this applies to the severity of the bug and the further risk of something else being wrong after the bug fix. But I would hope that someone checking a bug fix would notice if something else was wrong but maybe I have worked in very trusting environments the last 2 years and I need to re-adjust my thinking.

Joining a CD Team the experience report

Today is my last day at Songkick. It has been amazing. Unfortunately I am in no position to move closer to the office, which means commuting! Commuting has taken its toll on me and I decided to call it a day and cried a lot.

When asked during recent interviews what my dream job would be, I did honestly say, my current role at Songkick. It obviously had its challenges but I don’t think a role has pushed me and enlightened me as much. Thanks for giving me the opportunity to experience it.

With joining Songkick came the experience of joining a CD team. Amy did a great talk about the survival guide of joining a CD team at Testbash Brighton this year.

But what was my actual experience?

I cam from a scrum team. We had bi-weekly releases and a comprehensive suite of selenium tests that ran through the browser. By comprehensive I mean, we had 64 hours of selenium tests that when run in parallel took 4 hours. So if one test failed because it found a bug, is flakey, you name it, you lose another 4 hours.

This meant there was a 3-5 day code freeze before the release date as well. We also used a typical scrum board with columns including a testing column. Some were even so granular that there was a “Ready for test” and “In test” column.

scrum-task-board-template-powerpoint

Any bugs that were raised during the sprint ended up in the to do column and needed to be fixed and regression tested before the sprint could be closed off.

Typical problems I had at this job were:

  • Testers become the “bottleneck”
  • Convincing team to help test rather than start new work
  • Defining acceptance criteria
  • Everything had an acceptance test
  • No unit tests

 

When joining CD teams it is worth understanding what the CD stands for. At songkick we have both products working in Continuous Delivery and Continuous Deployment.

At the very basic level both have an automated test and release pipeline but the Continuous Delivery approach still has a manual step to deploy to production, whereas Continuous Deployment would deploy the code straight to production.

This environment came with many learnings for me.

Ready for test:

Being used to the “Ready for test” column it was quite a shock to join Songkick and its continuous delivery and deployment approaches. Don’t get me wrong I joined because of these approaches as well. In conference talks this often appears to be a mystical world of software development where you test your code on production and developers have relatively good dev environment set ups that they can test well themselves.

But when do I test? Also where do I test?

There was something like a complete loss of control. In certain work environments (not Songkick at all) when a bug is found in production, the test team gets blamed. “Why wasn’t this tested?” is a common phrase. If you have worked in such an environment and then go to continuous delivery you can initially feel panicky because things move so fast and you may not be, or are very unlikely to be doing the testing.

What helped was understanding what is actually going on when new features are being developed. A thing that helped was that testing is happening everywhere all the time.

So discovering that everywhere testing is happening and can happen is invaluable. So you (not you personally but the team) can and probably is testing all the time everywhere. This starts with Kick off conversations. Here you get the chance to test the idea, the problem and the ideas around implementation. It is also a good place to state how you would test the feature.

A great tip from Amy was that in previous roles she learned that telling people what you will do for testing the new feature meant it would already get done by the person developing it and hence there will be less bugs.

So overall these kick off conversations become the most valuable part of testing for a tester at Songkick. They help inform what is just enough testing for this particular release.

Forget about “Ready for test” columns. Everything is continuously being tested and not necessarily by you. Think about how you can integrate your tester mind-set to be heard.

 

Ultimate shift left:

Get in on the planning discussions, requirements, stories, kick offs, whatever you call them, just invite yourself to the meeting and ask questions such as:

  • What is the problem?
  • Who is having the problem?
  • Why are we fixing it now?
  • How will we test it?
  • What is the risk of shipping bugs?
  • Do we need to consider non functional testing such as performance or security?

 

Team and trust:

Build good relationships in your team, because you will be on holiday for a week and there will have been 50 small releases to catch up on that were not tested by a tester.

For me this was about letting go and trusting your team members. Remember no one wants to write bad code and ship broken features.

Like I mention above I felt lost in the beginning, where do I fit in? Who is testing all of this?

A lot of that was about letting go of control issues. Discuss things you care about in a team setting, such as quality features and processes so you all feel comfortable. We did a great session on what quality and quality features means to us and how we define “just enough” testing. This can vary across products and teams but is so valuable to talk about.

 

Mindset change:

Testing becomes more of a proactive role, where you do the testing early on by pairing with developers, or going through unit tests, or liaise with the business to make sure you have the right acceptance tests in place.

Some of this mindset change may be to embrace more of a coaching mindset and sharing experiences and testing knowledge with team members.

At least at Songkick you are an expert in your field and can leverage this to help the team get better at testing in general.

 

Forget a “release date”. Communication is key:

If something is ready to go it will be released. Establish good communication with teams affected by changes. Release log emails, cross functional team updates in slack or regular catch ups.

This was an awesome change for me, to see how much the team is communicating with other departments to let them know about changes.

 

Test in production

g1361999261125710696.jpg

Get to know your production data really well, as a lot of testing will be done in the live environment. This has great advantages as the code is being tested where it is being used with real data but it can be daunting, as you could be affecting real reporting. Liaise with the relevant departments to help sort this and avoid any confusion.

I was always worried I would notify people about fake events. Liaising with the relevant teams and setting up data in a way to not disturb real fans was fun and scary at the same time and we actually found a bug that we could then fix asap.

Joining Songkick was one of the best things I did and I am super sad to be leaving today, but I am also excited about sharing what I learned and bringing this to a whole new role on the near future.

Have you had any experiences joining CD teams? Any learnings to share?