Yes, developers should verify their bug fixes, but …

First of all, thank you to everyone who has spoken to me about this on twitter, on the blog, in person and even in the aisle of Tesco! 😉

This is part 2 of the conversations about my blog post “Should developers verify their now bug fixes?”. Thank you to all participants!

The main consensus is that “Yes developers should verify their bug fixes.”. And now come the many “buts” (is that even a word?).

Yes developers should verify their bug fixes, but…

  1. … still use the 4 eyes principle or two heads are better than one. It does not have to be a tester necessarily who helps with the bug verification, it could be another developer, a product owner/manager, scrum master or designer. -Basically, most have stated that another pair of eyes makes the team feel comfortable that the bugs were fixed correctly.
  2.  should this be a discussion about quality? I expect devs to replicate the bug and test it’s fixed as part of fixing it , regardless of if there’s another step in the process or not. –I would like to add here that I find the pointer towards quality interesting and want to explore that more. And I think both point 1 and 2 also point towards the fact that you should be owning your work and make sure it is getting the attention it needs.
  3. clear bug reports help build trust. – I totally agree. If you can rely on your team members to communicate issues effectively and clearly this builds trust and you know that if you follow the steps indicated you can then verify if the bug is now fixed.
  4. if we fall back to “aren’t the AC enough?” then why test at all? Aren’t the AC enough for a Dev to verify all their work? –I think this went in a slightly different direction. I still feel that testers’ time is well spent exploring stories and testing implementation of features. In the process they may find bugs and report these (hopefully clearly). And the steps and expected results in those bugs should be enough for anyone to verify the bug is fixed.
  5. is verifying a fix, not more than that? Is it not also retesting a full feature and doing a regression of the system’s area? Hence it should be a tester doing the verification. -Well, I think they are different tasks and exercises. At least in my current context. If the fixes to bugs are so large that the whole feature or area needs regression testing than maybe there are other process issues to think about.
  6. there should be testing expertise available if needed. -Yes!
  7. in certain context the risk is too high to not involve testers in bug verification. -This was generally the consensus from people working in financial industries. I totally agree and think this makes sense. These are often large legacy systems and you never know what you may find as a result of a bug fix. I mostly deal with newish code (less than 2 years old)

 

So this is my second post on this subject. I think a lot of my thinking comes down to team sizes and speed of working. I am trying to work out if we have enough testers across the projects and how we may have to change the way we work if more projects drop on us in the near future. One of these changes may involve the way we think about testing and what a tester’s role is on the team. Will their role still involve verifying all  bug fixes? I think i’d like to push for No and see what happens. More on this if I get somewhere, or not. 🙂

So far this has been great in getting an idea of how people may react and how it may affect the projects we are working on. Thank you all!

Should developers verify their own bug fixes?

This is my attempt at starting to document this discussion I am having in my life about if developers should check their own bug fixes?

In my previous job, we worked very differently to how the teams I work in now work. Not only because my previous teams worked in Continuous deployment and delivery  compared to a more rigid 2 weekly release cycle, it is also partially due to the fact that I now work on projects rather than a product the company owns and develops and maintains through and through. The projects can be anything from 6 weeks to 2 years long with the long term projects being a newer thing.

In the old job we were 2 testers across around 2 product teams and our time had to be used in the most useful and productive way. One task traditionally executed by testers is to verify bugs are fixed. I had forgotten about this because I hardly checked bug fixes in the last 2 years. Why did I not verify bug fixes for 2 years? Well, one argument is that in general a bug report has a well defined steps to reproduce and an actual and expected result. Hence anyone should be able to check if the bug is fixed. Why not the developer? It hardly takes testing expertise to check a bug is no longer happening. With verifying bug fixes off of my plate, I could use my time doing exploratory testing of new features and user journeys through the application. I would of course still log bugs and discuss them with developers, but there was less of the bug fix ping pong going on.

Now with me changing my job, I saw an opportunity because a big time sink of the few testers I now work with is doing bug verification. I tried to argue that maybe they should not be testing the bug fixes but it should be the developers’ responsibility to test their own fixes. I heard some interesting pros and cons for this internally and through twitter as well. So here we go:

Pros:

  • Frees up testers’ time to do actual exploratory testing. Bug fixes could be classed as “checking” and the bug reports should be clear enough for anyone to check the fix.
  • Trust – no-one wants todo bad work and a bit of trust goes a long way for good team dynamics.
  • If developers do more testing, they may find ways of making it easier/faster by writing scripts or little tools, that the test team may not have the skills for doing on the fly.
  • It may lead to more pairing to avoid bugs in the first place.

Cons:

  • Lack of trust – historically bugs were not fixed properly and there may not be trust that the developer completely fixed the bug.
  • Would you check your own homework? Devs testing their own code is frowned upon, does this also apply to bug fixes?
  • Is it really fixed if no-one else has seen the fix?
  • What about knock on bugs that appear as a result of the fix?

 

What are your pros and cons for verifying bug fixes? Do you check bug fixes? Does it take away from your time testing the application?

I do think there is an element of “it depends” here as well. And this applies to the severity of the bug and the further risk of something else being wrong after the bug fix. But I would hope that someone checking a bug fix would notice if something else was wrong but maybe I have worked in very trusting environments the last 2 years and I need to re-adjust my thinking.

Agile Testing: Frustrations can make us creative

  

creative space design of a restaurant

I love the fact that I have worked at a few different companies and hence have gotten to know so many awesome people.

It also has the great benefit that I still get to take part in different conversations around Agile Software Development practices and especially when the focus is on testing.

The other day I was forwarded a little article by Mountain Goat Software which focused around the fact that in an agile team, at the end of the sprint everyone tests. And some developers may not like manual testing so they will do automation testing or pair with testers to find efficiencies.

“At the end of the sprint, everyone tests.”

I am wondering a few things around this line of text. Why does everyone only test at the end of the sprint?

I know there have been many discussions that “testing” means manual testing being done by a human. But I do think that developers are often using Test Driven Design to write code and hopefully are also running the code they write to test that it does what they intended and maybe also how it does not (depending on experience, mind set, time pressures and learned habits).
Nevertheless the above quote is important for agile teams. I have experienced that as testers you can easily become a bottleneck because the developers think their part is done. Ideally they will have written the code and written automation tests as well. Now the manual testers get to test the new feature works and hasn’t broken anything.

But this leads to the problem that the developers are sitting there twirling their thumbs or starting more work, adding it to the bottleneck until the pipeline completely collapses.

So I like the fact that there was an emphasis put out that everyone should test as the team is responsible for shipping the work altogether.

I do hope though that testers and developers in a team get to test early as well, or maybe even designers or product managers, to make sure that the feedback loops is as fast as possible. The statement has the danger to be interpreted as such that testing should only happen at the end of the iteration. (I don’t think that is actually what they are saying.)

The excerpt I read actually states that the better the agile team the later the testing can occur. This seems wrong to me, but maybe I am taking it out of context and it is referring to the time needed for the whole team to be testers.

 

“Establishing a mindset of we’re all testers at the end, does not mean that programmers need to execute manual test scripts, which is something they’ll probably hate.”

 

The article went out to say that developers may dislike using manual test scripts for testing and hence maybe could focus on other things such as automation scripts.

I actually thing that it will be really beneficial for developers to do manual testing, not necessarily follow test scripts but explore the features they wrote instead.

By all means make sure your code is well tested using automation tools but you will not know if you have written the right code if you do not manually test it. You will only know that the code is written in the right way.

 

Frustrations can make us creative

I recently watched a video that Chris recommended on How frustrations can make us creative and it highlighted how doing something you maybe don’t like can actually result in you finding ways to make it better a lot easier than doing stuff you enjoy all the time. So developers definitely can really gain from doing some manual testing. And not just developers but hopefully the whole team.

A sort of related example of this is, that at Songkick we try and learn about what each department does and were encouraged to help the support teams. This has actually led to improvements being made to the admin systems when developers used it to help solve support cases. It is OK here to JFDI, when an improvement can really help someone.

A small example was that someone added extra information to be pulled in on one screen, so that you do not need 2 tabs open anymore to find all the information you need to close a support case. This was a huge time saving for everyone and was only achieved because a developer used the tools he created for the support team.

So I encourage everyone to test, collaborate and try something that frustrates you and see if you can make it less frustrating to do.

 

 

Real Life: On doing my first testing talk

“Where do you see yourself in 5 years?”
This is a typical interview question (that I personally avoid asking) but when I was last questioned about this I said I wanted to be speaking about testing and share my testing knowledge publically.


Since then two years have passed and yesterday I completed my first public talk at a local meet up. It was nerve wrecking and far from perfect but I have gotten some good pointers and one person admitted to learning something and others approached me to talk about testing challenges afterwards.
The meetup is normally full of developers  and I am the lone tester, but I managed to increase the tester attendees to 4 in total (that I was aware of)! Whoop!
This means we were over 10% as I think around 30 people may have turned up!


My Journey
My public speaking journey isn’t particularly special but here it goes anyways.


So after me stating in that interview that I want to be speaking publicly, I started a job at the company and around 8 months in did my first talk about Agile testing alongside my test manager.
We picked a few things from the book Agile Testing by Lisa Crispin and Janet Gregory and related the approaches to how we develop and test our current products and maybe how we can improve our sprint cycles, deployments to the test environments. The last part was an active discussion with the team. It went well but I read a script.


The second talk I did was very similar but in my next company where testing done by a professional was new. Again the subject was Agile testing and we brainstormed some things we could improve and set up action points.
This time I had a script I practiced but tried to mainly use it for guidance.


I should have mentioned that both talks I was sitting down in a meeting room and not standing.


I am not sure if it counts but before any of those I did a 99 seconds talk (well 60 seconds) at Testbash in Brighton.
I had written a little poem and just read that out to everyone. Funnily enough it meant a couple of people recognised me a year later and conversations just flowed about testing.


So just the small talk at testbash had a positive effect as it put me out of my comfort zone and then others felt happy to come and speak to me, which is great as I get shy and may not approach other testers for a chat. 🙂


And now finally I presented in front of an audience of mainly developers at a public meet up. This was probably the most challenging talk yet. I did get some good feedback and had talks with strangers about testing and how to improve their workflow afterwards.
But I can still improve loads.
What’s next?
I like to challenge myself so I am hoping to get some mentoring through the speak easy mentorship programme and maybe talk again at an event. I am hoping to gain guidance on how to structure a talk and some speaking tips.
Alongside this @northern_tester on twitter also shared some links with me about tailoring talks to your audience! Sounds like exciting stuff!


Any tips?
At this point I don’t have any tips really, but will give it a go:
1.Speak about your topic to as many people as you can. Start with individuals, friends, family, work colleagues.
2. Practice your arguments and points. Maybe use different outlets like twitter. Can you really summarise your point in 140 characters!?
3. Say your presentation out loud, maybe also to someone who will listen.
4. Less is more. That goes for text on slides and the words you say. A well placed pause can convey a lot of meaning.

How do you challenge yourself? Do you have a five year plan that would involve you getting out of your comfort zone?

Thursday@EuroSTAR – My notes and some pictures

These notes are a bit delayed. Sorry. And also a lot more ramble-like. I hate it when the real world just swamps you and you feel like you cannot do things as well as you may want to. It is not all bad though. 🙂
Hope you get some value from this!

Shmuel Gershon, Intel, Israel


Do we understand what the real value of software is?
Civilisation runs on software. Dependent on software!
Everyone stores, transfers and applies knowledge in their life.


Civilisation itself depends on testers now. We look for the most faithful representation of knowledge possible in software.
Testers are therefore researchers of knowledge.
Software really is endangering civilisation as we know it – big responsibility of testers!


We need to care about users and what they are looking for.


We come to conferences to save civilisation and for the beer! (Or in my case the food).



Testing Traps to Avoid in Agile Teams

Janet Gregory, DragonFire Inc., Canada
traps and risks due to them
waiting for the build – why arent we getting stories to done – next iteration testing
mini waterfall – ketchup effect
tests not complete – dod not reached
testers lose credibility
tech debt
plan for test infrastructure – build pipeline
testing taks in velocity
devs used to instant feedback
test where it makes sense
Testers not part of the team – be active
risks – wrong assumptions, team becomes divided, skills missing
become useful, power of three
testing is a team responsibility
quality police – do not want the power to stop a release into production
communication needs to be face to face
devs use testers as a safety net
testing as a service – consulting mindset
everyone needs to pick up testing tasks
technically aware
make testing visible
help deliver software successfully
manual testing – agile not sustainable without automation – minimum of run once a day
include automation time in estimates
encourage collaboration
understand power and risk of what you are automating
business logic back in the UI
big picture
devs add extra code after they are finished
integration too late
mind map of the big feature
more examples from product owners
story done and feature/release done



Diversity in your team – embrace it or lose the best thing you have!

Julie Gardiner, Redmind AB, Sweden
strength lies in differences not in similarities
get the balance right
Dreyfus model for skills aquisition
people at different skill levels need different management
excercise
friendly
approachable
business like
open
organised
social
logical
random
warm
perceptive
7


to the point
accepting
quick
4
pragmatists – efficiency – direct
facilitator – togetherness
the analyst – accurate
the pioneers – enthusiastic
belbin role types


Zeger van Hese, Z-Sharp, Belgium


interconnectedness – triangles
impossible objects
comfortable clone syndrome – avoid as it squashes diversity – testing needs diversity to find different types of problems
control dilemmas
randomness increases variety and serendipity – finding something valuable while looking for something else
creative vs critical thinking
we copy to gain understanding and knowledge which we can then transform and apply
everything is a remix
leadership makes the diverse mix work
empower people to do their best work
leave people in control
recognise good ideas



Programming for Testers – It is Easy!

Graham Thomas, Independent Software Testing Consultant, United Kingdom
don’t mix data types – str()
single or double quote
python 3 uses ()for strings


first line called the shebang – comment # ignored
first line is special #! = shriek
shows where python environment is

Some pics:


The TestLab!
I also want to give a shout out to the TestLab team! They were fun, engaged individuals working as a team to provide puzzles and exercises and brought a whole new level of engagement to the event!


What do we do after the conference?
Now comes the difficult part! When you are home how do you implement any of the great thing you learned without just suffering the come down from the conference?


I started with just a question:
Are we giving our customers the right information for the task?


Did you ask a question? How are you applying your newly gained knowledge?

Webinar: A New Agile Ecosystem

A couple of weeks ago I was fortunate enough to find the time to attend Michael Bolton’s webinar entitled a New agile ecosystem or “The (REAL) agile testing quadrants” as we believe it should have been”.

Disclaimer: the images are screen prints of the webinar. The link is above. I say the odd opinion myself but generally I tried to convey my understanding of the webinar.

He started with a quick mention that this talk was not an attack on agile testing or the role of a tester in an agile environment as in principle “agile development is humanist and cool!”

I think this was important for him to mention as the context driven school of testing can appear quite aggressive in expressing their opinions and thoughts on other testing schools of thought. I am unsure if this has well expressed by myself but this is just how it has appeared to me.

Anyway Michael points testing is learning how to do difficult things – increase understanding of the product and its context of working and is a central part of the development process. With this in mind he states that the original testing quadrants are unhelpful and misleading, to an extent.

What is the role of testing?

But before the webinar explored this we had to think about “what is the role of testing?”.


“Learning through exploration and experimentation.”
This means that scripted testing and exploratory testing are not opposites but rather scripted testing is a means of exploratory testing.
A tester’s role then is the prepare for testing, perform testing and report on the results of testing.


Meaning and history of “testing”

Testing has had a misunderstood meaning around quality policing. Michael adds that quality policing adds responsibility without authority.
Hence adding unnecessary distractions to the testing process.


Instead of policing quality testers add value by adding knowledge about the product and awareness of risk.


This, he explains, is a really challenging skill – hence adding things to it dilutes the testing effort overall.


We as testers help our “client” look good and this effort requires a lot of critical thinking.


Testing Skills

Michael moved onto the skills of a tester briefly and explains that curiosity is not a skill but rather a tester should have “model skills” and observe and connect the dots. “Model skills” were mentioned around being able to visualise and model a testing journey, be that the data needed or how the system is built up in your mind’s eye.
Other skills mentioned were:
  • generate ideas
  • define risk and problems
  • elaborate – refine and communicate
  • good reporting skills
  • illustrating and describing issues and products


Hence policing is a distraction of the goal of testing. As testers we are most helpful when we perform testing.


Should testers only look for bugs?


“A role is a heuristic not a prison”


To say testers only find bugs would be the same as to state that developers are terrible testers.
Actually they are also testing all time and can actually be very good.
Testing involves thought experiments – try things – model – conjecture – model, etc. which are inherent to the development process.
But of course developers are not perfect testers – you still have to develop certain other skills to become a great tester.


Hard to wear two hats at once


“Testing focuses on the task of finding problems that threaten value”


Michael starts the last third of the webinar with bringing this all together to talk about the original quadrants and how they feed a misrepresentation. It is easy to read into the original quadrants in my option also, that they favour automation, checking, over actual testing.

He believes tester should be at the centre of testing and not the tools – critical thinking over automation.

Time for an overhaul – evolution


Supporting the team and critiquing the product should not be a separate seemingly opposing quadrants.


You can read an opposition around automation and manual testing, but automation is not something that you do but something you use as a tool.


Tools or hands are info-mechanisms that provide information. We need to think in terms of testing and how tools can be supportive of the testing effort.


He goes on to say that the notion of testing should be in all quadrants.


Why are certain things in certain quadrants? It should be one big testing effort. Polarities only serve to convolute the effort and distract from the actual testing.


His main criticism was that critiquing seems to not be part of programming and supporting the team implies if testers do not write code they should not be in the agile team.


Michael sees a confusing and unnecessary distinction. And provides his own solution.
Example of a generic diversified test strategy


Quadrants inside of each other – building up the effort instead of opposing actions.
Testing is an active thought process and automatic checking is inside of this but it is not music, which it appeared to be in the original version of the agile testing quadrants.

Testing as an activity cannot be read

Representations can be read but business people don’t read cod.
Testing is to develop the vision of the product and help refine the design.
Prepare for testing and do deep testing – understand the product and make information about the product accessible to the team.
When are we done? Definitions?

Things need to be re-evaluated on the basis of new information


There is more than just one human user – api, browser  – he uses a broad interpretation of “user”. I actually really like thinking of a “user” meaning more than just human interaction. It has helped me with our new approach to creating a service orientated architecture which often requires services that do not have a front end.


As 100% testing is not possible we as testers need to recognise how we may not be done. This he states relates to many perspectives and from many different perspectives.

He points out a shift in adaptability in testing and not just repeatability, which automation aims for.
And maybe more importantly apply this adaptability approach to the product and can the product respond appropriately?


Development is not linear –  activities tend to overlap and swirl in big and small loops.

Agile testing is testing in an agile context


We therefore need productive polarities, and not opposing ones.


He finishes with that testing benefits from a diverse perspective – critical distance is the difference between one perspective and another.

Did you enjoy the webinar?
Do you agree with the view on the original testing quadrants?
Have you got the new agile testing book “More Agile Testing”? I have just ordered it and looking forward to read it. Let me know what you think!

Webinar: Integrate Test Activities in Agile Projects

Integrate Test Activities in Agile Projects


I attended the webinar from Rik Marselis about integrating test activities in agile projects (slides).
The topic caught my attention because I am currently solving this issue in my job especially around manual test activities.
The existing process used to focus mainly on web tests but has evolved to include unit tests and now also includes manual tests executed by a professional tester instead of just the PO.

The talk started with the acronym QD.

I had never heard “Quality development” as a term before when referring to agile development. Maybe it was implicit but I could relate to QD being described as “Quick Development”, especially when the emphasis was on just getting features out there quickly, to be revisited at a later date to improve usability.

In my experience this does not happen unless there is a blocker as in the new features becomes unusable for some reason or causes too much of a maintenance headache. 

I recently did a short talk at my company about Agile testing and how we are trying to integrate test activities (especially manual testing) into the scrum teams, and like Rik I also referred to the Agile Manifesto.
He, unlike myself, argued that it does not have strong footholds for testing in an agile development project. I would argue that it includes the main parts for testing to be successfully integrated namely, people and interactions, collaboration and responding to change. if these ingredients are not in your scrum team than testing cannot be successfully integrated in my opinion.
You need the team to be interacting and sharing to make testing possible and to avoid duplication of the testing effort. This goes hand in hand with collaboration and responding to change. As issues are uncovered the team needs to work together to fix these/make changes if new requirements are uncovered.


Rik went on to mention the Scrum guide which I have not read in detail and hence will not add anything to right now, but this did yield a couple of valuable pointers.
Testing is described as inspections performed by skilled inspectors with adjustments made asap and each increment being thoroughly tested.
Thoroughly is a word which actually rubs me up the wrong way and Rik seemed to agree. “What does thoroughly mean?” 100%? 100% test coverage is not possible, who decided what is thorough and adequate?


The development teams I have been involved in recently talked about a definition of done in terms of if all development streams are on the same page. We mentioned code/features/stories to be adequately commented/documented/tested (automation and manual).
I understand that in scrum methodologies these sort of guidelines for a definition of done need to be flexible to an extent but using words such as adequate and thoroughly when these are subjective seemed incorrect to me. Maybe more on that in another post.

Add caption



Testing during agile development should be about providing an insight into quality and risks. Defects should be identified as soon as possible, with exploration into root causes to avoid them in the future.
Through exploratory testing we  should yield re-usable test cases as each increment should be thoroughly tested and will therefore need a regression testing phase.
Keep testing transparent!
This is feedback we have had in my current teams that the developers were unsure about what I actually do. We try to combat this by writing test charters and acceptance tests and circulating them for anyone who wants to read them/use them as broad guidance to ensure code test coverage.
This follows onto the next point that was made that testing is a skill that should be integrated into the team, preferably in the form of a skilled/professional tester. Long gone ae the days of sub teams. We need to make the transition of an us vs them mentality and become a valuable asset of the dev team. By thinking outside of the box and like a user, testers can integrate into agile development teams to provide another set of skills.
However testing is not just a developer/tester activity. Test integration also needs the PO to review/test the stories and features, ideally before the actual end of sprint review. This can ensure they accept the result of the sprint as at the end of the day they will make the decision if a sprint was successful.

During planning the Scrum Master can act as a tester helping to define acceptance criteria and making sure the proposed designs are fit for purpose and will create a product of high quality. This process of questioning can minimise rework for the whole team and ensure they are focused on the same outcome.

This small list from the talk highlights the whole team approach that is so important for success.
To be fully integrated testing needs to be part of each step of the way, during planning to identify risks and start planning which tests and test tools may be needed. then while the features are developed, the tester creates and maintains their tools which ideally can be used by the whole team.
The test strategy will not just depend on the project but also on the sprint and tasks associated with this. To make testing more transparent Rik mentions adding tasks around testing to your sprint/scrum boards. I may well do this and try it as part of our next project as the team is growing and it would be nice to offer some guidance on what is needed to perform the tests required for that sprint and let other testers take over the creation of test data/environments etc. Tester in the context here also includes developers, Scrum master and POs.

The pitfall of a test strategy according to Rik, is to focus solely on unit testing or functional tesing only and how to automate them. Manual testing can expose certain scenarios that machine simply cannot and hence we need to consider the user perspective.

It may be useful to think of different testing objectives instead. Objectives of each level of testing need to be considered when defining and executing testing. What should the unit tests cover/ensure? What should the web tests cover? What risky areas should the manual tests look at? Is manual testing focused on usability?
We try to manage unit and business tests during the sprint and automate them accordingly. Also each story or feature has a web test added if appropriate and is tested manually.
We then perform full end to end tests when code is about to go live on our most like production environment (however this still could be closer to the actual live environment).
Throughout this process we run automated regression tests also, because manually regression testing is simply not possible anymore.

Like development we need to look at testing when integrating it into an agile development cycle to be sustainable. integrate activities as early as possible and as many layers of testing as possible. Assign tasks for the different layers to skilled team members, ie. unit tests are written by developers. testing needs to be considered to be a task of the whole team.

With that in mind consider reviews as a form of testing that can ensure quality from day 1.

I took a lot from this talk and hope to put forward the idea of also tasking up testing tasks to make them more visible to the team. I think this could be valuable as it can also be a good training tool and won’t be micro management. Certain tasks can be split across team members (not just testers) and reviewed in a peer review before being considered to be done.
This would offer a platform for learning and sharing in my opinion which is a big part in successfully integrating the testing effort into the team.