Thoughts: User Testing or A/B Testing

I have been involved in some user testing over the last couple of years. This was regardless of me being a professional tester, it was to provide feedback as a real user of a potential website. 

This has involved old design vs new design and general reviews of a design. Sometimes the website was coded and functional, other times it was just a wireframe. Nevertheless getting real users involved early to test can be a real eye opener and more importantly to the business a money saver.

The following scenario probably won’t be all that hard to imagine: Developers/Testers spend months or longer, developing and testing a product.
Once the product is released, users report bugs that were not found by the team. The question that gets asked is – how did the team miss these issues?

Common reasons for this are:
  1. The testing environment is too removed from the real life setting.
  2. This could be due to a lack of understanding of the user stories/use cases.Lack of time to test a use case fully.
  3. The user took different steps to fulfill the use case to the test team.  This could be as simple as doing things in a different order.
  4. Input values were used that were not considered during testing such as white space in a set of data ie. XXXX XX XXX.
  5. The defect that users reported could come from code that was not tested. It could either be due to having released untested code or the existing set of tests did not exercise the piece of code where users found defects.
  6. These challenges to get all the variables right increase with the complexity of the product. it is therefore of great value to test the product early with actual users.

Users know what they want from the system and are removed enough from it to test it for their individual use case. Whenever it is possible the opportunity should be taken to use actual users.
Watching an actual user of your system perform set tasks with it, is invaluable information in how they perceive the system, and how usable it actually is.
These sessions do not have to be long or expensive, a laptop and recording software such as camtasia and a microphone will do.

In Don’t make me think – A common sense approach to web usability Steve Krug, shows an example of budget user testing. The main points to take from it are:
  1. You only need to run tests with 3-4 users – this can easily be achieved in a morning.
  2. Anyone can do the testing for you – do not be afraid to involve friends or neighbours.
  3. Any location can be used – you do not need a usability lab – a quiet room with a laptop and microphone will do.
  4. Anyone can run the session and guide the user through the test.
  5. Tests can be run at any point, not much scheduling should be needed.
  6. Have a rough plan what you want to show and what tasks you want to set.
  7. Decide on a fair incentive for the user – I have been given £15 for 40 mins of user testing.
  8. Debrief the team with the user testing outcomes on same day as the tests are run.

Some user testing truths:
  1. Great sites and applications need to be tested – development is often too involved and affected by the “Curse of Knowledge”, meaning if they know something it is assumed everyone knows.
  2. Testing with 1 real user is better than none at all – testing will always work.
  3. Testing should happen early and often – changes can be made a lot easier early in a project.
  4. You do not need “representative” users – the most important thing is to test the site early and not necessarily using people like the future user.
  5. The point of the user tests are to gain information about your site/application – not to prove or disprove something ie. is navigation a better than b?
  6. User testing should be iterative – you cannot test just once – you develop something, test it, adjust it, test it again.

Have you ever done user testing in your company or as part of your contract? Where you surprised by the outcomes?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s