Usability Testing. Why it is the first step towards great Product Design

I was in a taxi in Melbourne recently and getting close to my destination. The driver was trying to queue up his next job on the machine on his dashboard by pressing to accept another pick-up nearby. He repeatedly pressed the button…. Although the machine was entirely unresponsive, he kept pressing the button harder and harder. I could see he was distracted from the traffic ahead because of this machine. He restarted it. Nothing. He continued to press the button every few seconds. I wondered if anyone had done a test where they watched a driver using this machine while driving. I was torn between this thought and wondering if he was going to plough into the back of the car ahead.

Observing people using the products and systems that you have designed is confronting, but is the most important thing you can do as part of creating or improving any Product (or service). As colleague, Jason Furnell said last year “if you don’t watch people trying to use your software, you are just not serious about building good systems”.

He is right. Just watching people use the system is a key way to understand if it will be successful. It is also the most flexible technique of all design techniques. You can do it at any stage of the design process. Even when nothing has been designed or built yet.
At its essence, Usability Testing is just watching someone try to accomplish something on a system. The important point here, is that they are “trying to accomplish something”, not just telling you what they think of it. It is important to observe people in action rather than ask their opinion.

So how do you do it? The simplest way is to sit (or stand) with someone while they try to complete a task. Something like trying to book a plane flight, or plan the whole holiday. The task can be narrow or broad. As they try to complete it, ask them to think aloud while they are doing it. It is a slightly odd thing for them to do, but is the best way to get a few clues as to why they find it easy or difficult. Afterwards share what you saw with your team. That’s it…

How to conduct the User Test (The short version)

I have outlined below the basic flow of a Usability Test and provided a few tips on how speed it up. It is better to test weekly with a few people, than comprehensively once or twice. Light, fast and continuously is our catch-phrase for this technique.

Step 1. Determine the purpose of the test.
Step 2. Write a User Test plan (an outline of what you want from the test)
Step 3. Develop a Test Protocol (Script) to follow when testing (until you get the hang of it)
Step 4. Choose what system to test with
Step 5. Recruit the people to test with (Users)
Step 6. Set up and conduct the tests (give a small gift to each user)
Step 7. Distill the findings from the tests and share with your team

Identify the aim of the test. What are your questions?
What are you trying to achieve with your Product and with your Usability Test?
Exploring ideas. If you are at the beginning of your project or product design, then Usability testing can be used as an exploratory technique. You can test to learn. Testing with an existing system or a similar or competitor system will give you insight into your project and product and how people could use it.

Validation. Once you have some ideas together, you may wish to test your ideas to see if a person using the system will ‘get it’. Other aspects that can be tested here are how quickly or efficiently someone completes a task. In some cases, it may be more important that someone can complete a task quickly or more important that it is entirely error-free. This will depend upon what your product is designed to do.

Choose the tasks
Where do tasks come from? A few places. You can set them yourself or you can ask the person you are testing with a few questions. If you set them yourself, then you need to identify what the system is intended to do and provide the task as a scenario.

A sneaky (and better) way to do it is to ask the person about their recent experience. Either on this system or something similar. A few questions will expose what their recently activities. You can use that task for this test. You can say “can you try to that same thing here on this system.”

For example, if you are working on a new travel website, you can ask about their recent business or holiday travel and use that same situation for the test. This is often a richer scenario than carefully crafted tasks as it is more meaningful to them. It is also a nice way to gather a bit of additional user research. Do keep in mind that you still need to ensure that you have the coverage of the different tasks or aspects of the product or system.
Who to test with

You will have an ideal target audience you are designing for and then you will have the challenge of trying to find someone from that group. You can use your network, the organisation’s network or a recruiting company to recruit the ideal group, but for some tests, just anyone who hasn’t seen the design will be a great start.
Dana Chisnell has a good article on how to find people. She suggests not using a recruitment company, but I think you have to be flexible. If that is your only avenue, then you will still get something useful. It is better to test with someone who is not exactly right, than not at all.

What system to test with
For exploratory testing, you can test with the existing system, competitor systems or paper prototypes of your ideas. These will give you insights into what users need and ideas for improvement and innovation.
If you are part way through your Product design and build, then test with wherever it is up to. You can supplement this by using an interactive prototype to test the breadth of the system, while a deeper, but narrower test can be done on a partially built system.

When to conduct User Tests
Before you start as part of your initial research and then throughout the project. Ideally weekly with a few people. Resist the urge to wait until everyone is happy with the design before testing. It will never feel ready before a test.

Where to test
In the project room. In the early stages of the project, the testing can be done in the same room as the project work. We always have a large room where the walls are covered in ideas, designs and research insights. Posters, drawing and sticky notes. Messy to the untrained eye. No problem, you can still test. Clear a space to work on and just bring people in. In my experience, people react very well to a working space. It is more casual and they relax into the testing easily.

In the field. Remember the taxi driver? If the Product is designed to be used on the move, then you will have to test it in action as soon as you can. If you are designing a mobile app or website, then testing in different environments (e.g. outdoors, noisy, bright sunshine, rain etc,) will give you a better sense of the challenges of using it.

What you should be doing
What you should not be doing is nodding wildly and say “uh huh” to everything they do. You should be as neutral as you can possibly be. Think Switzerland. Being neutral and mostly quiet is much harder than it sounds. If they ask a question, you need to deflect it with “What would you like to see instead”, “Where would you expect to find it” or something that brings them back to the task and away from you.
If they can’t find something, ask “What are you looking for?” –– this gives you the words that have meaning for them.
If they still can’t find it, then try “Where would you expect to find it?” –– This gives you where they are looking on the screen for the item.

Note. It is possible, indeed likely, that the participants will not use your designs or systems as you intended. This is the confronting part. You need to get used to this. Stay open and see it as data, not as an affront to all the work you have put in. Your team share back session will be very helpful when this happens.

Taking notes
This can often be the most challenging part because of the amount of data that you can get from a session and the speed of the test. Ideally have someone else be the observer and notetaker for you. Try to see what works best for you. A few suggestions:
Sticky notes: This can be a bit frustration to take notes, but can be really good for grouping and sharing back later.
Writing into a notebook: Generally comfortable to draw and take notes, but harder to share back. You can write the major points onto sticky notes as part of the share back.
Writing onto screen shots. You can print copies of the screens that the participant is using. This can be useful to have as well as unstructured notes. As the design progresses, this type of note-taking is more useful as you are looking to check the designs more than gather insights about the people.
Typing into a laptop. Typing into a Word or Text document can be fast and can be easily shared. It is generally not disruptive to the test, but won’t work for field tests.
Video: This will give you the richest, but most time-consuming. It is often useful to take the video as well as other notes, so you can go back to it to share with your team or if there is something you are not sure about.
Tools (e.g. Silverback). Silverback is software for Macs that captures a video of the participant’s face as well as the movements on screen.
If you are testing on your own: Focus on the test and less on the notes. Just note key moments, you particularly want to explore further. After you have conducted the test, walkthrough it again with the participant and write the notes together. Use prints of the screens to help remind them of what happened. Be open about what you are looking for and trying to do. Most people will be keen to help.

What to capture in the notes

• The steps they take to complete. This is useful early on in the design work.
• What works in the design. It is important to keep what works well.
• What fails in the design. If something fails, then ask “What would you prefer to see” as a way to try to uncover an improvement. Capture both the failure point and the idea for improvement.
• Points of confusion or frustration. Ask “What are you looking for” to get an understanding of what they are confused about.
• How people try to resolve any problem areas. Capture the different paths that people take to resolve a problem.

What to do with the results
It is important to share the results back with your team as soon as possible. Ideally, do one test, then share back what you found. Even if you are taking notes, you will remember additional observations.

• If you have an hour or two: Group the observations, first about the person and how comfortable they are with the using computers/mobiles or whatever you are testing on. Then focus on the design. Cluster the findings and add a theme above each. The themes should read like a story.
• If you have no time: then focus on the changes to the design. What worked, what didn’t and what kind of worked, but needs improving.

What if it goes wrong
In rare cases, the person who you are testing with is antagonistic. A few people can be confronted by it and feel under pressure no matter how much you reassure them that it is the system you are testing. In this case, wrap it up. Thank them and give them their gift. Always give them the gift even if you didn’t get what you needed.
Relax
Once you are through your first test, you will be amazed at how easily you can use this technique. I tend to say it is almost impossible to get this wrong. A small caveat on that is, provided you have someone who hasn’t seen the Product being designing and you genuinely observe and don’t help. Staying quiet is still our biggest challenge.

References
Here are a few references that I have found useful.

Rocket Surgery Made Easy, Steve Krug
This is his second book in 10 years. It is easy to read and clearly outlines how to conduct a user test. (His first and wildly successful book is “Don’t Make Me Think”). He includes a number of downloads on his website at:

Useful Variations on User Testing, Jared Spool

Best way to find people, Dana Chisnell

Handbook of Usability Testing, Jeffry Rubin and Dana Chisnell
This is a cornerstone handbook on how to conduct usability tests. It does focus more on lab-based testing, but it is very useful when bringing customers in-house to conduct the testing.