User testing metrics and how to use them

User testing is typically considered a qualitative activity, and for this reason most teams opt to skip the success measurement part of this activity. However, user testing activities can actually be tracked and measured, and the results are tremendously useful in supporting recommendations, getting internal/external buy-in, and even with fundraising. Here are a few metrics that you can track and measure for your user testing activities and projects.

 

Task completion 

When planning the things that you want to test with users, split them into a number of intended tasks that have a clear start and a clear stop. Examples could be signing up from your website or app, making a bill payment, or booking a meeting. During your test session, make sure every user starts at the same step, and measure completion of the task consistently across the board. Now when you summarize the results, you can gather data on how many users completed a particular task, vs how many users did not complete the task. Use this opportunity to outline the things that users found intuitive, vs the things that were not to support these results.

Time to complete

Measuring the amount of time it takes for a user to complete a particular task is also a powerful metric to use to get the most of your user testing sessions. At the beginning of the session, you should let the user know that the tasks will be timed, and that you are looking at the time taken across multiple users, not as an indication of pass or fail so that the user feels at ease. Now when you analyze of the results, you can clearly see how much time it takes a user to complete a particular task, and use it to benchmark the time against other tasks on your product/platform, against industry standards or competitors, and see if all your users are taking a similar amount of time. This can help you identify where you might be able to streamline the user experience further.

Number of users in specific segments

Another metric to use is not task related, but rather specific to user profiles. You can look through the profiles of each users that you ran the test with, and see if there are commonalities between the users, or anomalies in the profiles. For this, you can look at the base dimensions such as Gender, Location, Device, Industry, or Job Title, but may also extend to other behavioural dimensions such as people who shop at Store A vs Store B. This metric can really help surface the data behind your user testing efforts, and bring more insights without having to do more user testing sessions. 

Number of issues

Now as the users are asked to complete multiple tasks, the number of issues and challenges they face can also be another great metric to track during user testing sessions. This can bring to light the magnitude in which users are facing issues or challenges with your product/service com. This metric can also be used as a goal for you when you set off on the design of a specific feature or product, and provide you with evidence to pivot or consider a different design. Alternatively, it can also help your work standout if there are few or no issues discovered during testing.

Votes on alternatives 

When designing a new product or feature, in almost all cases you and your team may be searching for evidence to go with one of multiple alternative solutions. User testing can be a great activity to do to help support this decision. When you structure your test sessions, run the user through the same set of questions and tasks on two versions (we don't recommend doing more than two versions). Once you've gone through both options, ask users which option they prefer, and keep the count in your summary of findings. To keep the results unbiased, ensure you swap the order of presentation of both options, in case users always go with the second option vs the first, etc. This metric can help you break a tie when trying to make a decision in your team!

These are just some ways that you can measure the success of your user testing sessions, and maximize the value that you get from them. Whether it's task-related metrics, user segmentation, or simply asking users to vote, putting quantifiable metrics behind your user testing sessions will ensure that you continue to get maximum insights, and also buy-in to continue doing them!