Effective User Testing

 

Which quadrant do you want to be in?

 

Which quadrant do you NOT want to be in?

 

What about the other two quadrants?

 

How do you make sure you're in the top half?

User Testing!

 

Why User Test?

User Testing Challenge

Watch, Don't Ask

User Testing DON'T's

  • Don't have any buttons or links that don't work
  • Don't have an ugly OR a flashy design
  • Don't start with a tour of the app
  • Don't ask them to play with it (unless it's a game)
  • Don't help them when they look confused

User Testing DOs

User Testing

  • BEFORE: Write everything out in advance
    • Product box, pre / post surveys, task scenarios
  • DURING: Shut up, watch, record and take notes
    • Don't tell them what to do
    • Answer questions with questions
    • Let them make mistakes. Can they recover?
    • Look for friction and confusion
  • AFTER: Analyze
    • categorize and tally mistakes, confusions, hesitation, ...
    • focus on most critical and common problems

Pre-test Surveys

  • A very few questions about things that you hypothesize best describe your ideal user
    • E.g., for a recipe app, "how many times a week do you cook?"
    • Avoid subjective questions like "are you a frequent traveler?"
    • Demographic information (age, gender, ...) only if relevant

Task Scenarios

  • Written setup for a problem situation. Include realistic local details
    • "It's 4:55. You're just finishing a math class in Lunt. Your 394 team is meeting in Tech at 5:30. You're hungry. You'd like to see if any place on campus open right now is serving something vegan."
  • No instructions on how to use the app
  • User Testing Scenarios and Tasks

Post-test Surveys

  • A page of questions (at most) to see how users perceived your app
    • What do you think the point of this app is? Who is it intended for?
    • What were the 3 best and worst things about this app?

How to ask questions

Distractors

  • Things users clicked on, asked about, even though not obviously relevant to task
  • Could be a sign of what you should really be building
  • Could be an evil attractor

Getting Testers

  • Who you test matters at least as much what you test
  • Don't just find some random friends
  • Testers have to align your value proposition
  • Don't test a recipe app on people who never cook...
  • ... unless your app is meant for people who don't cook
  • Look in the places where your target users are more likely to be

After You Test...