Effective User Testing

 

Which quadrant do you want to be in?

 

Which quadrant do you NOT want to be in?

 

What about the other two quadrants?

 

How do you make sure you're in the top half?

User Testing!

 

Why User Test?

User Testing Challenge

Watch, Don't Ask

User Testing DON'T's

  • Don't give them any buttons or links that don't work or matter
  • Don't give them an ugly or flashy design
  • Don't give a tour of the app to them
  • Don't ask them to play with it (unless it's a game)
  • Don't help them when they look confused
  • Don't ask if they liked it

User Testing DOs

User Testing

  • BEFORE: Write everything out in advance
    • Product box, pre / post surveys, task scenarios, waivers
  • DURING: Shut up, watch, record and take notes
    • Don't tell them what to do
    • Answer questions with questions
    • Let them make mistakes. Can they recover?
    • Look for points of friction and unexpected attractors
  • AFTER: Analyze
    • categorize and tally mistakes, confusions, hesitation, ...
    • focus on most critical and common problems

Pre-test Surveys

  • Discover who might want and use your app
  • Short! 4 questions at most
  • Characteristics relevant to your target audience
    • E.g., if a recipe app, how often they cook, how adventurous they are, how do they get recipes currently
    • Not just age and gender -- way too broad

Task Scenarios

  • Set up the story: context, problem, realistic local details
    • "It's 4:55. You're just finishing a math class in Lunt. Your 394 team is meeting in Tech at 5:30. You're hungry. You'd like to see if any place on campus open right now is serving something vegan."
  • User Testing Scenarios and Tasks
  • No pretending to be someone else, e.g., "imagine you're a freshman..."
  • No instructions on how to use the app

Post-test Surveys

  • Discover how users perceived your app
  • A page of questions at most
  • Get past polite no-cost answers.
    • What do you think the point of this app is? Who is it intended for?
    • What were the 3 best and worst things about this app?
    • Would you be willing to test next version? If not, why not?

Attractors

  • Things users clicked on, asked about, even though not obviously relevant to task
  • Could be a sign of what you should really be building
  • Could be an evil attractor

Getting Testers

  • Who you test matters at least as much what you test
  • Don't just find some random friends
  • Testers have to align your value proposition
  • Don't test a recipe app on people who never cook...
  • ... unless your app is meant for people who don't cook
  • Look in the places where your target users are more likely to be
  • Use a very short survey to measure how well your testers match your target

After You Test...