Think aloud

Just another weblog

2nd day of testing

leave a comment »

It’s my second (and last day) of testing today – and I’m battling a flu and a throat ache. Of all the days to get sick! There’s no way anyone can take over for me, and rescheduling would be a nightmare, so I’ll muddle through, clipboard in one hand, painkillers in the other.


Written by Jules

June 8, 2009 at 8:49 am

Posted in Uncategorized


leave a comment »

Reviewing the audio tapes gave me opportunity to observe my own behavior during a test.  I also noticed a few things about my test user’s behavior, more specifically their behavior and thought patterns. Peculiarities that I should be aware of, the next time I do interviews.  I can’t let those things get in the way of the information I need to get.

Here’s what I noticed users do:

Users don’t finish their sentences

I wanted to cut a few quotes out of the audio tapes, to put in my report. I listed a few of them in my notes, like:

“Well I would never look for the Print layout function there, I’d expect it in the right corner”.

Perfect quote, right? Except, when I look it up in the audio tapes, what’s actually said is something more like this:

“Hmmm yeah.. (pause).. I don’t… eh… this function, I don’t think… I don’t see.. Here might be bet- but I guess the date is in that spot already.. hmms”.

Leaving me very little to work with. My coworker in press relations says it’s very common for people not to finish their sentences. They give media training to prominent figures to teach them things like that: to speak in complete sentences, and to give remarks that are easily quotable.

Of course my test users didn’t receive any media training, and if they did, they might not be as useful at test users. But up until I listened to those tapes, I never noticed how many people don’t finish their sentences. Now I notice it more and more.

Users don’t answer the question

At some point during the interview I asked someone “Do you find you use one medium for a specific kind of news?” and the test user went on to explain to me that he usually watched the TV news at night and surfed the internet in the morning. A few times also, I asked a question, and test users kept on answering the previous question. I guess I should have given them more time for each question, but there’s only so much time to go around!

I have to keep in mind that some part of the message always falls away, even in a two-point telephone game. A few times I was quick enough to notice the user wasn’t answering the question, and posed the same question again in a slightly different way. A few other times I didn’t catch it at all, and only noticed it when it was far too late.

Users have a hard time expressing themselves

I had a few hard questions in my interview. Questions like:

  1. “Do you use a particular kind of medium for a specific kind of news?”
  2. “What sort of links do you usually click on in an e-mail newsletter?”
  3. “Why is staying updated on current events so important to you?”

Exploratory questions whose answers are very interesting, giving subtle cues to their personality and to their needs. But they’re very hard for users to answer, requiring a lot of self-reflection. As I mentioned above, I sometimes noticed that two questions on, my users were still thinking about the first question they were asked! I

t might be good to make these questions easier and less abstract. It will require more work on my part, but the results might be better. For example:

  1. What kind of news do you watch on TV? And on the internet? And on the radio?
  2. (handing over an e-mail with 5 different kinds of links as identified in previous research) Which of these links appeal to you? Can you explain why?
  3. How do you use the news information that you gain? What else do you do with it? Which of those things is most important to you? Why?

Users constantly apologise for their behavior

“Yes, I overlooked that menu. It’s probably because I’m not wearing my glasses. I’m going to get some next month”
“That navigation was confusing but that’s probably just me, I normally only use the search bar”
“I missed that link but I’ve never been to this website before, so that’s normal, right? I should really look at this website more”.

In four interviews I heard countless excuses as to why it was their fault if something went wrong, definitely not the website’s fault. I wonder why. Is it because I didn’t stress enough that 1) it wasn’t my website, and 2) we’re testing the website, and not them? I certainly tried to explain those things very thoroughly.

Or is it just human nature to think “I’m not a typical human, they don’t need to take me into account”. I wanted to explain that a website should also work for badly-sighted people, for search bar-navigators, and for people who visited a website for the first time.  But I didn’t, I didn’t have the time and it wasn’t what I was there for.

Users will point out problems you never even thought of

In my very first test, my user got to a lower importance news page and said “wow, this page is horrible”.
“What makes you say that?”, I asked.
“Well I like to quickly scan the  titles to see which news items are of interest me, and here the titles are the same size and type as the rest of the text, I can hardly make them out”.

He was right, of course, and I felt pretty stupid for missing it in the first place. It wouldn’t be the last time a user pointed out something I never considered. Or thought of a solution or a new feature that I hadn’t considered yet.

Users will bend over backwards to help you out

Sure, they get a nice 20 euro voucher to spend in everyone’s favorite electronics store. They may come to your office for the money, but once they’ve arrived, they’ll do whatever they can to help you out. They’ll dilligently search through a webpage for five minutes for a feature that’s been styled to look like a disclaimer (good job, design guy!), they’ll answer difficult questions they normally wouldn’t even think about, they’ll scour their mind for any information that they think might help you. They wait patiently while you sort your task scenarios, they speak their mind eventhough they have cameras and audio recorders pointed at them.

Maybe there’s very little you can say about “the typical test user”. But I can sure say that “the typical test user” is certainly a very nice guy.

Users will prove you wrong

The newssite I’m testing has little quotes from famous people on the bottom of each page. “How stupid!”, I thought. “Totally unprofessional and not fitting to our image. I’ll put it in the test and we’ll get this thing sorted quickly”.

Except every test user I’ve had so far thinks they’re fun. Has no problem with them. Don’t think they detract from our image. One of them went so far as to say: “They’re fun, I don’t see how anyone could be bothered by them! You’d have to be a pretty sad person to find something like that objectionable”.

Sometimes what you expect to happen is nowhere near what will actually happen. That’s quite a grateful moment, because it’s when you realise you’re doing all this work for a very good reason.

Written by Jules

June 5, 2009 at 8:47 am

Posted in Uncategorized

Interview mistakes

with 2 comments

mistakesLast friday, my first four test users came in one by one, and I sat them down for an interview and a usability test after that. I was quite content with my own performance, but now that I’m looking through the audio- and videotapes, I see mistake after mistake. Don’t get me wrong, I still got a lot of information out of these interviews. A little too much, when I see how much I still have to process, and how lovely the weather outside is. But I could have gotten more if hadn’t been making these errors. I’ll note them here so they will seep into my conciousness and I’ll have a chance to work on them:

  • Not asking open questions
    I wrote down a lot of open questions in my interview plan. One could be : “What do you think of this website?” . But on the audiotapes I catch myself saying: “What do you think of this website? Do you think it’s interesting or boring or something else?” With every open question I’ve added some possible answers, thereby steering the entire response into one direction. The user will now answer the question with a scale of boring-interesting in mind, potentially disregarding his own spontaneous thoughts because they don’t fit in this spectrum.
    What I should do: Just ask the open question the way it’s on your page and let the user formulate his own answer. If he doesn’t understand the question, then you can expand on it a bit.
  • Not leaving room to think
    A few times in the audiotapes the test user finishes his answer and there’s a short pause. Then both I and the test user start talking again. Often then the user stops and I keep talking. Cringeworthy.
    What I should do: I should really count to three before I start talking again.
  • Not staying on topic
    One remark a user made reminded me of a similar short story, that I then related. Why? I don’t know. It just came out of my mouth. It didn’t break the flow of the interview but when i listen to the tapes again it feels out of place, or at least a waste of valuable interview time.
    What I should do: stay focused on your task.
  • Not being detached
    My first test user was a very attractive young man. The audio replay shows a lot more expressiveness in this interview than in the other ones. I am ashamed.
    What I should do: No really, just stay focused on your task

Written by Jules

May 31, 2009 at 2:24 pm

Posted in Uncategorized