Setting up a usability study for a mobile app

31 May 2013

I recently ran a usability study on a mobile app. Software like Morae makes it easy to record, observe, annotate and analyze a study using a laptop or desktop computer. But there’s nothing (yet) like that for testing mobile apps, so the setup was the complicated part.

This is a summary of how I set up a study on a mobile phone app.

Learn from others

First, I learned from a colleague by assisting her during a study she ran. I ran my study based on what I learned from her and what I discovered along the way.

Learn from others: find someone who’s done a study like yours, or find a blog post like this one.

My setup

The participant and I sat in one room, while we had a note-taker and observers in an adjacent room. We also had team members in remote locations. They all had to see and hear what was going on.

Did I mention that the study would occur 1000 miles from where I live and work? That was a bit nerve-wracking, but I had great help.

I sketched what I thought the setup would be and sent it to the people onsite for our discussions:

A sketch of the equipment setup

The left side represents equipment in the testing room. The right side shows what would be in the observers’ room. But it needed a little adjustment.

Here is a photo of the actual setup I used, followed by a discussion of how it worked and a list of what did not work.

The hardware and software setup in the testing room

What worked

  • Devices under test: The software ran on iPhones and Android phones. In the photo, the iPhone is on the right side of the screen, where the participant sat.
  • Positioning the phone: This involved conflicting goals: The phone had to be under the camera for recording and video transmission to the observers’ room. But the participants had to be able to use it as naturally as possible. Here’s the compromise: It’s sitting on an inverted clipboard, which tilts it at a reasonable angle for viewing by the participant. The clipboard is held in place by gaffer’s tape, and there’s a tape loop holding the phone in place. The participants were more important than the observers, so I let them pick up the phone when they needed to, but then asked them to replace it. There was an outline of the phone drawn on the white paper covering the clipboard as a target for replacing the phone.
  • Camera & microphone: Above the phone is an IPEVO Ziggi-HD document camera. It’s very compact and has a microphone. It connects to the laptop via a USB cable.
  • Image capture: You can see the image of the phone on the MacBook Pro laptop. It’s displaying the phone’s screen through the IPEVO Presenter software that comes with the camera. It was easier for me to watch the study on the laptop screen so I didn’t have to literally look over the participant’s shoulder. The Post-It on the laptop has all the resolution and exposure settings that seemed to work well. I always have a checklist of things to do between sessions and to start a new session, and this became part of it.
  • Recording: I used TechSmith’s Camtasia to record the audio and video during each session. It’s largely invisible during the study. (See below for what did not work.)
  • Remote video & audio broadcast: Audio for all the observers (local and remote) came from the camera’s mic, video from whatever was on the screen. As noted, we had remote observers, so I used GoToMeeting to transmit the audio and video to them. As long as their phones were muted, it was fine. The GoToMeeting UI is on the right side of the laptop screen. It can be minimized, but it can remain open next to the Presenter window so I could communicate with them via the built-in chat feature. Because I was mirroring my display in the observer room, the local observers could see any chat messages if the UI was exposed. (Mirroring the main display on a secondary monitor on a Mac is pretty easy, and happens in Displays System Preferences window.)
  • Local video broadcast: I connected a mini-DVI adapter to the Mac, and connected that to a long cable that went to a monitor in the observer room, which was next door.  (See below for what did not work.)
  • Local audio broadcast: Camtasia will feed video to an external source, but not audio. The simplest solution was to make a phone call from the desk phone in the testing room to the observer room, put it on speaker phone there and mute it. We just left the connection open all day.
  • Backup: What’s the worst thing that can happen when you’re recording a session? Losing the recording. So I backed up each one to a local server immediately. The files were 3GB each, so it was slow over the wireless system I had to use, but it worked.
  • Communication with observers: I’ve tried many things over the years for communicating with observers, including an earpiece that they could talk to me through. Now I use text messaging. My iPhone is in front of the desk phone, but during a session, I kept it on the laptop so I could see messages that came in.
  • Note-taking: Morae is great for taking notes in a usability session because it time-stamps them and they’re part of the video timeline. The app LogIt is good for simple time-stamped notes on a Mac, but for my PC note-takers, I created an Excel file that put a time stamp in column B as soon as they entered any text in column C.

So how long did it take to figure that out? Quite a while. Plan for that, and do a number of dry runs to be sure. The stopwatch app on the iPhone is good to run while you’re testing because it constantly updates, and you can tell if the video freezes.

What did not work

  • Recording: I didn’t use GoToMeeting for recording, because I have had some problems with that on the Mac platform. My laptop didn’t reliably record with QuickTime because it’s a mid-2009 model, so I switched to Camtasia. (My colleague used QuickTime to record her study, but her MacBook Pro is a year newer than mine.)  A benefit of Camtasia is that it writes the recording to disk as it progresses; QuickTime saves it in memory, which means it’s volatile and can disappear if various interruptions occur.
  • Local audio broadcast: Another thing that the mid-2009 MacBook pro doesn’t do is transmit audio over an HDMI connection, so I used the desk phone for audio. My colleague’s computer did work with HDMI, and she used HDMI to get audio and video to the observers. The item in the sketch (above) labeled “sound cable” didn’t work out because Camtasia doesn’t support monitoring audio.
  • My computer: Well, it did work, but I was afraid that it would melt, with Camtasia recording, the backup copying and GoToMeeting broadcasting. But it worked fine.

That’s why we test things. And call Tech Support (and they were very helpful at Camtasia).

And next…

Things will work differently for you, but this is one place to start. And now that I figured this out, I’m doing another study just like it, but with a few twists:

The next study

The next study, using an iPad and a picture-in-picture recording with Camtasia, the document camera and the built-in webcam.