Asking (very) personal questions in an application process

20 Nov 2015

Personal questions to create a Social Security account

If you create a my Social Security account online, the Social Security Administration (SSA) uses their “identity services provider” (the credit bureau Experian) to get information about you. You have to answer them to prove that you are you. Here’s a sample:

Text: You may have opened a student loan in or around February 1999. Please select the lender that you have previously or you are currently making payments to. If you have not received student loans with any of these lenders now or in the past, please select NONE OF THE ABOVE/DOES NOT APPLY. [Followed by radio buttons with choices]

Example questions in the my Social Security application (taken from a SSA online video)

If you can answer enough of them correctly, you create the account. If not, you have to wait 24 hours and try again. (That gives you time to look up old phone numbers and financial information.)

Personal questions in a usability study

In a recent usability study, participants entered a home address as part of an application process. The site then displayed a list of people who live there and cars that are registered to them. Some people thought it was an invasion of privacy or creepy. Others liked the convenience.

The range of reactions didn’t surprise me. What did surprise me is that no one was upset enough to want to leave the site, although some people wanted to know where the information came from.

Another surprise: Seeing that the website could look up some data seemed to change expectations. A few participants assumed that default values elsewhere in the form were not just suggestions, but other data that the site found about them. That seemed to make them more likely to accept the default values. (This needs more investigation, but it was an interesting observation.)

What do you think?

Have you seen personal questions like these in other situations? Have you created an account at SSA? Was it surprising to see what they could find out about you?


What do they do when they can’t do what they want to do?

5 May 2014

I watched people create forms in a study and saw a creative workaround for an unclear feature.

The model that the participants were copying had a Submit button, and the program automatically inserted one in the form. That seemed like a convenience, but there was no way for users to know that would happen.

There was no way to insert the button because it was automatic. One creative person inserted a text box and made the default value be “Submit”:

One form with a text box that says "Submit" and one with a Submit buttonIt looked like a Submit button, sort of. And it made the participant feel like he accomplished the task, but it was confusing and stressful to figure out.

The lesson is that people often do things we don’t expect. Sometimes customer research like this usability study is the only way to find that out and prepare for it.

How do you know if your customers want that feature you’re planning?

23 Dec 2013

I took a client out to talk with customers about a big new feature they were planning. After just a couple of interviews, it became clear that customers didn’t want the feature. A couple of interviews saved a lot of time and expense.

It’s amazing what you learn by watching and listening. My favorite work these days is customer research. I love learning how people work, and how they think about what they do.

Here’s an overview of customer research along with some favorite methods and examples of important things we’ve learned by listening to customers over the years.

You might see it called different things, like customer research, design research, user experience research and user research. But it’s all about listening and learning.

Why we do customer research

There’s no point in just guessing how people work or think or use your product. As Jakob Nielsen said back in 2004:

“User research is a reality check. It tells you what really happens when people use computers. You can speculate on what customers want, or you can find out. The latter is the more fruitful approach.”


Here are some of the research methods that I’ve used over the years:

  • Customer interviews. Conversations with individual users about an aspect of their work. We might look at the product that we’re interested in, but the main goal is to talk about tasks and needs.
  • Usability studies. A more-structured observation of an individual user working with a product or a prototype. We learn about user behavior, and have a chance to observe actual work, and ask questions while the user is working.
  • Design Workshops. My Design Workshop starts as a brainstorming session, but I ask participants to sketch out their ideas, which makes them think a little harder. It’s a great way to explore concepts or start a new design project.
  • Surveys. This is a way to get data from a lot of people. Followups by phone or email allow some deeper discussion. A survey might be based on what we learn in customer interviews.
  • Card sorting. This is a great way to find out if an existing or proposed information architecture matches the way real users think about the data.

Team members observing

I always want client team members to participate or observe. They may be in the next room watching a video monitor, or observing via GoToMeeting, or going out to visit customers with me. I ask them to take notes and we have debriefing sessions to share our observations.

They learn a lot, too. Here are some examples of what team members have said:

“I never believed it would be so hard. I never thought people would have used it this way.”

“What we saw him do wasn’t what we expected.”

Observation leads to product improvement because the more observers there are, the harder it is to ignore problems that turn up.

Things we’ve learned

In all cases, the goal in customer research is to find out what people think and how they work. Research results lead to design changes in the short term, and to better UX strategy in the longer term.

We use data from customer research to build and refine our personas, too.

These are some of the things I’ve learned by listening to customers:

  • Web tools for the construction industry. We stopped development on a new feature because the target audience didn’t want it. This saved a lot of development time.
  • Dictation tool for physicians. “Physicians don’t do that.” A doctor made it clear that office staffers are the ones to enter demographic information, so we changed the workflow.
  • Business reporting site. Customer visits turned up a need for business owners to delegate certain tasks to employees. They wouldn’t have been able to use the tool if we hadn’t learned about this.
  • Consumer software. Even though customer surveys showed great satisfaction, when I ran this client’s first usability study, it turned out that no one could actually use an important part of it.
  • Online magazines, online courses & business sites. People have told us over the years that they’re a bit shy about commenting on what someone else has written online.
  • Sales productivity tool. Observers saw problems they had not seen in the lab. They started a fix right away.

Is it time for you to start doing customer research?

Have you talked with your customers recently? Let’s talk about ways to improve your products through customer research.

Setting up a usability study for a mobile app

31 May 2013

I recently ran a usability study on a mobile app. Software like Morae makes it easy to record, observe, annotate and analyze a study using a laptop or desktop computer. But there’s nothing (yet) like that for testing mobile apps, so the setup was the complicated part.

This is a summary of how I set up a study on a mobile phone app.

Learn from others

First, I learned from a colleague by assisting her during a study she ran. I ran my study based on what I learned from her and what I discovered along the way.

Learn from others: find someone who’s done a study like yours, or find a blog post like this one.

My setup

The participant and I sat in one room, while we had a note-taker and observers in an adjacent room. We also had team members in remote locations. They all had to see and hear what was going on.

Did I mention that the study would occur 1000 miles from where I live and work? That was a bit nerve-wracking, but I had great help.

I sketched what I thought the setup would be and sent it to the people onsite for our discussions:

A sketch of the equipment setup

The left side represents equipment in the testing room. The right side shows what would be in the observers’ room. But it needed a little adjustment.

Here is a photo of the actual setup I used, followed by a discussion of how it worked and a list of what did not work.

The hardware and software setup in the testing room

What worked

  • Devices under test: The software ran on iPhones and Android phones. In the photo, the iPhone is on the right side of the screen, where the participant sat.
  • Positioning the phone: This involved conflicting goals: The phone had to be under the camera for recording and video transmission to the observers’ room. But the participants had to be able to use it as naturally as possible. Here’s the compromise: It’s sitting on an inverted clipboard, which tilts it at a reasonable angle for viewing by the participant. The clipboard is held in place by gaffer’s tape, and there’s a tape loop holding the phone in place. The participants were more important than the observers, so I let them pick up the phone when they needed to, but then asked them to replace it. There was an outline of the phone drawn on the white paper covering the clipboard as a target for replacing the phone.
  • Camera & microphone: Above the phone is an IPEVO Ziggi-HD document camera. It’s very compact and has a microphone. It connects to the laptop via a USB cable.
  • Image capture: You can see the image of the phone on the MacBook Pro laptop. It’s displaying the phone’s screen through the IPEVO Presenter software that comes with the camera. It was easier for me to watch the study on the laptop screen so I didn’t have to literally look over the participant’s shoulder. The Post-It on the laptop has all the resolution and exposure settings that seemed to work well. I always have a checklist of things to do between sessions and to start a new session, and this became part of it.
  • Recording: I used TechSmith’s Camtasia to record the audio and video during each session. It’s largely invisible during the study. (See below for what did not work.)
  • Remote video & audio broadcast: Audio for all the observers (local and remote) came from the camera’s mic, video from whatever was on the screen. As noted, we had remote observers, so I used GoToMeeting to transmit the audio and video to them. As long as their phones were muted, it was fine. The GoToMeeting UI is on the right side of the laptop screen. It can be minimized, but it can remain open next to the Presenter window so I could communicate with them via the built-in chat feature. Because I was mirroring my display in the observer room, the local observers could see any chat messages if the UI was exposed. (Mirroring the main display on a secondary monitor on a Mac is pretty easy, and happens in Displays System Preferences window.)
  • Local video broadcast: I connected a mini-DVI adapter to the Mac, and connected that to a long cable that went to a monitor in the observer room, which was next door.  (See below for what did not work.)
  • Local audio broadcast: Camtasia will feed video to an external source, but not audio. The simplest solution was to make a phone call from the desk phone in the testing room to the observer room, put it on speaker phone there and mute it. We just left the connection open all day.
  • Backup: What’s the worst thing that can happen when you’re recording a session? Losing the recording. So I backed up each one to a local server immediately. The files were 3GB each, so it was slow over the wireless system I had to use, but it worked.
  • Communication with observers: I’ve tried many things over the years for communicating with observers, including an earpiece that they could talk to me through. Now I use text messaging. My iPhone is in front of the desk phone, but during a session, I kept it on the laptop so I could see messages that came in.
  • Note-taking: Morae is great for taking notes in a usability session because it time-stamps them and they’re part of the video timeline. The app LogIt is good for simple time-stamped notes on a Mac, but for my PC note-takers, I created an Excel file that put a time stamp in column B as soon as they entered any text in column C.

So how long did it take to figure that out? Quite a while. Plan for that, and do a number of dry runs to be sure. The stopwatch app on the iPhone is good to run while you’re testing because it constantly updates, and you can tell if the video freezes.

What did not work

  • Recording: I didn’t use GoToMeeting for recording, because I have had some problems with that on the Mac platform. My laptop didn’t reliably record with QuickTime because it’s a mid-2009 model, so I switched to Camtasia. (My colleague used QuickTime to record her study, but her MacBook Pro is a year newer than mine.)  A benefit of Camtasia is that it writes the recording to disk as it progresses; QuickTime saves it in memory, which means it’s volatile and can disappear if various interruptions occur.
  • Local audio broadcast: Another thing that the mid-2009 MacBook pro doesn’t do is transmit audio over an HDMI connection, so I used the desk phone for audio. My colleague’s computer did work with HDMI, and she used HDMI to get audio and video to the observers. The item in the sketch (above) labeled “sound cable” didn’t work out because Camtasia doesn’t support monitoring audio.
  • My computer: Well, it did work, but I was afraid that it would melt, with Camtasia recording, the backup copying and GoToMeeting broadcasting. But it worked fine.

That’s why we test things. And call Tech Support (and they were very helpful at Camtasia).

And next…

Things will work differently for you, but this is one place to start. And now that I figured this out, I’m doing another study just like it, but with a few twists:

The next study

The next study, using an iPad and a picture-in-picture recording with Camtasia, the document camera and the built-in webcam.

Comments about online commenting

3 Jan 2013

Do you write comments online? Commenting is available all over the Web. It’s a great feature, but there are some things that can hold people back from using it.

In usability studies, I’ve seen people hesitate about writing comments for many reasons. (Even so, I hope you comment about your experiences, below.)

Example of a commenting UI

Example of the commenting UI in this blog

Social factors affecting posting & comments

People are concerned about what others will think of them. Unless you’re really sure of your point, questioning someone online, especially an expert, can seem scary. I’ve heard participants say that in usability studies.

In his book Design to Thrive: Creating Social Networks and Online Communities, Tharon W. Howard writes:

Because people don’t want to damage their reputation or their professional status in the community, they won’t post unless they’re fairly sure that their contribution will be received as a contribution and will not open them up to critique.

He suggests demonstrating “how [members] can ask questions productively” and in general, creating a safe and comfortable environment.

Anonymity and rude comments

Anonymity can make people more comfortable about posting comments, but if my local newspaper’s site is any indication, it encourages some people to be rude. A recent usability participant talked about that:

Online, people everywhere are willing to insult people’s mothers at the drop of the hat. Anonymity leads to people saying a lot of things.

On the NPR program, Science Friday, Dominique Brossard, lead author of a paper in the journal Science (“Science, New Media and the Public”) said that negative comments even affect how others interpret the main discussion:

So basically just being exposed to rude comments, even if the content of the comments themselves was the same, made people react differently to the content of the story. So the question is therefore: What do we do to encourage better understanding? … And then we should discuss what does it mean to have a civil discussion online.

Systems like Facebook and LinkedIn certainly allow people to comment on their own lives and other people’s posts, and they’re obviously popular. It may feel safer in those systems because you only sign up if you want to do it, and you have a controlled audience (depending on how well you understand and use your privacy settings).

UI issues in commenting

The commenting UI itself sometimes gets in the way, as I saw in a usability study awhile ago. The commenting form might appear at the top or bottom of the article. It’s not useful at the top, because you haven’t read anything yet. You might not get to the bottom to see the form there if you don’t read the whole article. And if you print the article to read, there’s no easy way to post a comment. Having a link float on the page so it’s always visible might be helpful but requires careful testing to see how people react to something that keeps moving.

In a system with a lot of topics (whether they’re courses, articles or products), many items may have few or no comments at all. That may make them look unpopular. Seeding a discussion, by having a staff member or the author post the first comment or question might help it look less like a vacant lot and more like a conversation.

Creating a sense of community

I belong to a professional email list that is for members only. That restriction is useful; it’s one of the things that the moderators do to make people comfortable. Even though most of us will never meet, we have a definite sense of community from having helped each other on topics of mutual interest.

In one recent study, a man indicated that he writes comments in an online course’s discussion area. When I asked for details, it turned out that he hadn’t written anything at all. He was really talking about what he might do, not what he actually does. (That’s why it’s important to not simply accept what people say in customer research and usability studies.) He prefers to talk with friends who also take the class; it seemed that he was just more comfortable with people he knows.

Your turn…

…In what situations do you post comments, if any? What makes you nervous or comfortable about doing it? Should we moderate comments, or let anyone say anything? Is it different if it’s social or professional? And don’t worry – I moderate the comments here and won’t let anyone be rude.

Web sites with attitude

11 Jan 2011

I’ve seen a couple of sites lately with mascots that have something to say and say it with an attitude. I guess whether you think their comments are clever depends on your point of view. Or your age. Or what you expect from a business Web site. Or your mood. Or something.

Here are some examples.


MailChimp is an email marketing service. Their mascot is a chimp, and he’s got a lot to say. Sometimes it’s clever, sometimes it’s, well, odd.

MailChimp saying "Monkey see, monkey do. I'm only wearing a mail bag, how about you?"

MailChimp saying "Hal, I'm just a poor boy from a poor family."

MailChimp saying "Hi Gail. New shirt? Very nice."

Notice that the middle one has a link. It goes to a YouTube video: “The Muppets: Bohemian Rhapsody”. They link to many clever things around the Web.


You’ve probably heard about Groupon: an online service that offers coupons for businesses in your area. Each day’s coupon comes via email, with a picture of Groupon the cat and the saying of the day. They’re funny, silly or just a little strange. Here are some examples:

Groupon cat and the statement of the day

Groupon says "Which dried fruits are specifically processed for the socially rejected?"

Groupon says "I named my son after my husband and my husband after my boat."

If you click on the arrow in the email they send, you go to a Web page describing the offer, and more detail about the saying. See what you get by clicking on the one about children and matches.

This one doesn’t have a mascot, but…

This one doesn’t have a mascot. But the business name and the domain name are a bit unusual:

Logo for Shit Creek Consulting

Note the the logo: they have the paddle.

And they have the right attitude for the name. Here’s part of the description: Smug? You bet your ass we’re smug. We won’t waste your time and all we ask is that you don’t waste ours with your indecision. Call us if and when you are serious.

So what?

Well, that’s a good question. Does having an attitude like this matter? Does it reflect a big change on the Web or in business? Or is it just a few companies pushing the boundaries?

I observed a usability study where some people were put off by an attitude like these examples. And I have to say that some of these comments surprised me. I have to guess, though, that they know what they’re doing. Maybe they don’t want customers who offend easily, or who expect a pin-stripe-suit attitude. Or maybe times have just changed.

You’d certainly want to know your users before doing this. And you’d want to carefully develop good personas for your development team to clarify the boundaries.

What do you think?

What have you seen? What do you think when you see a site with an attitude like this? Send me a note or write a comment below. If you have other examples, include a link.

How do people think about order in lists?

10 Dec 2010

How do people find items in a list?

When people see a list, they want to understand its organization. If it isn’t alphabetical, the order should be clear and related to the information and the task. You want your users thinking about the contents of the list, not its presentation.

I’ve seen different types of problems when usability study participants were looking for information in lists. Here are some:

  • A participant looked at a list of database field names that were ordered by the date they were added. He said, “There’s no order to these fields, so I have to fight this every day.”
  • Another participant sarcastically referred to the “alphabetized list of our 1000-plus reports”. While it was alphabetical, there were too many items to scroll through. A hierarchical list, or an easy way to remove old reports, or some filtering methods would have helped him.
  • In a related study, there was a long list that was organized hierarchically, but it was shown in a very small window. It was like looking into a busy room through a keyhole – people could see a little bit of information at a time, but never get a good feeling for the list as a whole.
  • Looking at an online discussion, a participant in another study remarked that he wanted to see “a threaded-message view”, which is another form of hierarchy. It was as a flat list, where it was hard to tell whether a message was a topic, a response or a response to a response.
  • Search engines generally show results sorted by relevance. In the early days of search, results included a relevance indicator (e.g., a bar whose length shows the relevance of each result). This isn’t used any more because, as we saw in usability testing, they weren’t meaningful to users.
  • Another recent study showed the top five items in a couple of categories. Some participants wondered how the list was put together. It could have been editorial choice, most-viewed or most-emailed. Understanding the source of the list would affect how they interpreted it.

Jakob Nielsen, in his Alertbox column wrote about a number of cases where alphabetical order isn’t the best choice, and other situations where alphabetical order was used, but presented in such a way that it wasn’t clear. Good points that illustrate the need to find the right order and present it in the right way.

One thing to take into account is how people will use the list. Understanding how each user type (or persona) approaches the task can help you decide to use alphabetical order, form groups that create a hierarchy, or find an order specific to your use.

Categories are shown by creation time, which doesn't help when you want to apply one next time

Search results from with Sponsored Links shown first

Users may not notice that the first few search results are ads. Is that OK?