I watched people create forms in a study and saw a creative workaround for an unclear feature.
The model that the participants were copying had a Submit button, and the program automatically inserted one in the form. That seemed like a convenience, but there was no way for users to know that would happen.
There was no way to insert the button because it was automatic. One creative person inserted a text box and made the default value be “Submit”:
It looked like a Submit button, sort of. And it made the participant feel like he accomplished the task, but it was confusing and stressful to figure out.
The lesson is that people often do things we don’t expect. Sometimes customer research like this usability study is the only way to find that out and prepare for it.
I took a client out to talk with customers about a big new feature they were planning. After just a couple of interviews, it became clear that customers didn’t want the feature. A couple of interviews saved a lot of time and expense.
It’s amazing what you learn by watching and listening. My favorite work these days is customer research. I love learning how people work, and how they think about what they do.
Here’s an overview of customer research along with some favorite methods and examples of important things we’ve learned by listening to customers over the years.
You might see it called different things, like customer research, design research, user experience research and user research. But it’s all about listening and learning.
Why we do customer research
There’s no point in just guessing how people work or think or use your product. As Jakob Nielsen said back in 2004:
“User research is a reality check. It tells you what really happens when people use computers. You can speculate on what customers want, or you can find out. The latter is the more fruitful approach.”
Here are some of the research methods that I’ve used over the years:
Customer interviews. Conversations with individual users about an aspect of their work. We might look at the product that we’re interested in, but the main goal is to talk about tasks and needs.
Usability studies. A more-structured observation of an individual user working with a product or a prototype. We learn about user behavior, and have a chance to observe actual work, and ask questions while the user is working.
Design Workshops. My Design Workshop starts as a brainstorming session, but I ask participants to sketch out their ideas, which makes them think a little harder. It’s a great way to explore concepts or start a new design project.
Surveys. This is a way to get data from a lot of people. Followups by phone or email allow some deeper discussion. A survey might be based on what we learn in customer interviews.
Card sorting. This is a great way to find out if an existing or proposed information architecture matches the way real users think about the data.
Team members observing
I always want client team members to participate or observe. They may be in the next room watching a video monitor, or observing via GoToMeeting, or going out to visit customers with me. I ask them to take notes and we have debriefing sessions to share our observations.
They learn a lot, too. Here are some examples of what team members have said:
“I never believed it would be so hard. I never thought people would have used it this way.”
“What we saw him do wasn’t what we expected.”
Observation leads to product improvement because the more observers there are, the harder it is to ignore problems that turn up.
Things we’ve learned
In all cases, the goal in customer research is to find out what people think and how they work. Research results lead to design changes in the short term, and to better UX strategy in the longer term.
We use data from customer research to build and refine our personas, too.
These are some of the things I’ve learned by listening to customers:
Web tools for the construction industry. We stopped development on a new feature because the target audience didn’t want it. This saved a lot of development time.
Dictation tool for physicians. “Physicians don’t do that.” A doctor made it clear that office staffers are the ones to enter demographic information, so we changed the workflow.
Business reporting site. Customer visits turned up a need for business owners to delegate certain tasks to employees. They wouldn’t have been able to use the tool if we hadn’t learned about this.
Consumer software. Even though customer surveys showed great satisfaction, when I ran this client’s first usability study, it turned out that no one could actually use an important part of it.
Online magazines, online courses & business sites. People have told us over the years that they’re a bit shy about commenting on what someone else has written online.
Sales productivity tool. Observers saw problems they had not seen in the lab. They started a fix right away.
Is it time for you to start doing customer research?
Have you talked with your customers recently? Let’s talk about ways to improve your products through customer research.
I bought stamps at the Post office today and saw that they have new terminals for swiping credit cards. It was this one, or one just like it.
The Ingenico iSC 350 Quick: where would you swipe your credit card?
Perhaps because I was looking at the pen, but I tried to swipe the card in the slot just under the pen. But that’s not where the card goes, that’s just a space between two parts of the device. The card goes in the space just above the keypad (which wasn’t lit up when I used it).
The funny thing is that the clerk said that I wasn’t the first person who did that! Maybe a little usability testing would have helped.
Update (7 Aug 2014)
When I bought stamps today, I noticed that the slot at the top of the machine was covered with tape. The clerk said they had to do it because so many people tried swiping their cards there.
Do you write comments online? Commenting is available all over the Web. It’s a great feature, but there are some things that can hold people back from using it.
In usability studies, I’ve seen people hesitate about writing comments for many reasons. (Even so, I hope you comment about your experiences, below.)
Example of the commenting UI in this blog
Social factors affecting posting & comments
People are concerned about what others will think of them. Unless you’re really sure of your point, questioning someone online, especially an expert, can seem scary. I’ve heard participants say that in usability studies.
Because people don’t want to damage their reputation or their professional status in the community, they won’t post unless they’re fairly sure that their contribution will be received as a contribution and will not open them up to critique.
He suggests demonstrating “how [members] can ask questions productively” and in general, creating a safe and comfortable environment.
Anonymity and rude comments
Anonymity can make people more comfortable about posting comments, but if my local newspaper’s site is any indication, it encourages some people to be rude. A recent usability participant talked about that:
Online, people everywhere are willing to insult people’s mothers at the drop of the hat. Anonymity leads to people saying a lot of things.
On the NPR program, Science Friday, Dominique Brossard, lead author of a paper in the journal Science (“Science, New Media and the Public”) said that negative comments even affect how others interpret the main discussion:
So basically just being exposed to rude comments, even if the content of the comments themselves was the same, made people react differently to the content of the story. So the question is therefore: What do we do to encourage better understanding? … And then we should discuss what does it mean to have a civil discussion online.
Systems like Facebook and LinkedIn certainly allow people to comment on their own lives and other people’s posts, and they’re obviously popular. It may feel safer in those systems because you only sign up if you want to do it, and you have a controlled audience (depending on how well you understand and use your privacy settings).
UI issues in commenting
The commenting UI itself sometimes gets in the way, as I saw in a usability study awhile ago. The commenting form might appear at the top or bottom of the article. It’s not useful at the top, because you haven’t read anything yet. You might not get to the bottom to see the form there if you don’t read the whole article. And if you print the article to read, there’s no easy way to post a comment. Having a link float on the page so it’s always visible might be helpful but requires careful testing to see how people react to something that keeps moving.
In a system with a lot of topics (whether they’re courses, articles or products), many items may have few or no comments at all. That may make them look unpopular. Seeding a discussion, by having a staff member or the author post the first comment or question might help it look less like a vacant lot and more like a conversation.
Creating a sense of community
I belong to a professional email list that is for members only. That restriction is useful; it’s one of the things that the moderators do to make people comfortable. Even though most of us will never meet, we have a definite sense of community from having helped each other on topics of mutual interest.
In one recent study, a man indicated that he writes comments in an online course’s discussion area. When I asked for details, it turned out that he hadn’t written anything at all. He was really talking about what he might do, not what he actually does. (That’s why it’s important to not simply accept what people say in customer research and usability studies.) He prefers to talk with friends who also take the class; it seemed that he was just more comfortable with people he knows.
…In what situations do you post comments, if any? What makes you nervous or comfortable about doing it? Should we moderate comments, or let anyone say anything? Is it different if it’s social or professional? And don’t worry – I moderate the comments here and won’t let anyone be rude.
The City of Boston recently announced the Boston Meter Card, a prepaid card to use at parking meters. It’s a great idea, but it was impossible for me to figure out because the card doesn’t work the way other cards work. You have to insert the card and keep it in the meter for 10 to 15 seconds.
What would you do when you walked up to a meter with the card? I thought about which way to put the card in, inserted it, took it out, and… nothing.
I was there with someone else, and we couldn’t figure it out. Was the card broken? Was the meter broken? What else could I have done?
Good thing I had quarters.
It doesn’t work the way you’d expect
When you insert the card, you have to hold it in for 10 to 15 seconds and wait while the small display updates a number of times. But you knew that, right?
Problem #1: It doesn’t work like any other card I use. I couldn’t figure it out. Was it user error, or a system-design problem?
Videos of using the Boston Meter Card
Watch video footage of checking in and out of a meter. It’s hard to read the display, but that’s part of the real-life situation.
Now that I know how it works, I understand the transitions in the display:
00:00 – there was no time on the meter when I arrived
25.00 – I have $25.00 left on the card
In – I’m checking in
4:00 – the maximum amount of time to park
The first time I tried the card, it took the full 15 seconds to get a response. It didn’t display “In” that time, but it did display “1111” for some reason.
How long do you have to wait and watch? And how many changes will there be? Not knowing makes it hard to know when it’s complete. Is it clear what each display means?? There was no explanation, and it was impossible to figure out the first time. A brochure came with the card, but didn’t mention any of this.
Problem #2: The displayed information isn’t always the same for the same operation.
Checking out of the space was even more confusing because there were more transitions in the display to figure out:
These were the transitions for checking out:
2:18 – the time left when I got back
1111 – no idea, what do you think?
1:42 – the time I had parked and would pay for now
22.85 – the money I would have left on the card
OUt – I was leaving
00:00 – the meter was reset and now had no time
Problem #3: There’s no way for a first-time user to know how many display transitions there will be, so there’s no way to know how long to wait before removing the card. (I think you have to wait, but I didn’t test that.) And it’s not clear what it all means.
It works like … nothing else
Even if you use an older ATM that holds on to your card, it reacts within a second or two. Most card-reading machines have instructions saying to “swipe” or “dip” the card; this was the only one that would use a word like “wait”. Here’s an example from a hotel I recently stayed at:
This hotel key card responded within a second. All I had to do was "dip" it in and remove it.
Using the card the first time
The first thing was to figure out how to insert it. This photo shows a graphic on the meter that corresponds to the chip on the back of the card. It’s hard to see and it’s not clear what it means.
The arrow points to a graphic that looks like the chip on the back of the card. Is that enough to tell you how to insert the card?
The sticker just below the slot would have been a good place to put some instructions. That would have been easier than trying to decipher that little mark under the slot.
Problem #4: The display is hard to read in bright light, and probably worse at night.
I inserted the card different ways, but it didn’t react (because I didn’t know to hold it in place). I spent a lot of time trying to make it work and a lot of time the next day on the phone finding out how it does work.
The problem: User error?
One person I talked with in the Parking Office said that it was “probably user error” because “that is the problem in 24 out of 25 cases.” I don’t generally believe in user error, so I took a deep breath and said that it’s more likely a system-design problem.
After awhile, I found someone who explained about having to hold the card in the meter for 10 to 15 seconds. I identified myself as a user experience designer, and we talked further.
More than user error, I think it was a failure to understand the users and their expectations.
Should a parking meter card need instructions?
He asked if I’d read the brochure that comes with the cards (PDF). This should be so simple that instructions aren’t needed. I don’t think people would read directions, save them or remember what they’d read. I mentioned that, and said that as a typical user, my copy was already in the recycle pile.
We talked about the instructions on the back of the card, too (ALL IN UPPER CASE) That text doesn’t say anything about holding the card in, it didn’t explain the transitions on the display and it didn’t explain when you’re done with a transaction. The brochure did mention holding the card in, but only for signing out.
The gold seal on the left must be the chip. The instructions at right ARE ALL UPPER CASE and don't mention holding the card in.
Problem #5: This system shouldn’t require documentation and what they provide is incomplete.
How can they fix this now that they’re already selling cards?
If the city doesn’t change something to make the system easier to figure out, I’m afraid that it will just fail.
It’s a system with many parts: the card, the display, the insertion method, the information on the meter and the brochure. Plus user expectations. Some parts are easier to change than others, but something has to change.
When I talked with someone in City Hall, I suggested reprinting the cards with complete instructions. He said that the cards came from the vendor. And that they had 10,000 of them. My card has a number in the 400s, so that won’t work.
Next, I suggested printing stickers with better instructions to cover the old text. Again, even if it were a lot of work, at least people would have the instructions with them.
It would help if the sticker on the meter had some instructions. I assume that changing the displays or how the meters work would be too involved, but we didn’t get to those topics.
We talked a little more and I wished him well.
Lesson: Design, test, redesign, test, …
Problem #6: The underlying problem is that the product design process probably didn’t involve any actual users or testing in real situations.
This is a system designed for anyone who parks a car at a meter, day or night, possibly in a hurry. How do you think someone like that reacts to this user experience the first time?
I don’t know who the vendor is, or who designed the system. And I don’t know how they’re going to resolve this problem. I’m pretty sure the program will not succeed without a big change.
I sent what I learned to Eric Moskowitz, the Boston Globe reporter who writes the Starts & Stops column about transportation issues. Maybe he can write a column and help teach people how it works.
It seems pretty clear to me that this whole system was designed the old-fashioned way. Rather than test the system with real users in real situations, they probably talked about it in a conference room and figured it would work out OK. If someone raised the obvious problems, I can imagine someone else saying, “Yeah, but all they have to do is…”
That phrase is the kiss of death for a design. I hope the City of Boston can make this project work because it’s a great idea.
When people see a list, they want to understand its organization. If it isn’t alphabetical, the order should be clear and related to the information and the task. You want your users thinking about the contents of the list, not its presentation.
I’ve seen different types of problems when usability study participants were looking for information in lists. Here are some:
A participant looked at a list of database field names that were ordered by the date they were added. He said, “There’s no order to these fields, so I have to fight this every day.”
Another participant sarcastically referred to the “alphabetized list of our 1000-plus reports”. While it was alphabetical, there were too many items to scroll through. A hierarchical list, or an easy way to remove old reports, or some filtering methods would have helped him.
In a related study, there was a long list that was organized hierarchically, but it was shown in a very small window. It was like looking into a busy room through a keyhole – people could see a little bit of information at a time, but never get a good feeling for the list as a whole.
Looking at an online discussion, a participant in another study remarked that he wanted to see “a threaded-message view”, which is another form of hierarchy. It was as a flat list, where it was hard to tell whether a message was a topic, a response or a response to a response.
Search engines generally show results sorted by relevance. In the early days of search, results included a relevance indicator (e.g., a bar whose length shows the relevance of each result). This isn’t used any more because, as we saw in usability testing, they weren’t meaningful to users.
Another recent study showed the top five items in a couple of categories. Some participants wondered how the list was put together. It could have been editorial choice, most-viewed or most-emailed. Understanding the source of the list would affect how they interpreted it.
Jakob Nielsen, in his Alertbox column wrote about a number of cases where alphabetical order isn’t the best choice, and other situations where alphabetical order was used, but presented in such a way that it wasn’t clear. Good points that illustrate the need to find the right order and present it in the right way.
One thing to take into account is how people will use the list. Understanding how each user type (or persona) approaches the task can help you decide to use alphabetical order, form groups that create a hierarchy, or find an order specific to your use.
Categories are shown by creation time, which doesn't help when you want to apply one next time
Users may not notice that the first few search results are ads. Is that OK?