Watch What Users Do, Don’t Listen to What They Say: Revealed Preferences vs. the Query Effect

Summary: Actions speak louder than words. Observation trumps inquiry. Reality defies expectation. Abandon the treacherous world of self-reporting bias and unlock true user desires through behavioral observation.

Think users know what they want? Ha! This article explains why watching users fumble through your designs is way more enlightening than listening to their well-intentioned fibs. Don’t pay attention to what customers say. This is somewhat of a heresy compared to much management literature, which recommends listening to customers over almost anything else.

As a usability guy, I certainly don’t believe in ignoring the customers. Customer-centric design is the way to success. But to better serve customers and solve their needs, we must watch what they do rather than listen to what they say.

Watching what users do provides superior insights into helping them compared to simply listening to what they say. (Midjourney)

A very simple example comes from YouTube’s recommended videos. You will hear many frequent YouTube users (including myself) say that it’s not very useful when YouTube keeps recommending the same videos that we have seen already. Better to recommend something new to help users discover videos they haven’t seen yet.

However, actual viewing data shows that users click these repeat video recommendations at a much higher rate than YouTube’s recommendations of new videos. The fact that I have already watched a video is a strong signal that I will enjoy it again. (I assume that YouTube’s algorithm takes into account whether the user watched the video to the end and also the genre: at least in my own experience, I’m more likely to rewatch a 3 minute K-Pop music video than a one-hour interview with Marc Andreessen. He’s a smart guy, but once I have heard his comments on some issue, I don’t need to hear them again. But Twice, you can watch more than twice.)

• What users say: don’t show me the same video recommendations again and again.

• What users do: they click those same videos at a high frequency, giving YouTube more revenue than if it were to recommend only new videos.

Recommending the same video repeatedly: the revealed preference of user clicks shows that this works, at least for music videos. (Ideogram — I finally had reason to use its “tiling” feature.)

The Query Effect: Ask, and People Will Make Up an Answer

Ask a question, and you will get an answer. People are polite. (Leonardo)

The “Query Effect” means that the very act of asking a question brings the answers into being: people will make it up, even if they never had an opinion about your question before. Such answers have no grounding in reality. (Ideogram)

Revealed Preference = What People Do

Because of the Query Effect, you’ll often get false answers if you simply ask users. Even if they don’t make up their answer on the spot, what they say may still not reflect reality.

If you ask about a future state, such as whether people would use a certain proposed new feature, people are purely guessing. They can’t predict how much they’ll like a feature until after they have used it. (Designers may have the capability of envisioning how something will work before it’s been built, but this is not a normal human skill.)

If you ask about the past, such as how frequently users use an existing feature or how useful it has been to them, people have to rely on their memory to produce the answer. Human memory is notoriously fallible, so even if answers about the past have higher validity than answers about the future, they are subject to a terrible error rate. Finally, people tend to censor their answers for social acceptability and to please you. If you ask them about a feature that they never liked much, they might still praise it because they think that’s what you want to hear.

Observing actual behavior is the way around all these deficiencies in asking users to report or predict their behavior. Analytics data will tell you exactly what people do on your website. Do they click those repeated video recommendations or not? (Analytics won’t tell you why users perform those actions, only that they happened. You need qualitative user testing to answer the “why” question.)

Finally, in a usability study, you can ask users to think aloud and tell you why they are doing something and what they expect will happen if they take a particular action. A small amount of filtering will still happen, even during a think-aloud session. Still, because people state what they’re thinking simultaneously with having those thoughts, their statements are much more credible.

Discovering why users behave as they do is a treasure chest for UX design. (Midjourney)

Recognizing User Pain Points

Elon Musk (head of the Tesla electric vehicle maker) once met with Tesla owners in Amsterdam. An owner complained that the way to program his car’s charging required specifying when the electricity should begin to flow. However, he wanted to program the time when he was planning to drive off in the morning: that’s the time when he needed the car to be fully charged. Leave it to the computer to figure out when to start charging to achieve this goal. Also, he wanted the car to start preheating sufficiently early to be warm when he left home.

Reportedly, Musk replied that these were reasonable requests, and sure enough, Tesla implemented them shortly afterward.

How to program the charging of a Tesla electric vehicle? System-oriented command: when to turn on the power. User-oriented command: when the owner plans to drive off with a fully-charged and heated car. (Midjourney)

This anecdote illustrates two things:

• The difference between system thinking and the user’s perspective. The engineers actually do have to turn on the electricity for charging to happen, and so they need to know the time when this should happen. But that doesn’t mean that this piece of data should be supplied by the user. At least for the one customer asking the question, it was more natural to think of the time when charging needed to be finished.

• Uncovering user pain points. When you live in Silicon Valley (like Musk and the Tesla designers did at the time), you don’t need to preheat your car in the morning. If you live in Northern Europe, it’s a significant pain to get into a cold car in the morning during winter.

System thinking (how it’s implemented) vs. user thinking (what it achieves and how it feels). Ideogram.

For both of these insights, it’s important to recognize that the insights gathered from that Dutch customer’s comments initially only reflect that one person’s opinion. It could have been the case that a user interface that specified the expected departure time would be confusing for most users — and even for that one user, since he hadn’t actually used this design but simply predicted that it would turn out to be useful for him. Similarly, he might have been the only driver in the world’s cold countries who liked his car to be warm in the morning.

In user research, you should observe more than one user to ensure you’re not overreacting based on a quirky, exceptional user. The additional insight is like moving from a single violin to a full symphony orchestra. (Luckily, most qualitative studies only need to test 5 users, so we get the effect of multiple instruments without the expense of the full orchestra.) Ideogram.

Because of these two caveats, the recommended design process would have been to mock up suggested UI designs that implemented the customer’s requests and then test those designs with additional users.

The bottom line, though, is that even without systematic user research, you often find nuggets of design gold from listening to user comments — especially regarding pain points caused by differences in their circumstances and your circumstances.

In the dance of design, actions whisper truths that words dare not speak. Watching human behavior unveils the hidden rhythms of user preference, guiding us to create experiences that resonate with the unspoken melodies of desire.

What users say is only the tip of the iceberg. What they do is a richer source of design inspiration. (Ideogram)

About the Author

Jakob Nielsen, Ph.D., is a usability pioneer with 41 years experience in UX and the Founder of UX Tigers. He founded the discount usability movement for fast and cheap iterative design, including heuristic evaluation and the 10 usability heuristics. He formulated the eponymous Jakob’s Law of the Internet User Experience. Named “the king of usability” by Internet Magazine, “the guru of Web page usability” by The New York Times, and “the next best thing to a true time machine” by USA Today.

Previously, Dr. Nielsen was a Sun Microsystems Distinguished Engineer and a Member of Research Staff at Bell Communications Research, the branch of Bell Labs owned by the Regional Bell Operating Companies. He is the author of 8 books, including the best-selling Designing Web Usability: The Practice of Simplicity (published in 22 languages), the foundational Usability Engineering (27,123 citations in Google Scholar), and the pioneering Hypertext and Hypermedia (published two years before the Web launched).

Dr. Nielsen holds 79 United States patents, mainly on making the Internet easier to use. He received the Lifetime Achievement Award for Human–Computer Interaction Practice from ACM SIGCHI and was named a “Titan of Human Factors” by the Human Factors and Ergonomics Society.

·       Subscribe to Jakob’s newsletter to get the full text of new articles emailed to you as soon as they are published.

·       Follow Jakob on LinkedIn.

·       Read: article about Jakob Nielsen’s career in UX

·       Watch: Jakob Nielsen’s 41 years in UX (8 min. video)