Every UX study is designed, run and interpreted by people, and people bring assumptions with them. These assumptions, known as biases, can quietly shape who we recruit, what we ask, how participants respond and what we conclude. This article covers five common biases in UX research, with examples from our own studies at the UVA Library and how our team works to reduce each one. It is meant as a reference for anyone who runs or takes part in user research, surveys or usability testing.
UX research at the UVA Library aims to understand how students, faculty and staff use our websites, services and spaces, so we can improve them. Design decisions built on skewed data can send us in the wrong direction. Bias is different from random error. Random error is noise that tends to even out as more data is collected. Bias is a systematic lean in one direction, so adding more participants does not necessarily fix it. Bias can't be removed entirely, but knowing where it comes from makes it easier to minimize.
Five Common Biases
1. Sampling Bias (Who We Study)
Sampling bias occurs when participants do not represent the wider user population. It is often the result of how we recruit rather than a mistake by any one person: volunteers may be more motivated or more comfortable with technology than the average user, and those who never respond may have very different experiences. To guard against this, we recruit across roles and use more than one method.
The Virgo catalog study combined interviews and contextual inquiry, two surveys, A/B testing, and multivariate testing with member checking, and we reported findings separately for students, faculty and staff so that no single group’s preferences stood in for everyone’s. Smaller studies, like the five editors in our publishing platform study, are treated as exploratory, and we always report the number of participants.
2. Leading Question Bias (How We Ask)
Leading questions nudge participants toward a particular answer through wording, assumptions or tone. Even small choices, like adding “easy” or “confusing” to a question, can shift responses. “How easy was it to find the room reservation button?” suggests the button was easy to find, while “How would you describe the experience of finding the room reservation button?” leaves room for any answer.
In the Virgo Item Details survey, we compared the new prototype against a cleaned-up version of the current page, not the old page as it was, so that respondents were reacting to the design change itself. Wording used in the survey questions were reviewed to minimize leading language and maintain a neutral tone.
3. Social Desirability Bias (How Participants Respond)
Participants often give answers they think are acceptable or expected rather than fully honest ones. In usability testing, this can look like blaming themselves (“I’m just bad with websites”) or downplaying frustration to be polite to the researcher. We try to make it easy to be candid.
Our publishing platform sessions were framed as understanding editors’ experiences, not assessing their capabilities, and our interview and A/B sessions used think-aloud methods, which let us compare what participants said with what they did. A hesitant pause or a wrong click can tell us something that a polite answer does not. It also helped that participants felt free to push back: in the Virgo A/B sessions, power users told us that splitting information into sections made their research harder because of the extra scrolling.
4. Confirmation Bias (How We Interpret)
Confirmation bias is the tendency to notice and remember evidence that supports what we already believe, and to overlook evidence that does not. Researchers, designers and stakeholders can all fall into it, especially when a design has taken a lot of effort to build.
In the Virgo A/B testing phase, we paid attention to the problems as much as the preferences: some participants misunderstood the new Availability section, and multiple “Request” buttons created a confusing flow. Those findings led to design changes and a second round of testing comparing a “Show more details” button with a toggle. We also avoided letting a single data source decide. The navigation buttons and link function had low usage in Matomo, but we kept them and renamed the latter “Permalink.”
5. False-Consensus Effect (Assuming Others Think Like Us)
The false-consensus effect is the assumption that other people share our knowledge, habits and views. Library staff work with Library terminology daily, so labels and workflows can seem obvious to those at the Library but not to first-time users. This is a main reason we test with people beyond the Library.
In the Virgo study, novice users found the original Item Details page overwhelming, while a library liaison preferred seeing more information on one screen. Rather than designing for either view alone, we made the most important information the default and added a “Show more details” button for those who want more. Our A/B testing study of the Library website showed a similar lesson: “Search, borrow, request” seemed clear to staff, but one participant read it as being only about books, so it became “Research: Search, borrow, request.”
Key Takeaways
Bias is a normal part of any research involving people, and the goal is to recognize and reduce it rather than pretend it isn't there. As the examples from our own studies show, bias can enter at every stage: through who we recruit, how we ask questions, how participants respond, and how we interpret the results. Here is what helps us keep it in check:
• Diverse recruitment: We recruit across students, faculty and staff, and use several methods, so no single group speaks for everyone.
• Neutral questions: We compare designs fairly and word questions so they don't point toward an answer.
• Quiet observation: We watch what participants do as well as what they say, using think-aloud sessions.
• Attention to what doesn't work: We report failed tasks and tradeoffs alongside the wins.
• Openness about limitations: We share sample sizes and what each study can and can't tell us, so others can use our findings appropriately.
No study is free of bias, but a team that knows where it comes from can design research that better reflects the people we serve.