Biased Questions: How to Identify & Fix Them in Surveys

Biased Questions: How to Identify & Fix Them in Surveys

Picture of Adam Ramshaw
Adam Ramshaw
Adam Ramshaw has been helping companies to improve their Net Promoter® and Customer Feedback systems for more than 15 years. He is on a mission to stamp out ineffective processes and bad surveys.
Download the Customer Feedback whitepaper

Designing a customer feedback survey in way that provides and concise data you can use in you business is not always simple.

Using Biased Questions is a common mistake among survey designers and this post outlines what they are, how to identify them and how to fix them.

Download the Customer Feedback whitepaper

What are Biased questions

Biased questions are ones that, intentionally or unintentionally, may influence the way the respondent will answer. Take this example:

example of biased question
Example of an Intentionally Biased Survey Question

By prefacing the question (“Overall how satisfied…”) with the pejorative statement “… is continually striving to achieve 5-STAR customer satisfaction…” the company has subtly altered the respondents’ feelings towards their experience.

Question bias can also be caused through the options the respondent is offered.

Here’s a perfectly balanced question turned into a biased question through the use of limited response options:

Balanced Survey Question Turned into Biased Question Through Response Options

Of course no company wanting to really understand customer loyalty would write the question like this but it demonstrates what can happen.

Why Biased Questions are a Problem in Surveys

The main reasons such questions are a problem in surveys is they distort the data collected.

Most of the time surveys and questionnaires are trying to accurately understand a situation or perception of a customer or respondent. By using biased questions the organisation writing the survey is distorting the data collected making it less reliable and accurate.

Interestingly not only can the data be distorted it can also be distorted in unknown ways.

Take the example question above.

example of biased question

Typically this style of distortion is done by organisation trying to maximise some feedback score but it’s a mistake because the respondent could go either way:

Reduce the score they were going to give: Some people will read the question and their expectations will rise, depressing the score they give: “well they were going for 5-star and that was only 3 star.”

Increase the score they were going to give: other people will subtly increase their score because they don’t want to disappoint the company by marking them down.

The problem is that you will never really know the effect.

Why Do Biased Questions Occur?

There are two main reasons biased questions turn up in surveys:

Unintentionally

Often it can slip into survey questions unintentionally, especially where the question writer has a strong affiliation with the organisation or subject of the survey and want it to be seen in a positive light, or not in a negative way.

Intentionally

However, just as often questions are written in a biased manner to attempt to influence either the response or the respondent. so the results of the survey analysis support one or other of the writer’s agendas. This is termed Push Polling.

What are not Biased Questions

There are plenty of poor ways to word survey questions that are not necessarily biased including the following types:

Just because a question is not well written does not mean it is biased.

Biased Question Examples

Here are a few more biased question examples so you can get the feel for them.

Loaded Questions

Do you think there’s a danger in giving young people guns and teaching them how to kill? 
Example Biased Question: Loaded Question

This question comes from the famous BBC comedy “Yes, Minister”. There are several pejorative words in this question: kill, danger, guns. Which biases people to answer in the affirmative – which was the intention.

What did you dislike about the service you received?
Example Biased Question: Loaded Question

Unless this question occurs AFTER someone has already expressed dislike of the service, e.g. they have answered the Net Promoter question as a detractor, you’re inviting them to find fault where there might have been none.

Will you be voting for this disrespectable political party at the next election?
Example Biased Question: Loaded Question

Adding “disrespectable “ to the question clearly biases the response.  

Language That Excludes Particular Groups

For example the use of gendered language in survey questions. Surveys that use language such as “he or she” or “him or her” to refer to participants may exclude individuals who do not identify as male or female, such as those who are non-binary or gender non-conforming. This can lead to a lack of representation of these groups in the survey results, potentially leading to inaccurate conclusions about the experiences or perspectives of those groups.

Other examples of this issue include racial, political or other social groupings.

How to Spot a Biased Question

Unlike Double Barrelled Questions, which are easy to spot, it can sometimes be difficult to spot a biased question in a survey because the form bias can occur in so many different ways. 

In general if you see emotive, pejorative or assumptive words of any type or limited response options in a survey question, there is a chance you have created a biased question.

Even if you only suspect the question to be a problem you should attempt to correct it.

How to fix Biased Questions

Fixing biased survey questions can sometimes be a little tricky.

The best approach is to simply re-write the question in a way that is as neutral as possible and eliminate emotional, contentious and pejorative wording.

For example to fix this biased question:

What did you dislike about the service you received?
Biased Question Example

You could rephrase it as:

Tell us about the service you received.
Corrected Biased Question

Biased Question FAQ

What are Leading Questions

Leading questions are a type of biased question intended to intentionally influences the respondent. For example this famous question from the BBC comedy “Yes, Minister”. 

“Do you think there’s a danger in giving young people guns and teaching them how to kill? “

Download the Customer Feedback whitepaper