Let’s face it — conducting surveys is an essential part of any business if you really want to understand your customers.
This feedback brings a wealth of knowledge — from awareness of your customers’ needs, their concerns, and overall benchmarks for improving products and services.
But, if you’re not careful in how you craft survey questions, you can introduce response bias, which harms, not helps, your organization.
Today, we’re going to cover the top 11 types of survey questions that introduce bias including leading questions, double-barreled questions, and double negatives.
- First…What is a biased survey?
- Survey questions: Do’s and Dont’s
- 5 Types of Biased Survey Questions
- How to avoid biased survey questions and results
- Survey Structure: Things to remember
First…What is a biased survey?
There are dozens of ways you can introduce bias into your survey.
According to The AP Statistics Tutorial, “a poor measurement process can lead to bias. In survey research, the measurement process includes the environment in which the survey is conducted, the way that questions are asked, and the state of the survey respondent.”
Your first task is to be able to distinguish which types of survey questions are biased.
Keep in mind that some respondents may become unresponsive to biased survey questions which, in turn, could negatively impact the outcome of your survey.
So how do you know if survey questions are biased?
Generally, a biased survey has a set of questions which may influence the response of participants — bringing inaccurate results.
The data you gather from survey questions affects the way you do business — you wouldn’t want to steer your company in the wrong direction with inaccurate data you gathered from biased survey questions. Arm yourself with a reliable customer survey software to conduct surveys and analyze their data.
Survey questions: Do’s and Dont’s
When formulating survey questions, keep the following in mind:
- Research objectives should be met
- Questions should be simple, clear, and straightforward
- Questions should be structured properly
- Questions should be progressive but neutral
- Options should include all valid responses
5 Types of Biased Survey Questions
1. Absolute Questions
Absolute questions are those that are answerable only by a “yes” or “no”. This leaves very little room for a respondent to expand on their feedback.
Words like “always,” “all,” “every,” and “ever” are commonly used for this type of survey question.
An example of an absolute question would be: do you always wash your hair at night? The only two options would be yes or no.
This single-mindedness makes absolute questions rigid and therefore unreliable when it comes to response accuracy.
The better way to ask this question is to break it into two survey questions, with various answer options:
How often do you typically wash your hair?
- Everyday
- Every Other Day
- Twice A Week
- Once A Week
- Other
What time of the day do you usually wash your hair?
- In the morning
- Mid-day
- At night
- Other
2. Double-Barreled Questions
Double-barreled questions are questions that tries to “kill two birds with one stone”. A double-barreled question actually asks a respondent for two answers to one question.
Here is an example of a double-barreled question: How satisfied are you with your salary and benefits? (Very Satisfied/Satisfied/Fairly Satisfied/Not Satisfied)
Because a respondent is forced to answer the question without being able to specify whether their satisfaction is based on their salary or on their benefits, their response becomes unclear.
In order for a survey question to be effective, it needs to address one measurable response at a time.
The best way to ask these questions is to make it more specific by splitting it into two separate survey questions: How satisfied are you with your salary? (Very Satisfied/Satisfied/Fairly Satisfied/Not Satisfied) How satisfied are you with your benefits? (Very Satisfied/Satisfied/Fairly Satisfied/Not Satisfied)
3. Leading Questions
A leading question is defined as a question that prompts or encourages the desired answer.
Conducting a survey requires honest answers from a respondent — regardless if they are affirmative or not.
Asking a leading question will greatly affect the accuracy of your results — defeating the purpose of running the survey in the first place.
Leading questions make use of adjectives and unnecessary additions to influence the reader into stating the desired response.
A leading question can come in two forms:
- A question that does not accommodate all valid responses
- A question which leads a respondent to state a desired opinion
An example of a leading question would be:
What do you think makes our product popular among young teens?
This elicits positive feelings about the subject and influences the way a respondent answers the question.
Make the question neutral by being straightforward and removing unnecessary words such as “popular.” Instead, phrase the question: As a young teen, what do you think of our product?
4. Loaded Questions
It can be a little difficult to distinguish a leading question from a loaded question.
A leading question leads a respondent to give an affirmative response; a loaded question may be more subtle in its approach by using assumptions and weighted words to incite a positive response.
A loaded question often provokes an emotional rather than a rational response from a respondent because of weighted words.
By making use of words that make a negative concept sound positive (or vice versa), the question manipulates a respondent’s thinking and persuades him to give an answer based on an emotional response.
Weighted words such as “never” and “always” can also turn a loaded question into an absolute one.
More often than not, respondents drop out of the survey or give unclear answers to loaded questions.
An example of a loaded question would be: where do you like to hang out on Saturday nights?
This question makes the generalized assumption that all respondents go out to socialize on Saturday nights.
What if some respondents actually have to work Saturday nights? What if a fourth of them choose to stay home and do laundry?
The question then excludes a number of respondents because they can’t relate to it.
The better and more neutral way to ask this question would be: what do you typically do on a Saturday night?
5. Multiple Answer Questions
When asking any type of multiple choice question, make sure that the answer options are clear-cut and do not overlap so that the respondent can pick one definitive answer.
An example of a bad multiple answer question would be: how many hours a week do you work?
- 10-20
- 20-30
- 30-40
- 40 and above
For a respondent who works 20, 30, or 40 hours a week, these options can be confusing. Avoid overlapping your choices and break them down into more defined terms.
Reword this question’s answers to be:
- 10-20
- 21-30
- 31-40
- 41 and above
How to avoid biased survey questions and results
To create an effective survey, it’s not only the questions that matter — the answer options matter just as much.
Here are some guidelines to help you through the process of creating survey answers:
- All possible answers should be included in your options. This is one of the essential do’s in creating an effective survey.
If you are unable to identify all possible options, the next best thing is to add “Other” as an option and allowing the respondent to input their own answer.
- “Prefer Not To Answer” should be included in your answer options, to avoid survey drop-outs.
For questions that are sensitive in nature, it is advisable to add this option so as not to make the respondent uncomfortable about answering them.
For example, “Prefer Not To Answer” might be used in gender or ethnicity questions.
- Accurate scales must be used.
For questions that require an answer based on a scale (e.g. Excellent/Very Good/Good/ Fair/Poor/Needs Improvement), your range should cover all possible options.
Survey Structure: Things to remember
A survey should have a brief but clear title so it’s easily understood by a respondent.
Your instructions on how to answer the survey should be simple and straightforward as well to avoid confusion.
The key is to make completing the survey as easy and as painless as possible.
Where possible, provide incentives or rewards to the respondent for completing the survey to thank them for taking time to accomplish it.
A good survey should have a sound structure — questions are asked in a logical and progressive order.
Generally, a funnel approach makes the most sense. Questions that are related should appear in succession.
The following order of questions may help you in structuring your survey questions more effectively.
- General questions should be asked at the beginning of the survey. This is so you can warm up to the respondent and make them comfortable about proceeding with the rest of the questions.
- Transition to more specific but non-personal questions. This is probably the most important part of the survey since more specific and detailed answers will be asked of the respondent.
- Follow through with easy-to-answer survey questions. This gives the respondent a chance to breathe in between questions.
- Questions that pertain to demographics or that are more personal in nature should be placed last. Always remember that the more anonymous a survey is, the more comfortable the respondent will be.
Now that you have all these guidelines in mind, it’s time to start formulating your next survey to feature non-biased survey questions.
It isn’t always easy to recognize bias. It’s especially harder for the individual creating the survey as compared to the one answering it. If you can, work with a professional test design team. If you can’t, have a few team members across the organization review the survey for biased questions and responses.
A biased survey will not only waste your respondents’ time and energy, it could drive your organization in the wrong direction.
