Years ago, a business acquaintance told me about a customer survey he had recently conducted. He was enthusiastic about the results and was preparing to launch several initiatives based on what he had learned.
During our conversation, he asked if I would take a look at his survey.
What I found concerned me. Many of the questions were poorly constructed. Several rating scales were unbalanced, and some of the response choices didn’t even fit the questions being asked.
I explained why these problems could make the findings unreliable and cautioned him about basing significant business decisions on the results.
He wasn’t persuaded. He believed in his numbers.
I never learned whether he went ahead with his plans or how things turned out. But the experience has stayed with me because it illustrates one of the greatest dangers in survey research.
A survey can produce results that look perfectly reasonable, even convincing, when the information behind them is fundamentally flawed.
And when those results become the basis for important business decisions, the consequences can be costly.
Over more than 40 years working in survey research, I’ve seen organizations make a variety of mistakes when gathering and interpreting information. Most weren’t the result of carelessness. They generally occurred because people didn’t understand some of the principles that make survey results trustworthy.
Here are ten mistakes worth recognizing — and avoiding.
1. Beginning Without a Clearly Defined Purpose
One of the most common mistakes happens before the first question is written.
Someone decides the organization needs a survey, and the discussion immediately turns to what questions should be asked.
But a survey should begin with a different question: What do we need to learn, and what decisions will that information help us make?
Without a clear purpose, questionnaires often become collections of questions that are interesting to ask but don’t necessarily produce useful information.
A well-defined purpose provides direction for everything that follows, including whom to survey, what to ask, and how to analyze the results.
The lesson: Before developing a questionnaire, be certain everyone understands what the survey is intended to accomplish.
2. Excluding Important Groups from the Survey
I once worked with a company that had an unusual approach to selecting customers for a survey.
Management wanted to exclude its least satisfied customers because it believed they would provide overly negative responses. It also wanted to exclude its most satisfied customers because their answers would be predictably positive.
The solution, management believed, was to survey only the customers in the middle.
I explained that by deliberately excluding customers at both ends of the satisfaction spectrum, the company would be creating a distorted picture of its customer base. Those customers might have some of the most valuable information about what the company was doing right and where it needed improvement.
Fortunately, after discussing the issue, the company agreed to change its sampling approach.
A less obvious version of this problem occurs when organizations exclude customers simply because they’re difficult to reach. Missing telephone numbers, outdated email addresses, or incomplete contact lists can leave certain groups out of the survey altogether.
The important question is whether those excluded customers might differ meaningfully from the customers who are included.
The lesson: A survey must give all relevant groups an appropriate opportunity to be represented. Convenience shouldn’t determine whose opinions count.
3. Believing a Larger Sample Automatically Means Better Data
There’s a common assumption that the more people you survey, the more reliable your results will be.
Sample size certainly matters. Too few completed interviews can limit the precision of estimates and make it difficult to examine important subgroups.
But a very large sample cannot correct a fundamentally unrepresentative one.
Imagine surveying 5,000 customers who regularly purchase from your company while excluding those who have stopped doing business with you. You may obtain highly consistent results, but they won’t necessarily tell you why customers are leaving.
Organizations can also make the opposite mistake: selecting too small an initial sample without allowing for a lower-than-expected response rate. The number of people invited to participate is not the same as the number who ultimately respond.
The lesson: Determine whom you need to represent, how many completed responses you need, and how you’ll obtain them. Bigger isn’t automatically better.
4. Trying to Ask Everything Imaginable
Once an organization commits to conducting a survey, it’s tempting to ask every department what information it would like to collect.
Sales wants questions about pricing. Marketing wants to explore brand awareness. Operations wants feedback on service delivery. Management wants satisfaction ratings, loyalty measures, and suggestions for improvement.
Before long, a focused survey becomes an oversized questionnaire attempting to address everyone’s interests.
Long questionnaires can discourage participation, increase respondent fatigue, and reduce the attention people give to later questions.
Even worse, they can bury the information the survey was originally designed to obtain.
The lesson: Every question should earn its place in the questionnaire by contributing to the survey’s defined purpose.
5. Asking Questions That Influence or Confuse Respondents
Question wording can have a powerful effect on the answers people provide.
Consider this question:
“How satisfied are you with our fast and responsive customer service?”
The question assumes the service is fast and responsive before the respondent has had an opportunity to evaluate it.
Or consider:
“How satisfied are you with our product quality and customer support?”
How should someone answer if the product is excellent but the support is terrible?
These are examples of leading and double-barreled questions. Others use ambiguous language, unfamiliar terminology, or wording that different respondents may interpret differently.
The difficulty is that the resulting numbers may appear perfectly normal. A spreadsheet won’t warn you that respondents misunderstood what they were being asked.
The lesson: Questions should be neutral, clear, and focused on one idea at a time.
6. Using Unbalanced Rating Scales
Rating scales are common in customer and employee surveys, but their construction deserves careful attention.
Consider a satisfaction question offering these response choices:
- Extremely satisfied
- Very satisfied
- Somewhat satisfied
- Neither satisfied nor dissatisfied
- Dissatisfied
The scale provides three positive response choices but only one negative choice.
That imbalance can influence how respondents express their opinions and may make results appear more favorable than they would with a properly balanced scale.
A better approach would provide comparable response opportunities on both sides of a neutral midpoint, when a neutral option is appropriate.
Consistency also matters. Changing the direction or meaning of rating scales throughout a questionnaire can confuse respondents and introduce errors.
The lesson: Rating scales should give respondents a balanced, understandable way to express their opinions without steering them toward a particular answer.
7. Providing Incomplete or Inappropriate Response Choices
Even a well-worded question can produce poor information when its response choices don’t fit the people answering it.
Suppose a company asks:
“How often do you contact our customer support department?”
And offers:
- Once a week
- Two or three times a month
- Once a month
- Several times a year
What’s missing?
The question provides no choice for customers who have never contacted support. It also creates potential confusion for someone whose experience doesn’t fit neatly into the available categories.
When response choices are incomplete, overlapping, or inappropriate, respondents may choose an answer that doesn’t accurately reflect their situation — or abandon the question altogether.
Where appropriate, options such as “Never,” “Not applicable,” or “Don’t know” can help prevent respondents from being forced into misleading answers.
The lesson: Response options should be comprehensive, clearly distinguished, and appropriate to the question being asked.
8. Looking Only at Overall Results
One of the most tempting things about survey results is a strong overall score.
Imagine a company conducts a customer satisfaction survey and discovers that 85% of its customers are satisfied.
That’s encouraging. Management might reasonably conclude that things are going well.
But what happens when the results are examined more closely?
Perhaps satisfaction is 95% among long-established customers but only 60% among newer customers. Or perhaps one geographic region is performing substantially worse than the others.
The overall result isn’t necessarily wrong. It’s simply incomplete.
In my experience, important insights often emerge when survey results are examined by meaningful subgroups — customer type, location, tenure, product usage, or other characteristics relevant to the business.
Of course, subgroup comparisons require adequate sample sizes and careful interpretation. Small differences don’t automatically represent meaningful differences.
The lesson: Overall results tell you how the population looks on average. Subgroup analysis can help reveal where important differences exist.
9. Interpreting Results Without Adequate Context
Survey results don’t always speak for themselves. Sometimes a number that appears encouraging — or disappointing — takes on an entirely different meaning when viewed in context.
Suppose a company conducts an employee survey and finds that 75% of employees are satisfied with their jobs.
Is that a good result?
It depends.
If satisfaction was 60% a year ago, the company may have made considerable progress. But if it was 90% last year, the same 75% represents a potentially troubling decline.
Even when previous survey results aren’t available, other information can help put the findings into perspective. Customer complaints, employee turnover, service performance, and other business measures may provide valuable context for interpreting what the survey reveals.
The important point is that a survey result shouldn’t be evaluated in isolation. Understanding what a number means often requires understanding what’s happening around it.
The lesson: Don’t judge a survey finding by the number alone. Consider trends, relevant business information, and other evidence before deciding what the results mean.
10. Confusing Correlation with Causation
This may be one of the most consequential mistakes organizations make when interpreting survey results.
Suppose a survey finds that customers who report difficulty using a product are also less likely to renew their contracts.
It’s tempting to conclude that difficulty using the product is causing customers to leave.
That may be true. But the survey has established an association, not necessarily a cause-and-effect relationship.
Other factors may be involved. Perhaps customers who use the product infrequently are less familiar with its features. Perhaps they perceive less value in continuing the business relationship. Or perhaps several factors are working together.
Understanding these relationships may require additional analysis, follow-up research, or other evidence.
Correlation can provide valuable clues about where to investigate. What it cannot do, by itself, is establish why something is happening.
The lesson: Treat relationships in survey data as evidence to investigate, not automatic proof of cause and effect.
The Real Danger Is False Confidence
What makes these mistakes especially troublesome is that they aren’t always obvious.
A questionnaire can look professional. A survey can generate hundreds or thousands of completed responses. The findings can be presented in attractive charts with percentages calculated to two decimal points.
And the results can still be misleading.
That brings me back to the acquaintance who was preparing to launch business initiatives based on his customer survey.
He wasn’t ignoring information. He was trying to use information to make better decisions. The problem was that he had confidence in evidence that might not justify it.
No information leaves you uncertain. Bad information can make you confidently wrong.
The good news is that producing reliable survey information doesn’t require every organization to employ a full-time survey researcher.
But it does require an understanding of some fundamental principles: how to define a survey’s purpose, select an appropriate sample, construct a sound questionnaire, and interpret the findings responsibly.
Having even one or two employees who understand those principles can make a meaningful difference when an organization needs answers to important questions.
Surveys can be enormously valuable business tools. When properly designed and conducted, they help organizations see beyond assumptions, understand the experiences of customers and employees, and make decisions with greater confidence.
The objective isn’t simply to collect answers.
It’s to collect answers you can trust.