Several years ago, an acquaintance of mine decided to conduct his own survey. He had some business decisions to make, and he believed a survey would provide the information he needed.
After completing the survey, he showed me what he had done. As I looked through his questions, I noticed several that concerned me. Some were worded in ways that could influence how respondents answered. Others left room for different interpretations of what was being asked.
I shared my concerns with him, but he was comfortable with what he had done. He had collected his data, reviewed the findings, and felt confident that he had the information he needed to move forward.
And that’s what he did.
I never learned how things turned out. Perhaps his decisions were good ones. Perhaps the problems I noticed had little effect on his conclusions. Or perhaps the results led him in a direction he might not otherwise have taken.
I simply don’t know.
But I remember the experience because it illustrates something I’ve come to appreciate over more than 40 years in survey research.
Sometimes the greatest danger isn’t having too little information. It’s having information you believe is reliable when it may not be.
When Having Answers Creates Confidence
Most business leaders understand the discomfort of making decisions without adequate information.
Something isn’t going as expected. Customer retention is declining. Employees seem less engaged. Sales are falling short of projections. Management has several possible explanations, but no one knows for certain what’s happening.
A survey seems like a logical next step.
And often it is.
Properly designed and conducted, a survey can help an organization understand what customers, employees, or other important groups are experiencing, thinking, and doing. It can uncover problems that management hasn’t recognized and challenge assumptions that have gone unquestioned.
But there’s another side to this.
Once the survey is completed, the organization has answers. There are percentages, comparisons, and perhaps a few unexpected findings. Managers discuss the results, develop conclusions, and begin deciding what to do.
The uncertainty that existed before the survey begins to disappear.
That’s exactly what we want to happen when the research has been done well.
But what if it hasn’t?
What if the questions didn’t measure what the organization needed to understand? What if important groups of people were missing from the survey? What if differences in the data were given more meaning than they deserved?
The organization may feel more confident without actually being better informed.
And that is where problems can begin.
The Appearance of Credibility
Early in my research career, I sometimes thought that a thicker final report would make a project appear more substantial.
This was back when reports were printed and bound, and clients received physical copies of our findings.
I would include the complete set of cross-tabulations, adding dozens of pages to the report. There was something reassuring about handing a client a substantial document filled with tables and numbers. It looked like a great deal of work had gone into the project.
And, of course, a great deal of work often had.
But looking back, I’ve come to appreciate that the thickness of the report had very little to do with the usefulness of the research.
A 150-page report isn’t necessarily more valuable than a 30-page report. In fact, the shorter report may be far more useful if it clearly identifies the findings that matter and explains what they mean for the decisions the organization needs to make.
Today, the presentation may be different, but the underlying issue hasn’t changed.
We have sophisticated software that can produce attractive charts, colorful dashboards, detailed tables, and statistics calculated to the second decimal place.
All of these can be useful.
But they can also create an impression of precision and authority that the underlying research hasn’t earned.
Consider a report showing that 63.47% of customers are satisfied with a particular service.
That number looks precise. It may even suggest a level of scientific rigor.
But what does it actually tell us?
Were the right customers surveyed? Was the question worded clearly? Did respondents interpret the question consistently? Was the sample sufficiently large and representative for the conclusions being drawn?
Until we understand those things, the two decimal places tell us very little about the reliability of the finding.
The appearance of precision is not the same as the presence of accuracy.
And an impressive-looking report can still contain information that leads us in the wrong direction.
When the Results Confirm What We Already Believe
There’s another reason questionable survey findings can be particularly persuasive.
Sometimes they tell us exactly what we expected to hear.
Imagine a company whose leadership believes customers are leaving because its prices are too high.
The company conducts a survey, and the results indicate that price is an important concern.
Management sees confirmation of its original belief. The survey has apparently validated what everyone suspected.
The discussion quickly turns to whether prices should be reduced.
But suppose the survey asked customers to identify the factors influencing their purchasing decisions and presented price prominently among a limited set of choices.
What if it didn’t adequately explore service responsiveness, product reliability, communication, or other factors that might be contributing to customer dissatisfaction?
The survey may have identified a legitimate concern without identifying the most important cause of the problem.
Price may indeed matter. But the research may not have established that lowering prices will improve retention.
This is an illustrative example, but the underlying risk is very real.
When survey findings support an existing belief, people may be less inclined to question how those findings were obtained.
The survey stops being a tool for examining assumptions and becomes a means of reinforcing them.
I have a saying that captures this concern:
The most expensive assumption is the one everybody in the room agrees with.
A survey should help us examine those assumptions, not simply give them a more convincing appearance.
Incomplete Information Isn’t Necessarily Bad Information
I recall another project in which the research never reached its intended conclusion.
For various reasons, the project fizzled out before all the planned work was completed. The organization decided to move forward with the information that had been collected, making use of whatever findings it believed could be salvaged.
I don’t know whether those decisions ultimately proved successful.
And I wouldn’t automatically criticize the organization for proceeding.
Business decisions can’t always wait until every question has been answered. Sometimes deadlines, financial considerations, or changing circumstances require leaders to act with less information than they would prefer.
There’s nothing inherently wrong with that.
The important issue is whether decision-makers understand the limitations of the evidence they’re using.
If a survey has reached only a portion of its intended audience, certain findings may still be useful. But those findings may not support conclusions about everyone the organization originally intended to study.
If some questions were completed but others were not, the available data might answer certain questions while leaving others unresolved.
The key is knowing the difference.
A manager who says, “We have some information, but there are important things we still don’t know,” is in a very different position from one who says, “The survey has given us our answer.”
Both may make the same decision.
But the first understands the uncertainty that remains. The second may not.
And understanding uncertainty is an important part of making a sound business decision.
The Questions Business Leaders Should Ask
None of this means that executives and managers need to become survey methodologists.
They don’t need to master sampling theory, statistical testing, or questionnaire construction to benefit from survey research.
But they should be comfortable asking a few basic questions before placing too much confidence in the findings.
What were we trying to learn?
A survey should begin with a clearly defined purpose. If the organization cannot explain what information it needed and what decisions that information would support, it becomes difficult to judge whether the findings actually answered the right questions.
Did we hear from the right people?
A large number of completed questionnaires doesn’t necessarily mean the survey represents the population the organization wants to understand. Who participated, who didn’t, and how respondents were selected can all influence what the findings mean.
Did our questions give respondents a fair opportunity to tell us what they think?
Poorly worded questions can introduce confusion or encourage particular answers. Even subtle wording choices can influence how people respond.
Are we interpreting the findings appropriately?
A difference between two groups doesn’t always mean the difference is meaningful. A relationship between two variables doesn’t establish that one caused the other. And an overall average can conceal important differences among customers, employees, or other groups.
What can we reasonably conclude, and what remains uncertain?
This may be the most important question of all.
Good research doesn’t always produce a definitive answer. Sometimes it narrows the possibilities, identifies the most likely explanations, or reveals where additional information is needed.
Those are valuable outcomes, too.
The goal isn’t to eliminate every uncertainty. It’s to reduce uncertainty enough to make a better-informed decision.
Confidence Should Be Earned
I don’t want to leave the impression that survey findings should automatically be viewed with suspicion.
Quite the opposite.
I’ve spent my professional life helping organizations use survey research to better understand their customers, employees, markets, and business challenges. I’ve seen how valuable reliable information can be when it challenges conventional thinking, uncovers an overlooked problem, or helps leaders make a difficult decision.
That’s why I believe it’s so important to get the research right.
When a survey is properly designed, conducted, and interpreted, decision-makers have good reason to place confidence in its findings.
But that confidence should come from the quality of the research, not from the number of completed questionnaires, the appearance of the final report, or how closely the findings match what management already believes.
Returning to my acquaintance and his survey, I sometimes wonder how things worked out for him.
I hope his decisions were successful. It’s entirely possible they were.
But I also wonder whether he might have reached different conclusions if the questions had been written differently or if he’d taken another look at the evidence before moving forward.
I’ll never know.
What I do know is that having survey results and having reliable evidence are not always the same thing.
And when important business decisions are at stake, that distinction matters.
The most dangerous survey isn’t necessarily the one with the most mistakes. It may be the one whose mistakes go unnoticed because everyone believes the results.