Over the course of my career, I’ve encountered many business leaders who were confident they already knew what was happening in their organizations.
Some told me they had all the information they needed to understand how their businesses were operating. Others believed they knew their customers well enough that conducting surveys wasn’t necessary. And more than a few assured me that if their customers had a problem, they would certainly let them know.
I never questioned the value of their experience or their familiarity with their businesses. Those are important assets. But I often wondered whether the information they were relying on could answer all the questions they might eventually face.
It’s an important distinction, because having information isn’t the same as having the evidence needed to make a sound business decision.
And sometimes, the greatest information gap is the one an organization doesn’t realize exists.
There’s Certainly No Shortage of Data
Most organizations today have more information available to them than ever before.
Sales reports track revenue by product, customer, territory, and time period. Financial statements document expenses and profitability. Customer management systems record purchases, interactions, and service histories. Operational reports track productivity, delivery times, quality, and other measures of performance.
For many business leaders, reviewing these reports is a regular part of managing their organizations. They can see which products are selling, whether costs are increasing, how individual departments are performing, and whether the business is meeting its financial objectives.
All of this information has value. In fact, it would be difficult to manage a business effectively without it.
But there’s something worth recognizing about most internal reporting systems.
They are generally designed to tell us what is happening but may tell us very little about why it is happening.
Consider a company whose customer retention rate has declined over the past year.
Its reports may show exactly how many customers were lost, which products or services they purchased, how long they had been customers, and how much revenue their departure represents.
That’s useful information. It establishes the extent of the problem and may even help identify where the losses are concentrated.
But does it tell management why those customers left?
Perhaps customers were dissatisfied with service. Perhaps competitors offered better pricing. Perhaps customers no longer needed the product. Or perhaps the company changed something that made doing business with it more difficult.
Internal reports may offer clues, but unless the organization has collected information that addresses those possibilities, management may still be left guessing.
The company has plenty of data. What it may not have is the evidence needed to understand the problem.
When Experience Becomes a Substitute for Evidence
There’s another source of information business leaders rely on, and it’s one that doesn’t always appear in a report.
It’s their own experience.
An executive who has spent 20 or 30 years in an industry has undoubtedly learned a great deal about customers, competitors, employees, and the marketplace. That knowledge can be invaluable, particularly when making decisions that require judgment.
But experience can also create a level of confidence that discourages further inquiry.
I’ve had business leaders tell me, in effect, that they didn’t need to survey their customers because they already knew what those customers thought.
And I understood their reasoning. They had built relationships, spoken with customers, resolved problems, and watched their businesses grow. Why wouldn’t they feel confident in their understanding?
The question, however, isn’t whether their experience has value.
It’s whether that experience provides a complete and accurate picture.
A company president might regularly speak with several major customers and believe those conversations provide a reliable understanding of customer satisfaction. But what about the smaller customers who seldom have direct contact with management? What about customers who have reduced their purchases? Or those who are considering taking their business elsewhere?
The customers who speak with leadership may have experiences that differ considerably from those who don’t.
And what management hears most frequently isn’t necessarily what customers are experiencing most widely.
That doesn’t make experience unreliable. It means experience, like any other source of information, has limitations.
The challenge is recognizing those limitations before relying on experience alone to make an important decision.
If Customers Have a Problem, Won’t They Tell Us?
Of all the reasons I’ve heard for not conducting customer research, one of the most interesting is the belief that dissatisfied customers will naturally make their concerns known.
On the surface, it seems reasonable.
If a customer receives poor service, encounters a problem with a product, or believes a company has failed to meet expectations, why wouldn’t that customer complain?
And certainly, some customers do.
But others don’t.
Some may decide that complaining isn’t worth the effort. Others may believe nothing will change. Some may quietly reduce their purchases, while others simply take their business elsewhere.
An organization that relies primarily on complaints to understand customer dissatisfaction is hearing from those who choose to speak up.
It may be missing many others who have a problem.
Now consider what happens when management reviews its customer complaint reports and finds relatively few complaints.
It would be tempting to conclude that customers are generally satisfied.
But is that conclusion supported by the evidence?
Not on its own.
A low number of complaints tells us that relatively few complaints were received. It doesn’t tell us how many customers experienced problems, how serious those problems were, or how many dissatisfied customers chose not to say anything.
The absence of complaints is not the same as evidence of satisfaction.
That distinction becomes particularly important when an organization makes decisions about customer service, pricing, product improvements, or customer retention.
A business can feel confident that its customers are satisfied while remaining unaware of problems that may eventually cost it those customers.
Even Accurate Data Can Lead to the Wrong Conclusion
There’s an important point here that sometimes gets overlooked.
Information doesn’t have to be inaccurate to be misleading.
A sales report can be completely accurate. A customer complaint log can faithfully record every complaint received. A financial statement can accurately reflect the company’s financial position.
The problem arises when we ask that information to support a conclusion it cannot reasonably support.
Suppose a company reports that sales increased by 8 percent during the past year.
That’s good news, and the number may be entirely accurate.
But imagine that management interprets the increase as evidence that its customers are more satisfied than they were a year earlier.
Does an increase in sales establish that?
Not necessarily.
Sales may have increased because prices rose, new customers were acquired, or existing customers purchased larger quantities. It’s even possible that some longtime customers became less satisfied while overall sales continued to grow.
The sales figure tells us what happened to revenue. It doesn’t independently establish what happened to customer satisfaction.
The same problem can occur with employee turnover, customer retention, production efficiency, and many other performance measures.
An organization may have accurate information about an outcome without having reliable evidence about the reasons behind it.
The danger isn’t always bad data. Sometimes it’s drawing conclusions that go beyond what good data can tell us.
From Information to Evidence
So what makes information useful as evidence?
It begins with the question we’re trying to answer.
Suppose a company president asks, “How are we doing?”
That’s a reasonable question, but it’s also a broad one.
Are we talking about financial performance? Customer satisfaction? Employee engagement? Product quality? Competitive position?
Each represents a different question, and each may require a different kind of information.
If the question is whether revenue increased, financial reports may provide precisely the evidence needed.
If the question is why customers are leaving, those same reports may be insufficient.
And if the question is whether customers are satisfied, conversations with a few familiar customers may offer useful insights without providing a dependable picture of the customer base as a whole.
The important point is that evidence isn’t defined simply by the amount of information available. It’s defined by whether that information can reliably answer the question being asked.
This is where well-designed survey research can make a meaningful contribution.
When properly planned and conducted, a survey can reach people whose experiences and opinions may not be represented in existing reports or informal conversations. It can help management understand how widespread a particular concern may be, whether different groups have different experiences, and which issues deserve closer attention.
But surveys aren’t automatically reliable simply because they produce additional data.
The people selected to participate, the questions asked, the way information is collected, and the interpretation of the results all influence whether a survey produces dependable evidence.
A poorly designed survey can actually make matters worse by giving management another set of numbers to trust, even when those numbers don’t accurately represent the people or issues being studied.
The goal isn’t simply to collect more information.
It’s to obtain information that can be trusted for the decision at hand.
Asking What We Still Don’t Know
I believe one of the most useful habits a business leader can develop is periodically questioning whether the information already available is sufficient.
Not because existing information is unimportant. And certainly not because experience and judgment should be disregarded.
Rather, because even well-managed organizations can develop blind spots.
A company may know exactly how many customers it lost without knowing why they left. It may have detailed employee turnover statistics without understanding what prompted employees to resign. It may closely monitor customer complaints without recognizing the concerns of customers who never complain.
In each instance, management possesses useful information.
But there may be something important missing.
That doesn’t mean every unanswered question requires a survey. Sometimes existing records can provide the answer. Sometimes a closer examination of internal data will uncover what management needs to know. And sometimes additional research is necessary to understand what the available information cannot explain.
The important thing is recognizing the difference.
Before making a consequential business decision, it’s worth asking three questions:
- What do we actually know?
- What are we assuming based on what we know?
- What additional evidence, if any, would help us make a better decision?
Those questions can help distinguish information that supports a conclusion from information that merely makes the conclusion seem reasonable.
Confidence Should Come from Evidence
I have considerable respect for business leaders who know their organizations well. Their experience, instincts, and understanding of the marketplace are often among their greatest strengths.
But the most experienced executive can still encounter a problem that existing information doesn’t adequately explain.
When that happens, the answer may not be another report, another dashboard, or another collection of statistics.
It may be recognizing that an important question remains unanswered.
Organizations rarely suffer from a shortage of data. More often, the challenge is determining whether the information they already possess is capable of answering the questions that matter most.
Because having plenty of information can create confidence.
Having reliable evidence is what helps make that confidence justified.