When I conducted my first survey for pay back in 1985, my data collection tools were pretty straightforward: a telephone, the U.S. Postal Service, and sitting down face-to-face with someone to ask questions.
Online surveys weren’t part of the picture. In fact, the technology we take for granted today would have been difficult to imagine.
More than 40 years later, the world of survey research looks very different. We can distribute questionnaires to thousands of people almost instantly, monitor responses as they arrive, and have data ready for analysis in a fraction of the time it once required.
For someone who began his career when questionnaires arrived in the mail and responses were entered manually, these advances have been remarkable.
But here’s something I have come to appreciate over those years: While the technology has changed dramatically, the principles of good survey research have not.
We still need to know what we’re trying to learn and why. We still need to survey the right people, ask carefully constructed questions, collect trustworthy information, and understand what the findings mean before making decisions.
Technology can make all of that easier. It can also make it easier to produce questionable results without realizing anything has gone wrong.
That’s what I want to explore here.
Online surveys have become an important part of the researcher’s toolkit, offering speed, convenience, flexibility, and often lower data collection costs. But they have also introduced challenges that deserve our attention.
I’ve organized 11 of those challenges around four questions that I believe every organization conducting online surveys should consider:
- Can respondents complete the survey well?
- Can we trust who is responding?
- Can we trust how the technology behaves?
- Can we trust the resulting data?
These questions provide a useful way to think about online survey quality, whether you’re conducting research yourself or working with a professional research organization.
Can Respondents Complete the Survey Well?
We sometimes become so focused on the information we want to collect that we give too little consideration to the experience of the people providing it.
But respondents are doing us a favor. They’re giving us their time, attention, and information. We should make that experience as straightforward and comfortable as possible.
Five considerations are particularly important.
1. Mobile-First Survey Design
Not all that many years ago, when planning an online survey, someone would eventually say, “Oh, and let’s make sure it works on a cell phone.”
That was a reasonable concern at the time. Today, however, I think we’ve reached the point where we need to reverse our thinking.
Rather than designing a questionnaire for a computer and then making certain it works on a phone, we should design it with the expectation that a substantial portion of respondents will use their phones from the outset.
That changes how we approach question layout, response choices, and even the amount of information we ask people to read.
For example, a matrix containing eight statements and five response choices may look perfectly reasonable on a desktop monitor. On a smartphone, it can become an exercise in scrolling, zooming, and trying to remember which response column you’re selecting.
When evaluating a questionnaire for mobile use, I would ask whether the questions are easy to read without zooming, whether response choices are clearly distinguishable, and whether someone can comfortably navigate the survey with one thumb.
I’d also look closely at lengthy instructions, excessive typing requirements, complicated matrices, unnecessary images, and anything requiring horizontal scrolling.
The important point is that these aren’t simply matters of appearance or convenience.
Poor questionnaire layout can influence the answers people give.
And when that happens, a technology issue becomes a survey measurement issue.
2. The Respondent Experience
When we begin designing a survey, our natural inclination is to think about what we want to know.
What questions should we ask? What information do we need? How much detail would be helpful?
Those are reasonable questions, but I would suggest adding another:
What are we asking respondents to experience when they take this survey?
Think about a questionnaire that begins with a lengthy introduction, asks several questions that seem repetitive, presents complicated response formats, and then requests personal information without explaining why it’s needed.
Individually, each of those elements might seem minor. Collectively, they can create a frustrating experience.
I’ve always believed that a good questionnaire should feel like a logical conversation. One question should lead naturally to the next. Instructions should be clear. Respondents should understand what they’re being asked and be able to provide answers without unnecessary effort.
We should also avoid surprising respondents with a survey that takes substantially longer than they expected or a progress indicator that gives them a misleading impression of how much remains.
Why does this matter?
Because a respondent who becomes frustrated, confused, or impatient may begin paying less attention to the questions. Some will abandon the survey entirely.
The result can be incomplete information or answers that don’t accurately reflect what the respondent thinks.
Respondent burden isn’t just an inconvenience. It’s a potential source of survey error.
3. Device Compatibility
One of the advantages of online surveys is that people can participate using whatever technology is convenient for them.
Unfortunately, that also means our questionnaire may be viewed on desktop computers, laptops, tablets, iPhones, Android phones, and a variety of screen sizes and browsers.
It may be completed using a keyboard, a mouse, a touchscreen, or assistive technology.
A questionnaire that works well in one environment may behave quite differently in another.
For that reason, I wouldn’t be satisfied simply knowing that a survey opens successfully on different devices.
We need to make certain that the device someone uses doesn’t affect how they understand a question or the answer they provide.
If a question looks different or is more difficult to answer on a phone than on a desktop computer, we may be introducing differences into the data that have nothing to do with what respondents actually think.
That requires testing.
Before launching an online survey, I would test it on desktop computers, tablets, and phones, using common browsers such as Chrome, Edge, and Safari. I’d pay particular attention to question formats that can be troublesome, including matrices, sliders, ranking exercises, long response lists, and questions involving images or video.
I’d also test navigation, the recording of responses, and what happens when someone skips a question or changes an earlier answer.
Modern survey platforms offer an impressive variety of question formats. Some are visually appealing and can make a questionnaire look quite sophisticated.
But I’ve learned that just because a platform makes something possible doesn’t necessarily mean it’s a good idea.
The objective isn’t to impress respondents with technology. It’s to make it easy for them to provide accurate answers.
4. Accessibility
Accessibility is another consideration that deserves more attention than it sometimes receives.
When designing a questionnaire, we need to recognize that some respondents may have visual, motor, cognitive, or other limitations that affect how they interact with technology.
A survey may need to accommodate screen readers, keyboard navigation, readable typefaces, appropriate color contrast, and straightforward instructions.
Certain interactive question types may create unnecessary barriers.
There’s an important consideration here.
Suppose we’re conducting a survey among the general population, but some members cannot participate because the questionnaire isn’t accessible to them.
We may unintentionally leave out an important group of people. And if those members have different experiences or opinions from the people who complete the survey, our findings may present an incomplete or misleading picture.
In other words, a problem with the way the questionnaire was designed can become a problem with the accuracy of our results.
And, of course, making reasonable accommodations so that people can participate is also an ethical responsibility.
Accessibility isn’t an optional finishing touch. It deserves attention while the questionnaire is being developed.
5. Survey Fatigue
We live in a world filled with requests for feedback.
Buy something at a store, eat at a restaurant, have a package delivered, visit a website, or contact customer service, and there’s a reasonable chance someone will ask you to complete a survey.
Sometimes the requests seem almost endless.
I believe this creates two kinds of fatigue.
The first occurs when people become tired of receiving survey invitations. The second happens within an individual questionnaire when it becomes too long, repetitive, or demanding.
Both can affect survey quality.
As respondents become fatigued, they may devote less attention to their answers, rush through questions, or stop participating altogether.
The usual response is to recommend shorter surveys. That’s sensible, but I think there’s a more important consideration.
We should be conducting more purposeful surveys.
Every question should have a reason for being included. Ideally, we should be able to explain what we expect to learn from it and how that information might contribute to a decision.
If we cannot explain why we’re asking something, perhaps we shouldn’t be asking it.
The goal isn’t simply to make the questionnaire shorter. It’s to collect the information necessary to accomplish the survey’s purpose without imposing unnecessary demands on respondents.
Can We Trust Who Is Responding?
A well designed questionnaire doesn’t accomplish much if we don’t have confidence in the people providing the answers.
With online surveys, that concern has become increasingly important.
Unlike many traditional survey methods, where researchers often had more direct contact with respondents, online data collection can involve individuals recruited through sources we know relatively little about.
Two considerations deserve particular attention.
6. Online Panel Quality
One of the first questions organizations sometimes ask when considering an online research panel is, “How much does it cost per respondent?”
Cost matters, certainly. But I would encourage asking a different question first:
Where do these respondents come from?
Not all online panels are alike.
Some surveys begin with an organization’s customer or employee list. Others recruit respondents through sampling methods in which people have a known chance of being selected. Still others rely on people who voluntarily sign up to participate in online research panels.
There’s an important difference between selecting people randomly from a known population and selecting people from a group that volunteered to take surveys.
For example, suppose we want to learn what adults throughout the United States think about a particular issue.
We could use a sampling approach in which people have a known chance of being selected. Or we could obtain respondents from a large online panel of people who signed up to participate in surveys.
Both approaches may provide useful information, but they aren’t necessarily interchangeable.
And randomly selecting people from that volunteer panel doesn’t automatically make the results representative of all American adults.
That’s why I believe it’s so important to understand where our respondents come from before worrying about how much they cost.
When evaluating a panel provider, I’d want to know how people are recruited, how their identities and qualifications are verified, how frequently they participate, and whether they belong to multiple panels.
I’d also ask about how the panel provider makes certain the survey includes the kinds of people we’re trying to reach, what steps are taken to identify poor-quality responses, and what happens when questionable interviews are discovered.
None of this means online panels are inherently unreliable. It means their quality depends on how they are built, maintained, and used.
And those details matter much more than the appearance of the finished questionnaire.
7. Bots and Fraudulent Respondents
This is one of the more troubling developments in online research.
In traditional research, we were primarily concerned with whether the right people had been selected and whether their answers were accurate.
Today, particularly when incentives are offered, we must also consider whether some of our apparent respondents are legitimate people at all.
Automated bots, duplicate submissions, misrepresented identities, and other forms of fraudulent participation can find their way into online survey data.
Some people may even participate repeatedly or provide invented information simply to qualify for a reward.
Fortunately, many survey platforms have tools to help identify suspicious activity.
But I wouldn’t rely entirely on automated detection.
We should examine response patterns that may suggest problems, including unusually fast completions, identical answers across long sets of questions, inconsistent information, duplicate identifiers, and nonsensical written responses.
Repeated or suspicious open-ended answers can also be revealing.
One caution is important: no single indicator necessarily proves that a response is fraudulent.
A person who completes a questionnaire quickly may simply be a fast reader. Someone who gives the same rating to several statements may genuinely feel that way.
We need to look for patterns and consider the evidence before deciding that a response should be excluded.
The larger lesson is straightforward.
A completed questionnaire isn’t necessarily evidence that a legitimate respondent provided trustworthy information.
Can We Trust How the Technology Behaves?
Online survey platforms have become remarkably capable.
They can direct respondents through different questions, customize wording based on earlier answers, rotate response choices, enforce quotas, and perform tasks that once required considerable manual effort.
These are valuable capabilities.
But as questionnaires become more technically sophisticated, they can also become more difficult to evaluate.
Two areas deserve close attention.
8. Automated Logic and Personalization
One of my favorite advantages of online surveys is the ability to ask respondents only those questions that apply to them.
For example, a customer who has never contacted a company’s technical support department shouldn’t have to answer a series of questions about the service they received.
Skip logic allows us to bypass those questions automatically.
Other features let us insert earlier responses into later questions, change the order of response choices, and customize the questionnaire according to a respondent’s circumstances.
When properly designed, these features make surveys more efficient and relevant.
The danger is that a questionnaire can appear to work perfectly while containing errors that are difficult to see.
Perhaps one answer sends respondents to the wrong section. A particular combination of answers might cause an important question to be skipped. Or a response may be recorded incorrectly in the data.
Think of an online questionnaire as having several possible routes through it. The route a respondent takes depends on the answers given along the way.
We need to test those different routes rather than simply run through the questionnaire once and assume everything works.
What happens when someone skips a question? What happens when two answers lead to different sections? Are the correct questions appearing, and are the answers being recorded properly?
These are the kinds of things that need to be checked before the survey goes live.
And when questions or response choices are presented in different orders to different respondents, we should make certain that feature works as intended.
Successfully testing a questionnaire once doesn’t establish that every possible route works correctly.
9. AI-Assisted Survey Platforms
Artificial intelligence is rapidly becoming part of the online survey process.
Many platforms now offer AI assistance with drafting questions, generating response choices, developing questionnaire logic, translating wording, categorizing open-ended responses, identifying themes, and summarizing findings.
These capabilities can be very useful. They can save time and help researchers work more efficiently.
But I see a potential problem that deserves careful consideration.
AI can help create a questionnaire that looks professionally designed without ensuring that it measures what we actually need to know.
The same concern applies to the analysis of results.
An AI-generated summary may be well written and sound convincing while overlooking an important distinction, misunderstanding responses, or reaching a conclusion the evidence doesn’t support.
This isn’t an argument against using AI. It’s an argument for exercising professional judgment when we do.
When AI helps write a questionnaire, someone still needs to determine whether the questions are clear and whether they’re really asking what we intended.
And when AI helps analyze responses, we need to make sure it hasn’t overlooked important comments, misunderstood what respondents meant, or drawn conclusions that the evidence doesn’t support.
I’d also want assurance that confidential or sensitive information wasn’t being exposed to an inappropriate AI system.
Most importantly, a knowledgeable person should review and evaluate the work.
AI can be a helpful assistant in survey research. But it needs supervision and cannot relieve us of responsibility for the quality of the research or the conclusions we draw from it.
Can We Trust the Resulting Data?
Ultimately, the value of any survey depends on the information it produces.
We need confidence that the data were collected using appropriate methods, that respondents understood and answered the questions, and that the results have been handled responsibly.
The final two considerations address that responsibility.
10. Security and Privacy
When someone participates in a survey, particularly one involving personal or sensitive information, it’s reasonable for that person to wonder what will happen to their answers.
Who’s conducting the survey? Why is this information needed? Who will see the results? Can my responses be traced back to me?
Those are legitimate questions, and researchers should be prepared to answer them.
If respondents are uncertain about how their information will be used, some may decline to participate. Others may be reluctant to provide completely candid answers.
Either outcome can affect survey quality.
We should explain who is sponsoring the research, why the information is being collected, how the findings will be used, and what protections are in place.
We should also be disciplined about what information we collect in the first place.
Just because a survey platform allows us to ask for personal information doesn’t mean we need it.
Before including a sensitive question, I’d want to know why the information is necessary, who will have access to it, how it will be protected, and how long it will be retained.
There’s another distinction I want to emphasize because I’ve seen the terms used interchangeably.
Anonymity and confidentiality are not the same thing.
An anonymous survey is one in which the researcher cannot identify individual respondents from the survey data. There is no name or other identifying information that permits the responses to be connected to a particular individual.
A confidential survey, on the other hand, is one in which the researcher can, or potentially could, identify respondents but has committed to protecting their identities and not disclosing identifiable information.
Both approaches can be appropriate, depending on the circumstances.
What matters is that respondents are given an accurate explanation of which protections apply.
Promising anonymity when a survey isn’t actually anonymous can undermine the very trust we’re trying to establish.
11. Online Survey Quality Control
Earlier in my career, data quality control was often associated with what happened after data collection ended.
We examined the completed questionnaires, checked for obvious problems, and cleaned the data before analysis.
Those activities remain important, but with online surveys, quality control needs to begin much earlier.
I find it helpful to think of quality control as occurring in three stages.
Before data collection: Testing
This is where we make certain the questionnaire is ready for respondents.
We test question wording, instructions, and how respondents are directed through the questionnaire based on their earlier answers.
We also check whether questions and response choices appear in the correct order, whether the right people are being allowed to participate, and whether their answers are being recorded correctly.
Mobile displays, different browsers and devices, accessibility, survey links, and the estimated time required for completion should also be evaluated.
The objective is to find problems before respondents encounter them.
During data collection: Monitoring
Once the survey is underway, we need to pay attention to what’s happening.
Are responses arriving at the expected pace? Are certain groups underrepresented? Are respondents abandoning the questionnaire at a particular point? Are some interviews being completed unusually quickly?
We should also watch for suspicious traffic patterns, duplicate submissions, unexpected differences among respondents using different devices, and questionable answers to open-ended questions.
The advantage of online data collection is that we often have access to this information while fieldwork is taking place.
We should use it.
After data collection: Evaluation
When fieldwork ends, we still need to examine the data carefully.
That includes reviewing incomplete questionnaires, inconsistent answers, respondents who give the same answer repeatedly without appearing to consider the questions, unusually fast completions, duplicate records, and signs of possible fraud.
We should also look for answers that suggest the questionnaire didn’t work as intended.
Open-ended responses deserve attention as well. They can reveal both valuable insights and possible quality concerns.
When decisions are made to exclude or modify records, the rules and reasons should be documented. The original data should be preserved.
Quality control isn’t simply an exercise in removing questionable responses after the fact.
It needs to be built into the research process from beginning to end.
Technology Changes. Good Research Principles Remain.
Looking back over more than 40 years in survey research, I’m struck by how much our tools have changed.
The telephone interviews and mailed questionnaires that were so familiar early in my career now share the stage with technologies that would have seemed extraordinary at the time.
And I expect the pace of change will continue.
Online surveys have opened opportunities to collect information faster, more efficiently, and in ways that once would have been impractical.
But they haven’t eliminated the fundamental responsibilities of the researcher.
We still need to clearly define the purpose of the study. We still need to reach the right people, ask meaningful questions, protect the integrity of the information, and interpret the results with care.
Perhaps the greatest risk with today’s technology isn’t that it fails to work.
It’s that it works so smoothly that we assume the research itself must be sound.
A questionnaire can look professional, attract hundreds or thousands of responses, and produce impressive charts in minutes. None of those things, by themselves, establishes that the information is reliable.
That requires thoughtful planning, sound methods, careful execution, and good judgment.
Technology can help us do better research.
But it remains our responsibility to make certain that we do.