When online survey companies remove respondents from their studies, it’s treated as an open-and-shut case, no questions asked. The industry believes there are good respondents and bad ones, and the overriding goal is to keep the bad ones out.
The model is built upon suspicion, and encourages it. But treating research audiences with suspicion can negatively affect those audiences, the data they produce, and, in the long run, what research companies can do with the audiences available to them.
What would happen if the core question was not just “Is this respondent good or bad,” but rather, “Is this respondent right for this study?” That’s how we’ve modeled our approach to survey respondent quality at Research For Good, and we’re already seeing exciting developments on both sides of the survey: a better respondent experience and better data.
The two go hand in hand, and improved respondent matching allows us to deliver both.
What Respondent Matching Looks For
It happens all the time: respondents pass fraud checkers and still deliver unreliable or dishonest responses in research activities. In other words, someone ended up in your study who shouldn’t be there.
It doesn’t necessarily mean that they were looking to game your study for incentives from the outset. (Such fraudsters are a separate group we’ve identified through behavioral segmentation.) Still, incentives might keep them there, if not genuine interest in taking a survey.
Say a study about luxury goods draws someone in with an appreciation for high-end jewelry, and a confidence that they know what they’re talking about. But their responses make it clear they don’t quite fit the demographic profile the brand intends to explore, thus posing a data risk.
Might that same person be useful in another study? A study about luxury goods may prompt them to over-claim ownership of high-end jewelry because they think they have to, or because they’re answering aspirationally. But in a study about streaming television consumption, they may have no reason to act this way.
In other words, a respondent who overstates or misrepresents their own circumstances in one context may give entirely accurate information about other topics in a different one. Aspirational answering and inflated ownership claims are often contextual responses, not fixed characteristics.
If we followed an approach to unreliable respondents that asks “Is this respondent good or bad?” we would discard this respondent. They don’t fit the profile; therefore they don’t belong in the study. But the question “Is this respondent right for this study?” leads to another question: “Is there a study they would be right for?” That question is at the core of respondent matching.
How Respondent Matching Improves Our Practice, And Our Data
At Research For Good, our focus on respondent matching has had two major benefits: greater control over the data that studies produce, and a more reliable recruitment pool.
Using behavioral markers to identify audience members who are mismatched to a study allows us to make an informed decision about whether to keep their responses, and whether to select those people for studies where they might be a better fit.
In other words, we now know more about who these respondents are, what kind of risk they represent to any given study, and how to account for it — whether that’s exclusion, or simply more selective matching.
These capabilities are only growing. DataForce, our platform for participant recruitment and management, has built a profiling system containing nearly 1,000 targetable attributes to optimize respondent-to-activity fit. This level of specificity is unprecedented.
The result is better data and a better respondent experience. A person who arrives at a survey that’s actually relevant to their life and their interests is more engaged. More engaged respondents give more careful, more accurate answers. The quality of the data, and the quality of the experience for the person providing it, align.
The Human Impact Of Leading With Suspicion
The industry’s survey data quality problem is in large part a human experience problem. Questioning survey respondent quality puts the onus on research participants, when we should be looking at the experience we’re delivering for these people.
Audience providers and researchers assume that the risk of letting in bad respondents outweighs the cost of subjecting everyone to the same suspicion. That’s why attention checks, captchas, speed traps, and red herrings are deployed uniformly, regardless of whether participants have behaved suspiciously after passing fraud checks.
But attention checks and speed traps take a real toll on a genuine participant’s experience. They slow the survey down and introduce friction for people who showed up to answer questions honestly.
We want people to want to take surveys. But negative experiences have become the norm in online research over the years.
We want people to want to take surveys. But negative experiences have become the norm in online research over the years, and fewer and fewer people want to take surveys. Recruitment and participation are consistently challenging for firms, and that is just as much a factor in poor data quality in market research as fraud, if not more so.
Those who do keep participating often deliver lower-quality responses because they’re worn down. Meanwhile, participants with a unique tolerance for poor survey experience can’t be taken as representative of the general population.
“Keep or toss” is an increasingly unhelpful mindset for the online research industry. The next step forward for researchers is to think of research audiences primarily as people looking for the right place. We can vouch for the results.
When Removal Is the Right Answer
None of this argues against removal. Fraud detection is table stakes; there are respondents who are not appropriate for any study, whose answers are unreliable across contexts, and who should be excluded.
Instead, we’re calling for a higher burden of proof for removal. Removing a respondent for behavior inconsistent with the study’s subject matter is very different from removing a respondent because an algorithm flagged their IP address as suspicious.
Researchers who put that difference into practice will produce better data, recruit more effectively in the future, and give clients a clearer view of their methodology.
In addition to asking whether a respondent is out to game the system, ask whether they belonged in the study. Before removing them, ask whether you understand why their answers look the way they do. Before treating every respondent as a suspect, ask whether that’s the relationship you want to have with the people whose honest participation your research depends on.
Because research is a relationship with your audience, and data quality measures that relationship’s strength. Respondent matching is one of many tools meant to give that audience what they need: studies that match their interests, fair incentives, and a research experience that’s neither hostile nor burdensome, and empowers them to answer comfortably and effectively.
When you give your audience what they need, they’ll be sure to return the favor.
