When online survey companies remove respondents from their studies, it’s treated as an open-and-shut case, no questions asked. The industry believes there are good respondents and bad ones, and the overriding goal is to keep the bad ones out.
The model is built upon suspicion, and encourages it. But treating research audiences with suspicion can negatively affect those audiences, the data they produce, and, in the long run, what research companies can do with the audiences available to them.
What would happen if the core question was not just “Is this respondent good or bad,” but rather, “Is this respondent right for this study?” That’s how we’ve modeled our approach to survey respondent quality at Research For Good, and we’re already seeing exciting developments on both sides of the survey: a better respondent experience and better data.
The two go hand in hand, and improved respondent matching allows us to deliver both.
It doesn’t necessarily mean that they were looking to game your study for incentives from the outset. (Such fraudsters are a separate group we’ve identified through behavioral segmentation.) Still, incentives might keep them there, if not genuine interest in taking a survey.
Say a study about luxury goods draws someone in with an appreciation for high-end jewelry, and a confidence that they know what they’re talking about. But their responses make it clear they don’t quite fit the demographic profile the brand intends to explore, thus posing a data risk.
Might that same person be useful in another study? A study about luxury goods may prompt them to over-claim ownership of high-end jewelry because they think they have to, or because they’re answering aspirationally. But in a study about streaming television consumption, they may have no reason to act this way.
In other words, a respondent who overstates or misrepresents their own circumstances in one context may give entirely accurate information about other topics in a different one. Aspirational answering and inflated ownership claims are often contextual responses, not fixed characteristics.
If we followed an approach to unreliable respondents that asks “Is this respondent good or bad?” we would discard this respondent. They don’t fit the profile; therefore they don’t belong in the study. But the question “Is this respondent right for this study?” leads to another question: “Is there a study they would be right for?” That question is at the core of respondent matching.
How Respondent Matching Improves Our Practice, And Our Data
At Research For Good, our focus on respondent matching has had two major benefits: greater control over the data that studies produce, and a more reliable recruitment pool.
Using behavioral markers to identify audience members who are mismatched to a study allows us to make an informed decision about whether to keep their responses, and whether to select those people for studies where they might be a better fit.
In other words, we now know more about who these respondents are, what kind of risk they represent to any given study, and how to account for it — whether that’s exclusion, or simply more selective matching.
These capabilities are only growing. DataForce, our platform for participant recruitment and management, has built a profiling system containing nearly 1,000 targetable attributes to optimize respondent-to-activity fit. This level of specificity is unprecedented.
The result is better data and a better respondent experience. A person who arrives at a survey that’s actually relevant to their life and their interests is more engaged. More engaged respondents give more careful, more accurate answers. The quality of the data, and the quality of the experience for the person providing it, align.
The Human Impact Of Leading With Suspicion
The industry’s survey data quality problem is in large part a human experience problem. Questioning survey respondent quality puts the onus on research participants, when we should be looking at the experience we’re delivering for these people.
Audience providers and researchers assume that the risk of letting in bad respondents outweighs the cost of subjecting everyone to the same suspicion. That’s why attention checks, captchas, speed traps, and red herrings are deployed uniformly, regardless of whether participants have behaved suspiciously after passing fraud checks.
But attention checks and speed traps take a real toll on a genuine participant’s experience. They slow the survey down and introduce friction for people who showed up to answer questions honestly.
We want people to want to take surveys. But negative experiences have become the norm in online research over the years.
We want people to want to take surveys. But negative experiences have become the norm in online research over the years, and fewer and fewer people want to take surveys. Recruitment and participation are consistently challenging for firms, and that is just as much a factor in poor data quality in market research as fraud, if not more so.
Those who do keep participating often deliver lower-quality responses because they’re worn down. Meanwhile, participants with a unique tolerance for poor survey experience can’t be taken as representative of the general population.
“Keep or toss” is an increasingly unhelpful mindset for the online research industry. The next step forward for researchers is to think of research audiences primarily as people looking for the right place. We can vouch for the results.
When Removal Is the Right Answer
None of this argues against removal. Fraud detection is table stakes; there are respondents who are not appropriate for any study, whose answers are unreliable across contexts, and who should be excluded.
Instead, we’re calling for a higher burden of proof for removal. Removing a respondent for behavior inconsistent with the study’s subject matter is very different from removing a respondent because an algorithm flagged their IP address as suspicious.
Researchers who put that difference into practice will produce better data, recruit more effectively in the future, and give clients a clearer view of their methodology.
In addition to asking whether a respondent is out to game the system, ask whether they belonged in the study. Before removing them, ask whether you understand why their answers look the way they do. Before treating every respondent as a suspect, ask whether that’s the relationship you want to have with the people whose honest participation your research depends on.
Because research is a relationship with your audience, and data quality measures that relationship’s strength. Respondent matching is one of many tools meant to give that audience what they need: studies that match their interests, fair incentives, and a research experience that’s neither hostile nor burdensome, and empowers them to answer comfortably and effectively.
When you give your audience what they need, they’ll be sure to return the favor.
Fraud detection and data quality are not the same thing. While many researchers and audience providers treat them as synonymous, the terms refer to different questions. Fraud detection asks whether a participant is real and acting in good faith, while data quality asks whether what they told you is accurate.
Tools like IP filtering and device fingerprinting were built to answer the first question, but were never designed to answer the second. So how can researchers answer it?
Going beyond fraud detection means changing what determines who gets tested, and why. The identity signals that trip fraud detection tools aren’t enough to help researchers understand who is engaging with the research instrument, and whether their responses accurately reflect them.
Behavioral segmentation is the next step for online market research. While fraud detection boots out obvious bots and known bad actors, segmentation works as a fine-toothed comb that catches the stragglers. In some cases, that means spotting more bad actors; but in many cases, it means finding genuine participants who ended up in the wrong place.
Together with traditional fraud detection, it allows for greater control over the data that surveys produce, with major implications for survey data quality. Here’s how it works.
The Bad Actors Traditional Fraud Checkers Overlook
It’s assumed that once the fraud checkers have done their job, the answers that research audiences provide are reliable. In practice, this isn’t always the case, which compromises survey data quality. But we’ve found that unreliable responses follow reliable patterns.
At Research For Good, our own participant data showed that people who entered certain zip codes — 10001 for Midtown Manhattan, 90210 for Beverly Hills — didn’t behave like the rest of the population.
On their own, these zip codes weren’t fraud signals. None of these audience members triggered connection or device checks. But their survey responses were inconsistent and suspicious enough for us to determine that they posed a risk to data quality. It begged the question: did these people actually live in these neighborhoods?
There are a couple reasons why someone might report a zip code that isn’t their own. Privacy is one reasonable conclusion; people withhold real details online for all sorts of defensible reasons. But privacy doesn’t explain why Beverly Hills, one of the most recognizable and wealth-coded zip codes in the country, would be the go-to answer for such audience members.
Unreliable responses follow reliable patterns.
In some cases, audience members might be providing aspirational answers: describing the life they’d like to have, not the one they do have. This is a real psychographic pattern we’ve observed. But these over-claimers make up a small minority.
In most cases, we’ve found that these are fraudulent actors inputting attributes that get them into surveys they aren’t qualified to join and collecting incentives across multiple studies.
We dubbed these individuals “operators”: a variety of fraudsters who can slip past traditional fraud detection because they often aren’t using any of the technological deception methods that fraud checkers look for. They’re simply lying the old-fashioned way. Sometimes they’re part of an enterprise, and other times they fly solo, but the manual misrepresentation is the same.
Without an additional detection layer, operators slip through with ease. In low-incidence or specialty audiences, unreliable responses can reshape findings entirely, which makes operators a serious threat to survey data quality.
Behavioral segmentation is a progression from traditional fraud detection. It allows us to recognize, flag, and test for suspicious behaviors.
Using behavioral segmentation, we’ve identified a cluster of behaviors consistent enough across studies that we can now anticipate when these bad actors tend to show up, test them accordingly, and, should their responses line up with what we’ve identified as a fraudulent behavior pattern, flag them as a data risk. Here’s what it looks like.
How We Spot Operators, And How They Affect Your Data
An input like a high-risk zip code is not a red flag on its own. But when certain behaviors follow, we know we’ve found an operator.
Over-claiming is the clearest example. Across our data, we’ve observed that certain respondents in the 90210 and 10001 zip codes claim ownership of luxury vehicles and high-value financial assets at rates that significantly outrun real-world ownership statistics.
A look at auto ownership claims among US males aged 18–65, all of whom passed standard fraud detection, showed stark differences between respondents who passed our behavioral tests and those who failed them. BMW ownership was claimed at nearly triple the rate among test-fail respondents. Alfa Romeo ownership was claimed at more than ten times its real-world rate.
Financial asset data showed even more extreme variation. The share of test-fail respondents claiming no financial assets was a quarter of the rate seen among test-pass respondents. Most asset types appeared at multiples of their realistic prevalence.
This specific behavior pattern is reliable enough that we test for operators within this zip code group using survey questions specifically designed to elicit it. When a participant’s responses line up with the behavior patterns of an operator, we know that we can exclude that participant’s responses from the data.
The key difference in Research For Good’s approach is who gets tested and why.
How Behavioral Segmentation Catches What Fraud Detection Misses
Misrepresentation is not demographic in origin, but psychographic. You cannot identify it through methods like IP filtering or device fingerprinting. Catching that misrepresentation requires different tools — ones that don’t test everyone the same way, like traditional fraud checkers, but instead apply scrutiny where the risk is highest.
Research For Good’s system is called the Quality Interception Point, or QuIP. Rather than applying the same battery of checks to every respondent, QuIP uses behavioral and profile markers to identify respondents who are more likely to exhibit suspicious response patterns before the study begins. For example: a 90210 area code.
Once identified, these audience members receive the attention questions and speed traps that, in typical survey design, are applied to every respondent. But we also serve them questions that are deliberately designed to elicit the kind of aspirational or inconsistent responding that signals unreliability.
Critically, the tests are designed not to be obviously identifiable as tests. Answer lists are carefully managed; anchor points that give respondents somewhere predictable to land are removed; questions mix real and unrealistic scenarios, so there is no intuitive way to determine what the “correct” response is supposed to be.
Respondents prone to over-response, such as operators, will reveal themselves under these conditions by failing the test outright. But the system is not singularly focused on booting the operators. When a participant’s responses show some, but not all, of the patterns of an operator, that indicates that the participant might simply be mismatched to the survey.
With QuIP, we can flag the operators who pose a high risk to the data, wave through the low-risk participants who pass every behavioral test, and make an informed decision about what to do with the participants who fall somewhere in between: for instance, discarding their responses in this study, but matching them to a more appropriate one in the future.
How Behavioral Segmentation Empowers Researchers
When Research For Good applied behavioral segmentation to our own work, our internal rejection rate fell by roughly 40%. Our overall rejection rate — the share of completes discarded for fraud, duplication, or quality issues — sits at approximately 4%, compared to an industry average of around 18%.
The most exciting gain is control. When a study returns results that don’t look right, data interpretation is driven not by a hunch, but by clear definitions for likely-unreliable respondent groups. For example: a diagnostic quota can be run to understand what’s actually driving the anomaly. On a tracker, a small oversample at baseline allows calibration across later waves, so a shift in who answered doesn’t get mistaken for a shift in the market.
None of it requires auditing a file record by record, question by question. The behavioral layer surfaces what the traditional fraud detection layer can’t see, painting the most accurate data picture possible and surfacing viable options.
The goal of behavioral segmentation in market research is not merely to remove more respondents and make up for a matching problem upstream. The goal is to understand more about the respondents who are there.
What’s more, the benefits of this approach don’t end with the study that’s in front of you now. This approach can also help researchers recruit the best possible audience for their next study.
The relentless rise of survey fraud has put data quality at the top of every survey researcher’s mind. But fraud and data quality are different concerns.
To be sure, bad actors are a real threat that leads to unreliable findings and inflated costs. But treating fraud detection and data quality in market research as synonymous is a mistake.
The assumption is that if you keep bad actors out, what’s left afterward must be good: no fraud, all quality data. However, even when fraud prevention works perfectly, researchers still note high rates of what appear to be bad-faith actions by respondents, such as straight-line responses and speeding.
The persistence of these behaviors doesn’t mean your fraud checker isn’t doing its job. It means your fraud checker missed something it wasn’t designed to catch: for example, a distracted respondent who misreads a prompt mid-survey and answers the opposite of how they would have responded with their full attention.
Fraud detection is only one part of the overall equation: a minimum that every credible supplier can meet. Going beyond that minimum means uncovering more about the audience behind the responses. It means putting more resources into capabilities that bring us closer to our audience and make the most of what each audience member gives us.
This article will dive into what fraud detection tools actually accomplish, and how they actually impact data quality in market research. We’ll also outline what fraud detection doesn’t accomplish, a gap that should interest those who care deeply about data quality, and researchers who want to know more about their audience.
The Table-Stakes Survey Fraud Detection Tools, and What They Do
Before diving into what fraud detection misses, let’s review what gatekeepers and traps do catch. (For a full rundown of the different tools, head to the appendix.)
Survey gatekeepers largely detect and verify identity. For the most part, they ask three questions: Is this respondent a real human, or an automated script? Has this person already taken this survey? And is this person connecting from where they claim to be?
Duplicate detection and device fingerprinting flag repeat survey-takers, while IP filtering and Geo-IP validation catch anyone masking their true connection location. The question of bots spans multiple methods; whether human or synthetic, bad actors all try to hide their identity by masking their connection and device, which is why one set of tools addresses bots and incentive farmers alike.
Where gatekeeping examines the connection, survey traps examine the response. Attention, logic, and speed checks catch anyone speedrunning a survey; open-ended text analysis catches generative AI while CAPTCHAs handle bots.
Fraud detection plays an important role, especially as fraud enterprises grow more sophisticated. But these tools miss plenty once the real humans are let into the study.
What Fraud Detection Doesn’t Catch
A respondent can clear every gatekeeper and still give you answers you can’t use. Sometimes this amounts to a less sophisticated form of fraud. There are also very human reasons a genuine person taking your survey might give you unreliable answers. Traditional fraud checkers miss both.
Why Genuine People Give Unreliable Answers
Sometimes the survey itself is the problem. A long interview, a dense block of open-ended questions, a heavy grid, or a structure with no clear endpoint can wear a participant down until they disengage and straight-line through a grid or settle into formulaic answers.
Disengaged answering might set off straight-line detectors, speed checkers, and open-ended text analyzers, but the respondent isn’t trying to trick anyone; they’re responding to an activity that wasn’t designed with their experience in mind.
In other cases, a genuine, non-fraudulent participant may fit the study’s demographic profile, but their interests may be misaligned with the activity’s design. They’ll give careless or low-effort answers because the topic doesn’t engage them. Matching respondents to studies is more complex than checking off demographic markers; a frequent traveler who qualifies for a study on airline loyalty programs won’t necessarily want to spend fifteen minutes on point-redemption details.
People might also fib here and there. Certain overstatements, such as claiming ownership of a luxury car, are fairly common among participants who are otherwise completely genuine in their responses. Such behavior is psychographic, and not in and of itself an indicator of fraud; these people may simply be describing the lives they wish they had, not the ones they do have.
Fraud detectors don’t help data suppliers account for any of this, nor for the fraud that can still slip past traditional fraud detection.
What Fraud Looks Like When It Clears Fraud Detection
Overclaiming, as mentioned, can manifest in small fibs and white lies within an individual survey participant’s responses. By contrast, other participants overclaim job titles, income, and details like auto ownership across multiple studies deliberately to misrepresent themselves and gain access to as many studies as possible.
A participant engaging in this behavior is unlikely to trip a fraud checker. Survey gatekeepers have nothing to catch: the participant isn’t trying to access the same study multiple times, but many studies once. This means that device fingerprinting and duplicate detection won’t flag them, and unless the person happens to be using a proxy for internet privacy reasons, IP detection will wave them through as well. Similarly, just because they’re accessing multiple studies doesn’t mean they’re speeding through them, which renders most survey traps moot.
A genuine participant and a fraudster have very different reasons to misrepresent themselves, but both pose data risks; they can inflate incidence rates, distort segmentation models, and introduce bias into trackers, where wave-over-wave comparability is the entire point.
Fraud Detection Is Table Stakes In Online Research
The core shortcoming of survey fraud detection is that it can’t tell you anything about who got into the study, short of the fact of their humanity. As we’ve established, fraud can still happen once the checkers have done their job. Even the absence of fraud does not guarantee quality; the human factors, such as disengagement and survey mismatches, make a real impact.
At Research For Good, we’re focused on developing tools that minimize the distance between researchers and individual audience members. The goal isn’t just to root out fraud more easily. It’s delivering participants who will engage enthusiastically to studies where they’re a perfect fit, thereby helping brands see their audience clearly and better serve them.
Closing that distance is what defines data quality in market research. Fraud detection gets bad actors out of the way; the next step is getting the right people “in the room,” and seeing them clearly once they’re there.
Appendix: An Overview of Standard Fraud Detection Methods
Standard survey fraud detection methods fall into two categories: gatekeepers and survey traps.
Gatekeepers: Pre-Survey Fraud Detection
Gatekeepers largely detect and verify identity. For the most part, they ask three questions: Is this respondent a real human, or an automated script? Has this person already taken this survey? And is this person connecting from where they claim to be?
Each tool serves as a validation method:
Device fingerprinting identifies the same device returning under a new identity.
IP filtering and proxy/VPN detection detect when a respondent’s connection is being masked with a proxy. For instance, a bot running in a data center may route its connection through a residential proxy so that the connection looks like it’s coming from someone’s house.
Geo-IP validation checks the respondent’s claimed location against the connection’s location.
Suspect-activity databases collect respondent identities flagged for suspicious activity by various survey providers.
Duplicate detection checks whether the respondent is attempting to take your survey a second time.
Whether human or synthetic, bad actors all try to hide their identity by masking their connection and device, which is why one set of tools addresses bots and incentive farmers alike.
Survey Traps: Mid-Survey Fraud Detection
Where gatekeeping examines the connection, traps examine the response. They sit inside the survey itself, and they ask a different set of questions: Is this person actually reading? Do their answers hold together? And… did a human write this? (If the answer to that last question is “no,” that means a bot slipped through the fraud checker — which is why both checkers and traps are necessary.)
Each tool tests a different signal:
CAPTCHA distinguishes a human from an automated script.
Attention checks pose a question with one obviously correct answer — e.g., select the color blue — to confirm someone is reading rather than clicking.
Logic and consistency checks compare answers across the instrument, catching contradictions a careful respondent wouldn’t produce.
Speed checks flag when a respondent answers questions faster than a human could possibly read them.
Straight-lining detection identifies responses that march down a grid in a single column: a pattern rather than a set of answers.
Open-end text analysis evaluates written responses for gibberish, copy-paste, machine translation, and increasingly, the linguistic signatures of generative AI.
Gatekeeping sees the door; traps see the room. One confirms who arrived, the other watches what they do once inside.
Together, gatekeepers and survey traps ensure with a high degree of accuracy that only real humans engage with the research instrument.
Join our mailing list to be kept in the know on events, information and useful resources from around the market research sample industry.
Good news, you're on the list and will be the first to know when we've got good info to share!
Categories
Archives
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
