Market researchers and sample buyers want the highest-quality data. To get it, they normally point to one solution: survey fraud detection.

Survey fraud detection does an important job: bot detection, checking for duplicates, speed checks, and attention traps. It asks whether a participant is real and acting in good faith. But it doesn’t ask whether what they told you is accurate. A participant can pass the first check and fail the second, and nothing catches it.

If enough of those participants get into a data set, the numbers start to drift. For market researchers, that means inflated or diluted incidence rates; segmentation models absorb the distortion and return clusters that don’t give a faithful picture of who the audience is. Brand trackers are most exposed of all, where a shift in who happened to answer carelessly can read as a shift in the market itself.

Those same contaminated estimates hit sample buyers differently: misjudged feasibility, hard-to-fill quotas, and runaway pricing that stakeholders will want an answer for.

Fraud detection is table stakes: it can’t solve the problem of data quality alone. But the solution isn’t an additional layer of filtration. Participants responding irregularly in good faith only add noise to the data if they’re in the wrong place; going beyond fraud prevention means finding where in your data these irregular participants belong.

The Participants Fraud Detection Isn’t Built to Catch

Like most online sample providers, we’ve noticed irregular responses from participants our fraud checkers have waved through. In cases where our audience pool was limited, this posed a dilemma: do we strike irregular responders from the data and shrink the pool, or do we preserve the quota at the expense of data quality?

In a general-population study, a handful of discarded participants is absorbable. But in a study of a narrow audience, the same handful can reshape findings. (Making matters even more urgent, studying such specialty audiences usually carries a higher price tag.)

At Research For Good, this surfaced when we took a look at our own traffic. A review of our participant data turned up an odd pattern: participants who entered the zip codes 10001 (Midtown Manhattan) and 90210 (Beverly Hills) didn’t behave like the rest.

It’s worth asking why a participant would supply a zip code they don’t live in. Privacy is one reasonable answer, as people online withhold real details for all sorts of defensible reasons. But privacy doesn’t explain Beverly Hills; protecting your location doesn’t oblige you to claim one of the most recognizable, wealth-coded zip codes in the country. Another possible answer: aspiration. (More on that later.)

Following this oddity led us to findings that were more useful than a list of suspect zip codes: recognizable patterns of behavior, consistent enough to define, name, and sort by.

These participants constituted a distinct group of people. They weren’t gaming the system for payout, and nothing they did tripped a fraud checker. They were real people whose answers, for whatever reason, didn’t hold up as a reliable account of their lives. Participants like this make it into data sets all the time… and we’ve found that sometimes, that’s right where they belong.

How Behavioral Segmentation Improves Upon Fraud Detection

Fraud detection is premised on prevention: keep the wrong people out, and whatever remains will be sound. In an environment as fundamentally human as research, unwanted outcomes can’t be engineered away entirely. It’s more realistic, and more beneficial, to give researchers what they need to understand the data they actually have.

Standard fraud prevention tests everyone at the door and admits only those who pass, treating participants as suspect until proven otherwise. To go beyond standard methods, we have to change what determines who gets tested. At Research For Good, we’ve collected years’ worth of participant data that reveal which profile and behavioral markers precede irregular responses, and we use them to identify likely operative groups before a study begins. 

An operative group is identified by a cluster of behavioral markers that tends to precede unreliable answers. For example, aspirational answering: participants claiming ownership of luxury vehicles and financial assets at rates that outrun reality. Belonging to the group doesn’t make a participant dishonest, but they are likelier to warrant a second look. This is a less severe claim than a fraud flag makes, and a more useful one. It’s the difference between sorting and filtering.

The operative group receives the attention questions, the speed traps, the deliberately obvious tests. Everyone else simply takes the survey. This applies necessary scrutiny to irregular responses while preserving the survey experience for everyone else, as attention traps and speed checks take a real toll on a genuine participant’s experience, and they land hardest on the people least deserving of the suspicion.

Testing produces three outcomes instead of a pass or a fail. Some participants do pass outright or fail outright, but a substantial number land in between; that gray area is where the research team’s judgment matters more than ours. Each group carries a risk level, from normal to moderate to high, and those markers become part of the sampling definition itself, which means quotas can be set against them.