Online opt-in polls can produce misleading results

In this analysis, we’ll show how some recent findings from an opt-in poll may have unintentionally misled the public about the sensitive issue of Holocaust denial among young Americans.

Online Opt-in polls can struggle to measure the attitudes of young people, Hispanics.

Online opt-in polls have become increasingly popular. And for some purposes, such as election polling, they can perform similarly to more traditional survey approaches.

There is evidence, however, that the online environment in which they operate is somewhat unstable.

In particular, several recent studies have documented large errors in online opt-in surveys due to the presence of so-called “bogus respondents.” These respondents do not answer questions sincerely; Instead, they attempt to complete surveys with as little effort as possible to earn money or other rewards.

Studies have shown that bogus respondents can cause opt-in surveys to overestimate rare attitudes and behaviors, such as ingesting bleach to protect against COVID-19, belief in conspiracies like Pizzagate or support for political violence.

At Pew Research Center, we’ve found that this type of overreporting tends to be especially concentrated in estimates for adults under 30, as well as Hispanic adults. Bogus respondents may be identifying this way in order to bypass screening questions that might otherwise prevent them from receiving a reward, although the precise reasons are difficult to pin down. Whatever the underlying cause, the result can be unreliable estimates for those groups.

For example, in a February 2022 survey experiment, we asked opt-in respondents if they were licensed to operate a class SSGN (nuclear) submarine. In the opt-in survey, 12% of adults under 30 claimed this qualification, significantly higher than the share among older respondents. In reality, the share of Americans with this type of submarine license rounds to 0%.

The problem was even worse for Hispanic estimates. About a quarter (24%) of opt-in cases claiming to be Hispanic said they were licensed to operate a nuclear sub, versus 2% of non-Hispanics.

Another Pew Research Center study comparing three online opt-in samples and three probability-based panels had similar findings, though on different topics. We asked respondents 16 yes/no questions on topics ranging from smoking and hypertension to collecting Social Security and receiving workers’ compensation. Young and/or Hispanic opt-in respondents were significantly more likely than other respondents to answer “yes” to 10 or more of these questions, claiming combinations of characteristics that are virtually nonexistent in reality. Critically, on the most rigorous probability-based surveys, very few respondents in any age group gave the same kind of implausible answers to these questions.

Recent online opt-in poll made headlines on Holocaust denial among young adults

In the context of what we’ve learned from our investigations into opt-in polls, we took particular notice of a recent online opt-in survey that had a starting finding about Holocaust denial among young Americans. The survey, fielded in December 2023, reported that 20% of U.S. adults under 30 agree with the statement, “The Holocaust is a myth.” This alarming finding received widespread attention from the news media and on social networks.

From a survey science perspective, the finding deserved a closer look. It raised both of the red flags in the research literature about bogus respondents: It focused on a rare attitude (Holocaust denial), and it involved a subgroup frequently “infiltrated” by bogus respondents (young adults).

Other questions asked in that December opt-in poll also pointed to a need for scrutiny. In the same poll, about half of adults under 30 (48%) expressed opposition to legal abortion. This result is dramatically at odds with rigorous polling from multiple survey organizations that consistently finds the rate of opposition among young adults to be much lower.

In an April 2023 Pew Research Center survey, for instance, 26% of U.S. adults under 30 said abortion should be illegal in all or most cases. This was 13 points lower than the share among older Americans (39%). Our estimate for young adults was similar to ones from other, more recent probability-based surveys, such as an AP-NORC survey from June 2023 (27%) and a KFF survey from November 2023 (28%).