Why 50+1 isn't collecting "synthetic polls"

AI-generated "polls" are on the rise

Last week, Axios reported on a new maternal-health survey commissioned by the organization Heartland Forward. The poll found that “nearly 9 in 10 Americans (88%) view maternal mortality as a serious problem in the United States. Yet fewer than half (43.6%) understand the U.S. has a higher maternal mortality rate than peer nations.”

Axios’ write-up of the survey looks like any other poll release. The bottom of the “poll” release, however, looks like this:

The survey is not a set of interviews with humans, explains Aaru, the company that conducted the “poll,” but instead based on interviews with artificial “agents” — computer programs that simulate humans by inputting demographic and other data into a series of artificial intelligence and machine learning systems, often referred to as “synthetic data.”

This is not an isolated case. In fact, it’s not even the only case this week! Here’s the methodology for a new “poll” that was released by The Public Sentiment Institute from Monday, March 21, 2026:

The business of synthetic data is booming. Aaru has been pitching synthetic political research for some time; Semafor reported in 2024 on the company’s use of AI agents instead of human respondents in election-related work. Meanwhile, Qualtrics announced this month that clients can now combine synthetic and human panels on one platform. And in February, a company called Simile raised $100 million to build AI systems for predicting human behavior.

I think we are going to see a lot more of these AI-generated polls in the future. And since our job here at FiftyPlusOne.news is to aggregate polls to better estimate issue attitudes and voting behavior, we decided we should have a public policy for dealing with them. In short: 50+1 will not be using any “poll” that includes AI-generated “survey respondents” in our work. Here’s why.

First principles: a poll measures opinion by asking people how they feel

We see two big issues in using AI-generated survey results: one based in theory about what polling is for, and one statistical.

When it comes to the purpose of polling, the core issue with AI-generated data is simple: polls are supposed to be a way of talking to people not predictions of what people would say by algorithmic systems, AI-generated or otherwise.

In 2022 I wrote a book about how polls work and why they are important to the democratic process. A “poll” is something that measures public opinion by interviewing people selected to represent a population. That is the basic logic of survey research, and it remains the standard account of how opinion polling works. When George Gallup was developing some of the foundational ideas about political polling, he intended polls to “take the pulse of democracy.” Surveys, from the democratic point of view (note lower-case “d”), are just another way of talking to the voters. The political scientist Sidney Verba used to say that polls do what democracy is supposed to do in its most ideal form, giving everyone an equal chance to participate in the democratic process.

Synthetic systems do something different. They do not measure opinion by asking people what they think. They infer what people would probably say from demographic profiles, training data, prior surveys, text corpora, behavioral signals, or other model inputs. That sounds fancy, but primarily the way this happens is a researcher tells a large language model to act like a person given some set of demographic variables. What would you say in response to X question if you were a white, 30-44 year old man without a college education that lived in Virginia and made between 50,000 and 100,000 dollars per year? What if you were afraid of heights?

To be clear, such simulations are useful for many purposes. They may even produce estimates that are close to observed public opinion on one or many questions. I have written several academic reports on these approaches so I may have a better idea than most people about what’s going on under the hood here, and I think there is potential for some very accurate emulation of human behavior in the future — perhaps even the near future.

But the predictions these simulations spit out are still only model outputs, not direct measurements of how people feel. And that is why we do not consider such predictions to constitute “polls.”

Why that distinction matters for us at 50+1

The second concern we have is that aggregating predictions from outside models could violate the underlying statistical properties of polling aggregation and election forecasting.

See, polling averages are not just piles of numbers. They work because real polls share certain statistical properties, and can be “combined” using statistical methodologies that detect trends in noisy data. The most fundamental property of these models is that each survey is based on a finite set of actual respondents — and so comes with a measurable level of uncertainty to communicate to the model. On top of that, different pollsters make different choices about sampling, fieldwork, weighting, mode, screening, and question wording that can cause additional bias and error. Averaging helps reduce some noise between pollsters because the mistakes are not perfectly shared across firms.

Synthetic outputs, however, do not have the same type of error structure. If no humans were interviewed, there is no respondent-level sampling error in the usual sense. You can generate 500 synthetic respondents or 50,000, but that does not create new information about what the public thinks. It just produces more draws from the same underlying model. Evidence suggests simulated respondents tend to cluster around an answer more than human respondents, creating both high bias and low variance. (This is not something you want in a polling average.)

Just as important, synthetic estimates are likely to be highly correlated with one another — and in some cases with the polling ecosystem more broadly — because they often rely on overlapping sources of information. They may draw on similar priors, similar public text, similar demographic assumptions, similar media signals, and sometimes even prior polls themselves. If synthetic data is based on polls, and polls are based on synthetic data, then we enter a feedback loop where actual public opinion ends up lost.

That means adding synthetic outputs to a polling average would not diversify error in the way that combining independent surveys can. It would risk hard-coding model-based bias into the average. And our models wouldn’t know how much to update underlying trends in opinion given this new data, because they don’t know how much information these new data points contain.

You don’t have to take my word for it. In their 2024 Political Analysis paper, Synthetic Replacements for Human Survey Data? The Perils of Large Language Models, Joshua Clinton and Jennifer Larson find that while ChatGPT can sometimes reproduce broad averages, it is “not reliable for statistical inference”: the responses show too little variation relative to real surveys, and the estimated relationships often differ substantively from those found in the American National Election Study, a large, annual gold-standard survey of American voters.

A newer February 2026 paper, Do LLMs Track Public Opinion?, reaches a similarly skeptical conclusion. Using daily LLM queries benchmarked against high-quality 2024 election-cycle polls, the authors find “systematic directional miscalibration.” In their headline result, every model overpredicted Kamala Harris’s favorability — often by large margins.

Our policy

So our policy is straightforward. FiftyPlusOne will not collect or aggregate synthetic data in our polling averages or election forecasts. If a source does not interview real human beings, it is not a poll for our purposes.

In addition, we will not aggregate predictions generated by AI-augmented approaches, such as Aaru’s and others, and those based purely on statistical models. This policy applies to fully synthetic systems and to hybrid designs that blend a human interviews with model-generated respondents.

We care about what Americans think, because our government is intended to serve the people. Until robots get the franchise, it’ll be all human at 50+1.

---

Paid subscribers to 50+1 get access to premium analysis, plus sortable tables and complete data access on our polling website. If you want to follow the 2026 cycle with the best data at your fingertips, become a paid subscriber.