In the last month, we’ve seen an odd thing happen a couple of times: Candidates for federal office have been conducting surveys of their races independently, without the involvement of a professional pollster. The first such surveys we saw came from Peter Crosby, the Democratic candidate in Utah’s 2nd Congressional District. At the time, I thought it was unusual, so I reached out to the Crosby campaign for more information.
Crosby replied to me himself. This is unusual, as when we reach out to campaigns for polling methods information here at FiftyPlusOne, we usually hear back from the campaign’s communications team or campaign manager. Crosby explained that he has worked in survey research professionally for years, and provided me with some details about his methodology. His response made clear to me that he was an experienced pollster, and FiftyPlusOne aggregated the poll as we would any other campaign internal.
But then this Monday, October 5, we discovered a similar thing from another Utah Democrat: Jonny Larsen, candidate for Utah’s 4th Congressional District, also released a poll he conducted himself that raised some red flags for me. First of all, the initial ballot in the survey showed Larsen ahead of incumbent Republican Mike Kennedy by over 20 percentage points, wildly different from the only independent poll of the race (conducted on behalf of the Utah Debate Commission) which showed Larsen trailing by 9 points. The result from Larsen was highly improbable in a district Trump won by over 30 points in 2024.
Now at FiftyPlusOne, we generally don’t dismiss polls just because the results don’t align with our priors — but large differences often indicate to us that something is awry with a survey. When I looked deeper, a few things stuck out as strange. For example, in the methodology statement, rather than providing sample sizes for various demographic groups, the campaign included information about how much each group was weighted, which tells us very little about the nature of the sample (knowing that Democrats were weighted at 0.81 does not tell me how many Democrats were in your survey). The release said to treat the poll like “a focus group,” which is confusing, since these are two very different kinds of instruments for measuring public opinion.
All this sparked some internal discussion about how we should treat these kinds of surveys. On one hand, we have at least one competent pollster who also happens to be a candidate for office. On the other hand, due to easier access to technology for conducting polls, pretty much anybody who has $10,000 could get their hands on a voter file and start sending out text messages, yielding vast quantities of iffy data.
Ultimately, though, the more important issue is about accountability. When a professional pollster puts out surveys that are extremely inaccurate, it has consequences. They may lose clients or take reputational damage that could impact their careers. Even when conducting campaign internals, pollsters know that their reputation as a reliable and accurate source of data is on the line. And while internal polls have been shown to systematically overstate their candidate’s chances by around 4 or 5 points, that is a predictable bias that we can (and do) account for in our polling averages and models.
For candidates, no such incentives for accuracy exist. They can literally just say anything, and if they lose their race, they can go back to their regular job (that probably doesn’t include public opinion research) with no consequences. There is no reason for a candidate to produce accurate data, and inaccurate data might actually help them in some ways — perhaps by getting their name in the news, or improving fundraising, for example.
And this, poll lovers, is the crux of the problem for us. Because there are no systematic incentives for candidates to produce accurate results, FiftyPlusOne will no longer aggregate internal campaign polls that were conducted without the involvement of a professional pollster. Because this rule was not in place at the time of the survey releases, we will keep Crosby’s previous polls in our database and up on our site, but we will not publish any additional research conducted by the campaign, and we will not aggregate the survey from Jonny Larsen’s campaign. This change can now be found in our Methodology Statement and will apply to all future poll releases from candidates for office.
You didn't mention how Peter Crosby's polls compare with independent polls (if there are any) and expectations based on prior elections.
The accuracy or inaccuracy of an internal poll may depend on the candidate's motives for doing it. With his experience in polling Peter Crosby may have been trying to be as accurate as possible so that he can use his polls to direct and fashion advertising, to determine where best to hold in-person events, and to determine where to concentrate on-the-ground efforts.
Although it seems very reasonable to adjust polls for reliability, I have for a while seen eg how Nate Silver uses a very complicated pollster rating system, that,. as compared to say RCP, tends to push his polls to a stronger Dem bias than RCP which has a milder one.
It is not that the methodology is not reasonable, but whatever phenomenon has pushed the polls to have anywhere from mild to moderate Dem bias since at least 2016 whether shy Trump effect or Republican non responders, rating and adjusting the pollsters only seems to makes it worse and why eg RCP has been more accurate than Silver and also Sabato who doesn't aggregate but somehow comes out with a higher over estimation of Dem prospects.
Typically Silver's aggregations were more favorable to the Dems than RCP and yet in the actual elections even RCP had a slight Dem bias. For example just looking at the congressional generic comparing the final polls with the actual national congressional vote, RCP since 2002, has overestimated the Dems in 9 of 12 elections with an average bias of 1.4%. Silver was off by 2.5%. The phenomenon holds for Presidential and Senate.