Copilot leaned on an error-prone AI voter site in 75% of Post tests

Chatbots asked about the US midterm elections rely heavily on three obscure voter-information sites that were themselves built with AI, according to a Washington Post test summarized by Techmeme. The paper reported that Microsoft’s Copilot cited one of them, Ballot Brief, in 75% of tests, and ChatGPT in 22%.

The Post describes Ballot Brief as error-prone. The site’s creator, per the Post’s account, started it out of frustration at how hard it is to find information on political candidates, and aimed to give voters a single place to look. We could not find a primary source for the study, such as a methodology page or a company statement, so everything below rests on the Post’s reporting and on secondary write-ups of it. The full article was not available to us.

What the tests reportedly found

Per the headline and feed summary, the Post’s tests showed three small sites feeding answers to the big assistants. The 75% and 22% figures are the share of tests in which each chatbot cited Ballot Brief.

A separate summary by Let’s Data Science says the Post tested six popular chatbots by asking them to build voter guides for addresses across the country, and analyzed more than 3,000 of the results. It reports that many errors traced back to sites that appeared to be partly AI-produced. It also says the chatbots missed congressional districts in 48 percent of cases, and that the guides included a nonexistent Missouri Senate race and a Washington House candidate who is not on the ballot. These details come from that secondary write-up, not from a document we read from the Post.

What the companies and the site’s founder say

According to the same reports, Ballot Brief’s founder says the site draws on official government sources and lets users enter an address for fuller candidate information. Microsoft said it was looking into the results and that it is “committed to ensuring that Copilot provides accurate, high authority election information for its users.” The reports add that OpenAI and Amazon appeared to change how their chatbots respond to voter-guide requests after the Post contacted them. We have not verified that independently.

Why a citation count matters for voters

A citation tells readers where an answer came from, and it can make a weak source look like an authoritative one. If a chatbot pulls a district or a candidate list from a site that gets either wrong, the user sees a confident, linked answer rather than an obvious error.

The scale of use raises the stakes. NPR reported in October that voters are increasingly asking chatbots to compare candidates and check facts. A September survey by the American Association of Political Consultants Foundation, cited in the search coverage we read, found about one in five voters consult AI chatbots for election news and voting guidance.

What the reporting does not establish

The public summaries do not say how many prompts made up the 75% and 22% figures, which chatbots besides Copilot and ChatGPT were measured on this metric, or how the Post judged a site to be AI-built. They also do not show whether the chatbots’ errors came from these sites or from the models themselves. Ballot Brief’s founder disputes the characterization of its accuracy only in the general terms above. Voters who use an assistant for ballot information can check the answers against their state or county election office, which is the kind of official source the site’s founder says Ballot Brief draws on.

For more on how AI systems can be steered by what they read, see our report on a ChatGPT share-link flaw.