Study finds key flaws in Twitter's Birdwatch fact-checking program

Less than a month after Twitter started its Birdwatch fact-checking program, a study has unearthed some troubling issues with the crowdsourced initiative. The Poynter Institute for Media Studies analyzed almost 2,700 contributions and found that less than half of the program's participants submitted reliable sources and citations to back up their fact checks. The institute also determined that there's a partisan slant to some of the submitted notes.

Birdwatch, which is currently a pilot project, has around 1,000 users. They can flag tweets as misleading and, in a note, they can cite a source that disputes claims made in a tweet or explain why it may not be accurate. The notes are displayed under the tweet and other Birdwatch users can rate how helpful they are. They can be filtered by recency or helpfulness.

Among the notes that Poynter analyzed earlier this week, only around seven percent were rated as helpful. More than a third of those have source links that are just another tweet or nothing at all. The cited tweets may also be inaccurate, which amplifies the problem.

The report cites a tweet from YouTuber Tim Pool as a catalyst for a change in the algorithm. Pool referenced a Time article that suggested the presidential election was “fortified” by “a cabal of elites” who allegedly changed rules and laws and manipulated "the flow of information."

Twitter added a label to the tweet that reads, "This claim of election fraud is disputed." It turned off engagement on the tweet "due to a risk of violence." Some Birdwatch users added misleading notes to the tweet. One reads, "According to the officiating [sic] source of Time there was a well organized group of secret participants in a shadow organization that sounds like a cabal that worked together to sway the election in favor of Joe Biden.” Other users rated such notes as helpful. Twitter this week removed the "helpful" label from the notes and tweaked the Birdwatch algorithm.

A note now needs at least five ratings (up from three) before the “rated helpful” label is applied. Twitter also increased the helpfulness rating threshold from 0.5 to 0.84. Under five percent of Birdwatch notes meet the updated requirements for the label, according to Poynter, and three quarters of those link to a source away from Twitter.

The institute also found that there's a degree of partisanship among Birdhouse participants. "A majority of the most prolific Birdwatch user’s notes mark tweets critical of the right as 'misleading' and those critical of the left as 'not misleading,'" according to the report. It determined that the five most active users account for more than a tenth of Birdhouse notes and four of them "have similar activity as the most prolific user." Just one of the top five users, the second most-frequent contributor, includes a corroborating source link with all of their notes.

The Twitter handles of contributors are included alongside their notes, as Poynter's Alex Mahadevan points out, which could make them a target of harassment or abuse. Participants can register for Birdwatch under a pseudonym, according to Twitter.

Birdwatch is a work in progress, but fact-checking isn't easy. Leading fact-checkers suggested to Poynter that Twitter should offer training and incentives to Birdwatch users while employing professionals to assess high-ranking notes.

Twitter's vice president of product, Keith Coleman, told the publication that the company doesn't have a timeline for rolling out Birdwatch more broadly and that it's "working to learn as much as possible and iterate while the pilot is small." He noted that Twitter values openness in terms of who can contribute to the project and that "through input from a diverse group, the most helpful notes can be elevated."

Correction 2/22 9:35AM ET: We've added additional context around Tim Pool's tweet. A previous version made it sound as if Pool himself claimed the election was rigged. We apologize for the error.