The Daily Mail Snopes Story And Fact Checking The Fact Checkers
Letters arranged to spell "facts." (Shutterstock)
When I first read through the Daily Mail article I immediately suspected the story itself must certainly be “fake news” because of how devastating the claims were and that given that Snopes.com was so heavily used by the journalistic community, if any of the claims were true, someone would have already written about them and companies like Facebook would not be partnering with them. I also noted that despite having been online for several hours, no other major mainstream news outlet had written about the story, which is typically a strong sign of a false or misleading story. Yet at the same time, the Daily Mail appeared to be sourcing its claims from a series of emails and other documents from a court case, some of which it reproduced in its article and, perhaps most strangely, neither Snopes nor its principles had issued any kind of statement through its website or social media channels disclaiming the story.
On the surface this looked like a classic case of fake news – a scandalous and highly shareable story, incorporating official-looking materials and sourcing, yet with no other mainstream outlet even mentioning the story. I myself told my colleague I simply did not know what to think. Was this a complete fabrication by a disgruntled target of Snopes or was this really an explosive expose pulling back the curtain on one of the world’s most respected and famous fact checking brands?
In fact, one of my first thoughts upon reading the article is that this is precisely how the “fake news” community would fight back against fact checking – by running a drip-drip of fake or misleading explosive stories to discredit and cast doubt upon the fact checkers.
In the counter-intelligence world, this is what is known as a “wilderness of mirrors” – creating a chaotic information environment that so perfectly blends truth, half-truth and fiction that even the best can no longer tell what’s real and what’s not.
Recommended by Forbes
It was with incredible surprise therefore that I received David’s one-sentence response which read in its entirety “I'd be happy to speak with you, but I can only address some aspects in general because I'm precluded by the terms of a binding settlement agreement from discussing details of my divorce.”
This absolutely astounded me. Here was the one of the world’s most respected fact checking organizations, soon to be an ultimate arbitrator of “truth” on Facebook, saying that it cannot respond to a fact checking request because of a secrecy agreement.
In short, when someone attempted to fact check the fact checker, the response was the equivalent of “its secret.”
It is impossible to understate how antithetical this is to the fact checking world, in which absolute openness and transparency are necessary prerequisites for trust. How can fact checking organizations like Snopes expect the public to place trust in them if when they themselves are called into question, their response is that they can’t respond.
When I presented a set of subsequent clarifying questions to David, he provided responses to some and not to others. Of particular interest, when pressed about claims by the Daily Mail that at least one Snopes employee has actually run for political office and that this presents at the very least the appearance of potential bias in Snopes’ fact checks, David responded “It's pretty much a given that anyone who has ever run for (or held) a political office did so under some form of party affiliation and said something critical about their opponent(s) and/or other politicians at some point. Does that mean anyone who has ever run for office is manifestly unsuited to be associated with a fact-checking endeavor, in any capacity?”
That is actually a fascinating response to come from a fact checking organization that prides itself on its claimed neutrality. Think about it this way – what if there was a fact checking organization whose fact checkers were all drawn from the ranks of Breitbart and Infowars? Most liberals would likely dismiss such an organization as partisan and biased. Similarly, an organization whose fact checkers were all drawn from Occupy Democrats and Huffington Post might be dismissed by conservatives as partisan and biased. In fact, when I asked several colleagues for their thoughts on this issue this morning, the unanimous response back was that people with strong self-declared political leanings on either side should not be a part of a fact checking organization and all had incorrectly assumed that Snopes would have felt the same way and had a blanket policy against placing partisan individuals as fact checkers.
In fact, this is one of the reasons that fact checking organizations must be transparent and open. If an organization like Snopes feels it is ok to hire partisan employees who have run for public office on behalf of a particular political party and employ them as fact checkers where they have a high likelihood of being asked to weigh in on material aligned with or contrary to their views, how can they reasonably be expected to act as neutral arbitrators of the truth?
Put another way, some Republicans believe firmly that climate change is a falsehood and that humans are not responsible in any way for climatic change. Those in the scientific community might object to an anti-climate change Republican serving as a fact checker for climate change stories at Snopes and flagging every article about a new scientific study on climate change as fake news. Yet, we have no way of knowing the biases of the fact checkers at Snopes – we simply have to trust that the site’s views on what constitutes neutrality are the same as ours.
When I asked for comment on the specific detailed criteria Snopes uses to screen its applicants and decide who to hire as a fact checker, surprisingly David demurred, saying only that the site looks for applicants across all fields and skills. He specifically did not provide any detail of any kind regarding the screening process and how Snopes evaluates potential hires. David also did not respond to further emails asking whether, as part of the screening process, Snopes has applicants fact check a set of articles to evaluate their reasoning and research skills and to gain insight into their thinking process.
This was highly unexpected, as I had assumed that a fact checking site as reputable as Snopes would have a detailed written formal evaluation process for new fact checkers that would include having them perform a set of fact checks and include a lengthy set of interview questions designed to assess their ability to identify potential or perceived conflicts of interest and work through potential biases.
Even more strangely, despite asking in two separate emails how Snopes assesses its fact checkers and whether it performs intra- and inter-rater reliability assessments, David responded only that fact checkers work together collaboratively and did not respond to further requests for more detail and did not answer whether Snopes uses any sort of assessment scoring or ongoing testing process to assess its fact checkers.
This raises exceptionally grave concerns about the internal workings of Snopes and why it is not more forthcoming about its assessment process. Arguing that because multiple fact checkers might work on an article, reliability is not a concern, is a false argument that shows a concerning lack of understanding about reliability and accuracy. Imagine a team of 50 staunch climate deniers all working collaboratively to debunk a new scientific study showing a clear link between industrial pollution and climate change. The very large team size does not make up for the lack of diversity of opinion. Yet, David provided no comment on how Snopes does or does not explicitly force diversity of opinion in its ad-hoc fact checking teams.
A robust human rating workflow must regularly assess the accuracy and reproducibility of the scores generated by its human raters, even when they work collaboratively together. Typically this means that on a regular basis each fact checker or fact checker team is given the same article to fact check and the results compared across the groups. If one person or group regularly generates different results from the others, this is then evaluated to understand why. Similarly, an individual or group is also periodically given the same or nearly identical story from months prior to see if they give it the same rating as last time – this assesses whether they are consistent in their scoring.
More troubling is that we simply don’t know who contributed to a given fact check. David noted that Snopes’ “process is a highly collaborative one in which several different people may contribute to a single article,” but that “the result is typically credited to whoever wrote the initial draft.” David did not respond to a request for comment on why Snopes only lists a single author for each of its fact checks, rather than provide an acknowledgement section that lists all of the individuals who contributed to a given fact check.
One might argue that newspapers similarly do not acknowledge their fact checkers in the bylines of articles. Yet, in a newspaper workflow, fact checking typically occurs as an editorial function, double checking what a reporter wrote. At Snopes, fact checking is the core function of an article and thus if multiple people contributed to a fact check, it is surprising that absolutely no mention is made of them, given that at a newspaper all reporters contributing to a story are listed. Not only does this rob those individuals of credit, but perhaps most critically, it makes it impossible for outside entities to audit who is contributing to what fact check and to ensure that fact checkers who self-identify as strongly supportive or against particular topics are not assigned to fact check those topics to prevent the appearance of conflicts of interest or bias.
If privacy or safety of fact checkers is a concern, the site could simply use first name and last initials or pseudonyms. Having a master list of all fact checkers contributing in any way to a given fact check would go a long way towards establishing greater transparency to the fact checking process and Snopes’ internal controls on conflict of interest and bias.
David also did not respond to a request for comment on why Snopes fact checks rarely mention that they reached out to the authors of the article being fact checked to get their side of the story. Indeed, Journalism 101 teaches you that when you write an article presenting someone or something in a negative light, you must give them the opportunity to respond and provide their side of the story. Instead, Snopes typically focuses on the events being depicted in the article and contacts individuals and entities named in the story, but Snopes fact checks typically do not mention contacting the authors of the articles about those events to see if those reporters claim to have additional corroborating material, perhaps disclosed to them off the record.
In essence, in these cases Snopes performs “fact checking from afar,” rendering judgement on news stories without giving the original reporters the opportunity for comment. David did not respond to a request for comment on this or why the site does not have a dedicated appeals page for authors of stories which Snopes has labeled false to contest that label and he also did not respond to a request to provide further detail on whether Snopes has a written formal appeals process or how it handles such requests.
Putting this all together, we simply don’t know if the Daily Mail story is completely false, completely true or somewhere in the middle. Snopes itself has not issued a formal response to the article and its founder David Mikkelson responded by email that he was unable to address many of the claims due to a confidentiality clause in his divorce settlement. This creates a deeply unsettling environment in which when one tries to fact check the fact checker, the answer is the equivalent of “its secret.” Moreover, David’s responses regarding the hiring of strongly partisan fact checkers and his lack of response on screening and assessment protocols present a deeply troubling picture of a secretive black box that acts as ultimate arbitrator of truth, yet reveals little of its inner workings. This is precisely the same approach used by Facebook for its former Trending Topics team and more recently its hate speech rules (the company did not respond to a request for comment).
From the outside, Silicon Valley looks like a gleaming tower of technological perfection. Yet, once the curtain is pulled back, we see that behind that shimmering façade is a warehouse of good old fashioned humans, subject to all the same biases and fallibility, but with their results now laundered through the sheen of computerized infallibility. Even my colleagues who work in the journalism community and by their nature skeptical, had assumed that Snopes must have rigorous screening procedures, constant inter- and intra-rater evaluations and ongoing assessments and a total transparency mandate. Yet, the truth is that we simply have no visibility into the organization’s inner workings and its founder declined to shed further light into its operations for this article.
Regardless of whether the Daily Mail article is correct in its claims about Snopes, at the least what does emerge from my exchanges with Snopes’ founder is the image of the ultimate black box presenting a gleaming veneer of ultimate arbitration of truth, yet with absolutely no insight into its inner workings. While technology pundits decry the black boxes of the algorithms that increasingly power companies like Facebook, they have forgotten that even the human-powered sites offer us little visibility into how they function.
At the end of the day, it is clear that before we rush to place fact checking organizations like Snopes in charge of arbitrating what is “truth” on Facebook, we need to have a lot more understanding of how they function internally and much greater transparency into their work.
No comments:
Post a Comment