Events

Mr. Yu Ding
Ph.D. candidate in Marketing
Columbia Business School

The veracity of information is of critical importance given the explosion of news transmitted to, and shared by consumers across different media. However, the scale of existing fact-checking organizations is limited, resulting in a very small proportion of news articles being fact-checked. We address the challenge of scaling up fact-checking operations in the domain of science-related articles by proposing and testing a novel crowdsourcing solution. A big challenge with asking lay consumers to rate the credibility of scientific news articles is that they are likely to be biased by their prior beliefs. We overcome this bias by proposing the use of article similarity ratings rather than credibility ratings, using articles that have been rated for veracity by scientists as a starting point. We find that asking lay consumers to rate the similarity between scientist-rated and unrated articles can provide an unbiased, effective, and efficient way to scale up veracity ratings of scientific articles. Our proposed method (human similarity-judgments) outperforms algorithm-rated similarity (e.g., by TF-IDF and by Word Embedding) to more accurately predict an article’s scientific veracity. Our method also outperforms previous approaches to judging veracity such as using algorithms that detect semantic markers of false news. We compute a “transitivity index” to identify consumers who are likely to be more accurate at making similarity judgments and show how the veracity predictions can be improved by paying close attention to the consumer segments recruited for the similarity-judgment task. We demonstrate that our method can predict the scientific veracity of articles with over 95% accuracy and that both type-I and type-II errors are minimized. Involving consumers in fact-checking operations can not only help scale up these operations, but can also increase consumer trust in the media ecosystem driven by consumer involvement.