Analyzing the Veracity of Tweets during a Major Crisis

A research team at Yahoo recently completed an empirical study (PDF) on the behavior of Twitter users after the 8.8 magnitude earthquake in Chile. The study was based on 4,727,524 indexed tweets, about 20% of which were replies to other tweets. What is particularly interesting about this study is that the team also analyzed the spread of false rumors and confirmed news that were disseminated on Twitter.

The authors “manually selected some relevant cases of valid news items, which were confirmed at some point by reliable sources.” In addition, they “manually selected important cases of baseless rumors which emerged during the crisis (confirmed to be false at some point).” Their goal was to determine whether users interacted differently when faced with valid news vs false rumors.

The study shows that about 95% of tweets related to confirmed reports validated that information. In contrast only 0.03% of tweets denied the validity of these true cases. Interestingly, the results also show  that “the number of tweets that deny information becomes much larger when the information corresponds to a false rumor.” In fact, about 50% of tweets will deny the validity of false reports. The table below lists the full results.

The authors conclude that “the propagation of tweets that correspond to rumors differs from tweets that spread news because rumors tend to be questioned more than news by the Twitter community. Notice that this fact suggests that the Twitter community works like a collaborative filter of information. This result suggests also a very promising research line: it could posible to detect rumors by using aggregate analysis on tweets.”

I think these findings are particularly important for projects like *Swift River, which try to validate crowdsourced crisis information in real-time. I would also be interested to see a similar study on tweets around the Haitian earthquake to explore whether this “collaborative filter” dynamic is an emergent phenomena in this complex systems or simply an artifact of something else.

Interested in learning more about “information forensics”? See this link and the articles below:

28 responses to “Analyzing the Veracity of Tweets during a Major Crisis

  1. This report suggests that “Truthiness” has some natural buoyancy, that the Truth “floats to the top” on Twitter — and I am really happy to hear this. It’s a fantastic thing that type of noise is dampened by the crowd. This confirms my anecdotal observations. I remember in the Mumbai attacks in 2008 as the first time that Twitter got serious about policing rumors. It would be great to see these results for that data set as well.

    As we know, an organic rumor could be extremely dangerous, leading for instance to the death of first responders, the expulsion or aid workers, or the neglect of victims. I think about an instance where a rumor, “the aid worker poisoned the wells” would cause someone to get murdered, or provoke a riot.

    So I am cheered to see that Truth seems to “float more” on Twitter.

    But an intentional rumor, propagated by one or more bad actors, would be a much more dangerous case, and this is not covered in these data sets. The consequences of a rumor, even if it is not widely entertain as truth, could be truly catastrophic under the wrong circumstances. The rumors from Chile were criticized on Twitter (hooray!) but they are also serving as a blueprint for a first generation of network-mediated crisis exploitation (oh no!) Note that we have (only since January 2010) really begun to see the emergence of sophisticated scam artists in ad hoc digital crisis networks. So I think we have many worse situations to come than merely dealing with

    I am warily watching to see what happens in the next few years. Our challenge socially, I think, is to be more engaged as critics of the news, and to invest now in infrastructure (filters, algorithms) and training (eg. via local emergency preparation groups). Regarding infrastructure, the authors of the paper conclude with a great recommendation for an infrastructural improvement in our ability to critique in this sense — a Twitter client could (and should!) implement this today:

    “As an application, given that detecting when a tweet is asking for information … could for instance warn people that many other users are questioning the information item they are reading. This would provide signals for users to determine how much to trust a certain piece of information.”

    Swift River is directly addressing this from an external perspective, but something native to Twitter could help prevent rumors, not just spot them.

    And still we need better information literacy — more than any technological solution.

    My fear is that even a single high profile failure will damage “the reputation of the crowd” in a way that will have a strong chilling effect on the incredible experimentation and energy that is happening presently. I do not want to look back on this as a lost golden age of crisis mapping — we are only barely getting started!

    Toward this goal, a close examination of the Chilean dynamics is really valuable — we just need to keep it in perspective as a snapshot of the early heydays.

    • Hiya Chris, you make some stellar points, as always. I share your concern re:

      “…a single high profile failure will damage “the reputation of the crowd” in a way that will have a strong chilling effect on the incredible experimentation and energy that is happening presently. I do not want to look back on this as a lost golden age of crisis mapping — we are only barely getting started!”

  2. Pingback: Top 3 FAQs for Social Media and EM Beginners « idisaster 2.0

  3. Pingback: 7 Ways to Be a Good Twitter Citizen During a Crisis | eVentures in Cyberland

  4. Maybe one day we’ll be able to govern ourselves instead of relying on centralized institutions.

  5. Pingback: Seeking the Trustworthy Tweet: Can “Tweetsourcing” Ever Fit the Needs of Humanitarian Organizations? | iRevolution

  6. Pingback: How to Verifying Social Media Content: Some Tips and Tricks on Information Forensics | iRevolution

  7. Pingback: Detecting Emerging Conflicts with Web Mining and Crisis Mapping | iRevolution

  8. What about deliberate obfuscation, which might be salient during political violence mapping?

  9. Pingback: Communicating with Disaster Affected Communities | iRevolution

  10. Pingback: Communicating with Disaster Affected Communities (CDAC) | iRevolution

  11. Reblogged this on Caledoniyya and commented:
    Measuring the immeasurable?

  12. Pingback: Truth in the Age of Social Media: A Social Computing and Big Data Challenge | iRevolution

  13. Pingback: Analyzing the Veracity of Tweets during a Major Crisis « Transmedia Camp 101

  14. Pingback: What Was Novel About Social Media Use During Hurricane Sandy? | iRevolution

  15. Pingback: Predicting the Credibility of Disaster Tweets Automatically | iRevolution

  16. Pingback: Forscher entwickeln Lügendetektor für Twitter | BASIC thinking

  17. Pingback: Parenting: Just let the Robots handle it. « With a Wiffle Ball Bat

  18. Pingback: Forscher entwickeln Lügendetektor für Twitter - YEPA NEWS

  19. Pingback: Desarrollan nuevo sistema para descubrir veracidad de tuits en Twitter | La mirada de José Altimiras Lampré

  20. Pingback: How To Tell If A Tweet Is Credible - AllTwitter

  21. Pingback: An Algorithm to Limit Inaccurate Tweets : Stephen E. Arnold @ Beyond Search

  22. Pingback: Building a Better Truth Machine | ForensicsPress.com

  23. Pingback: Using Crowdsourcing to Counter the Spread of False Rumors on Social Media During Crises | iRevolution

  24. Pingback: Twitter Under Crisis: Can we trust what we RT? | kredible.net

  25. Pingback: Twitter no volverá a matar - Aurora Ferrer | Aurora Ferrer

  26. Pingback: Crowd reporting puts the squeeze on traditional journalism - Social Media - English - DW.DE

  27. Pingback: The Good, Bad, and In-Between of Social Media In Crisis Situations | iStrategyLabs – A Digital Agency That Invents Solutions Online and Off

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s