Twitcident: Filtering Tweets in Real-Time for Crisis Response

The most recent newcomer to the “tweetsourcing” space comes to us from Delft University of Technology in the Netherlands. Twitcident is a web-based filtering system that extracts crisis information from Twitter in real-time to support emergency response efforts. Dutch emergency services have been testing the platform over the past 10 months and results “show the system to be far more useful than simple keyword searching of a twitter feed” (NewScientist).

Here’s how it works. First the dashboard, which shows current events-of-interest being monitored.

Lets click on “Texas”, which produces the following page. More than 22,000 tweets potentially relate to the actual fire of interest.

This is where the filtering comes in:

The number of relevant tweets is reduced with every applied filter.

Naturally, geo-location is also an optional filter.

Twitcident also allows for various visualization options, including timelines, word clouds and charts.

The system also allows the user to view the filtered tweets on a map. The pictures and videos shared via twitter are also aggregated and viewable on dedicated tabs.

The developers of the platform have not revealed how their algorithms work but will demo the tool at the World Wide Web 2012 conference in France next week. In the meantime, here’s a graphic that summarizes the platform workflow.

I look forward to following Twitcident’s developments. I’d be particularly interested in learning more about how Dutch emergency services have been using the tool and what features they think would improve the platform’s added value.

Does the Humanitarian Industry Have a Future in The Digital Age?

I recently had the distinct honor of being on the opening plenary of the 2012 Skoll World Forum in Oxford. The panel, “Innovation in Times of Flux: Opportunities on the Heels of Crisis” was moderated by Judith Rodin, CEO of the Rockefeller Foundation. I’ve spent the past six years creating linkages between the humanitarian space and technology community, so the conversations we began during the panel prompted me to think more deeply about innovation in the humanitarian industry. Clearly, humanitarian crises have catalyzed a number of important innovations in recent years. At the same time, however, these crises extend the cracks that ultimately reveal the inadequacies of existing organiza-tions, particularly those resistant to change; and “any organization that is not changing is a battle-field monument” (While 1992).

These cracks, or gaps, are increasingly filled by disaster-affected communities themselves thanks in part to the rapid commercialization of communication technology. Question is: will the multi-billion dollar humanitarian industry change rapidly enough to avoid being left in the dustbin of history?

Crises often reveal that “existing routines are inadequate or even counter-productive [since] response will necessarily operate beyond the boundary of planned and resourced capabilities” (Leonard and Howitt 2007). More formally, “the ‘symmetry-breaking’ effects of disasters undermine linearly designed and centralized administrative activities” (Corbacioglu 2006). This may explain why “increasing attention is now paid to the capacity of disaster-affected communities to ‘bounce back’ or to recover with little or no external assistance following a disaster” (Manyena 2006).

But disaster-affected populations have always self-organized in times of crisis. Indeed, first responders are by definition those very communities affected by disasters. So local communities—rather than humanitarian professionals—save the most lives following a disaster (Gilbert 1998). Many of the needs arising after a disaster can often be met and responded to locally. One doesn’t need 10 years of work experience with the UN in Darfur or a Masters degree to know basic first aid or to pull a neighbor out of the rubble, for example. In fact, estimates suggest that “no more than 10% of survival in emergencies can be attributed to external sources of relief aid” (Hilhorst 2004).

This figure may be higher today since disaster-affected communities now benefit from radically wider access to information and communication technologies (ICTs). After all, a “disaster is first of all seen as a crisis in communicating within a community—that is as a difficulty for someone to get informed and to inform other people” (Gilbert 1998). This communication challenge is far less acute today because disaster-affected communities are increasingly digital, and thus more and more the primary source of information communicated following a crisis. Of course, these communities were always sources of information but being a source in an analog world is fundamentally different than being a source of information in the digital age. The difference between “read-only” versus “read-write” comes to mind as an analogy. And so, while humanitarian organiza-tions typically faced a vacuum of information following sudden onset disasters—limited situational awareness that could only be filled by humanitarians on the ground or via established news organizations—one of the major challenges today is the Big Data produced by disaster-affected communities themselves.

Indeed, vacuums are not empty and local communities are not invisible. One could say that disaster-affected communities are joining the quantified self (QS) movement given that they are increasingly quantifying themselves. If inform-ation is power, then the shift of information sourcing and sharing from the select few—the humanitarian professionals—to the masses must also engender a shift in power. Indeed, humanitarians rarely have access to exclusive information any longer. And even though affected populations are increasingly digital, some groups believe that humanitarian organizations have largely failed at commu–nicating with disaster-affected communities. (Naturally, there are important and noteworthy exceptions).

So “Will Twitter Put the UN Out of Business?” (Reuters), or will humanitarian organizations cope with these radical changes by changing themselves and reshaping their role as institutions before it’s too late? Indeed, “a business that doesn’t communicate with its customers won’t stay in business very long—it’ll soon lose track of what its clients want, and clients won’t know what products or services are on offer,” whilst other actors fill the gaps (Reuters). “In the multi-billion dollar humanitarian aid industry, relief agencies are businesses and their beneficiaries are customers. Yet many agencies have muddled along for decades with scarcely a nod towards communicating with the folks they’re supposed to be serving” (Reuters).

The music and news industries were muddling along as well for decades. Today, however, they are facing tremendous pressures and are undergoing radical structural changes—none of them by choice. Of course, it would be different if affected communities were paying for humanitarian services but how much longer do humanitarian organizations have until they feel similar pressures?

Whether humanitarian organizations like it or not, disaster affected communities will increasingly communicate their needs publicly and many will expect a response from the humanitarian industry. This survey carried out by the American Red Cross two years ago already revealed that during a crisis the majority of the public expect a response to needs they communicate via social media. Moreover, they expect this response to materialize within an hour. Humanitarian organizations simply don’t have the capacity to deal with this surge in requests for help, nor are they organizationally structured to do so. But the fact of the matter is that humanitarian organizations have never been capable of dealing with this volume of requests in the first place. So “What Good is Crowd-sourcing When Everyone Needs Help?” (Reuters). Perhaps “crowdsourcing” is finally revealing all the cracks in the system, which may not be a bad thing. Surely by now it is no longer a surprise that many people may be in need of help after a disaster, hence the importance of disaster risk reduction and preparedness.

Naturally, humanitarian organizations could very well chose to continue ignoring calls for help and decide that communicating with disaster affected communities is simply not tenable. In the analog world of the past, the humanitarian industry was protected by the fact that their “clients” did not have a voice because they could not speak out digitally. So the cracks didn’t show. Today, “many traditional humanitarian players see crowdsourcing as an unwelcome distraction at a time when they are already overwhelmed. They worry that the noise-to-signal ration is just too high” (Reuters). I think there’s an important disconnect here worth emphasizing. Crowdsourced information is simply user-generated content. If humanitarians are to ignore user-generated content, then they can forget about two-way communications with disaster-affected communities and drop all the rhetoric. On the other hand, “if aid agencies are to invest time and resources in handling torrents of crowdsourced information in disaster zones, they should be confident it’s worth their while” (Reuters).

This last comment is … rather problematic for several reasons (how’s that for being diplomatic?). First of all, this kind of statement continues to propel the myth that we the West are the rescuers and aid does not start until we arrive (Barrs 2006). Unfortunately, we rarely arrive: how many “neglected crises” and so-called “forgotten emergencies” have we failed to intervene in? This kind of mindset may explain why humanitarian interventions often have the “propensity to follow a paternalistic mode that can lead to a skewing of activities towards supply rather than demand” and towards informing at the expense of listening (Manyena 2006).

Secondly, the assumption that crowdsourced data would be for the exclusive purpose of the humanitarian cavalry is somewhat arrogant and ignores the reality that local communities are by definition the first responders in a crisis. Disaster-affected communities (and Diasporas) are already collecting (and yes crowdsourcing) information to create their own crisis maps in times of need as a forthcoming report shows. And they’ll keep doing this whether or not humanita-rian organizations approve or leverage that information. As my colleague Tim McNamara has noted “Crisis mapping is not simply a technological shift, it is also a process of rapid decentralization of power. With extremely low barriers to entry, many new entrants are appearing in the fields of emergency and disaster response. They are ignoring the traditional hierarchies, because the new entrants perceive that there is something that they can do which benefits others.”

Thirdly, humanitarian organizations are far more open to using free and open source software than they were just two years ago. So the resources required to monitor and map crowdsourced information need not break the bank. Indeed, the Syria Crisis Map uses a free and open source data-mining platform called HealthMap, which has been monitoring some 2,000 English-based sources on a daily basis for months. The technology powering the map itself, Ushahidi, is also free and open source. Moreover, the team behind the project is comprised of just a handful of volunteers doing this in their own free time (for almost an entire year now). And as a result of this initiative, I am collaborating with a colleague from UNDP to pilot HealthMap’s data mining feature for conflict monitoring and peacebuilding purposes.

Fourth, other than UN Global Pulse, humanitarian agencies are not investing time and resources to manage Big (Crisis) Data. Why? Because they have neither the time nor the know-how. To this end, they are starting to “outsource” and indeed “crowdsource” these tasks—just as private sector businesses have been doing for years in order to extend their reach. Anyone actually familiar with this space and developments since Haiti already knows this. The CrisisMappers Network, Standby Volunteer Task Force (SBTF), Humanitarian OpenStreetMap (HOT) and Crisis Commons (CC) are four volunteer/technical networks that have already collaborated actively with a number of humanitarian organizations since Haiti to provide the “surge capacity” requested by the latter; this includes UN OCHA in Libya and Colombia, UNHCR in Somalia and WHO in Libya, to name a few. In fact, these groups even have their own acronym: Volunteer & Technical Communities (V&TCs).

As the former head of OCHA’s Information Services Section (ISS) noted after the SBTF launched the Libya Crisis Map, “Your efforts at tackling a difficult problem have definitely reduced the information overload; sorting through the multitude of signals on the crisis is not easy task” (March 8, 2011). Furthermore, the crowdsourced social media information mapped on the Libya Crisis Map was integrated into official UN OCHA information products. I dare say activating the SBTF was worth OCHA’s while. And it cost the UN a grand total of $0 to benefit from this support.

Credit: Chris Bow

The rapid rise of V&TC’s has catalyzed the launch of the Digital Humanitarian Network (DHN), formerly called the Humanitarian Standby Task Force (H-SBTF). Digital Humanitarians is a network-of-network catalyzed by the UN and comprising some of the most active members of the volunteer & technical co-mmunity. The purpose of the Digital Humanitarian platform (powered by Ning) is to provide a dedicated interface for traditional humanitarian organizations to outsource and crowdsource important information management tasks during and in-between crises. OCHA has also launched the Communities of Interest (COIs) platform to further leverage volunteer engagement in other areas of humanitarian response.

These are not isolated efforts. During the massive Russian fires of 2010, volunteers launched their own citizen-based disaster response agency that was seen by many as more visible and effective than the Kremlin’s response. Back in Egypt, volunteers used IntaFeen.com to crowdsource and coordinate their own humanitarian convoys to Libya, for example. The company LinkedIn has also taken innovative steps to enable the matching of volunteers with various needs. They recently added a “Volunteer and Causes” field to its member profile page, which is now available to 150 million LinkedIn users worldwide. Sparked.com is yet another group engaged in matching volunteers with needs. The company is the world’s first micro-volunteering network, sending challenges to registered volunteers that are targeted to their skill set and the causes that they are most passionate about.

It is not farfetched to envisage how these technologies could be repurposed or simply applied to facilitate and streamline volunteer management following a disaster. Indeed, researchers at the University of Queensland in Australia have already developed a new smart phone app to help mobilize and coordinate volunteer efforts during and following major disasters. The app not only provides information on preparedness but also gives real-time updates on volunteering opportunities by local area. For example, volunteers can register for a variety of tasks including community response to extreme weather events.

Meanwhile, the American Red Cross just launched a Digital Operations Center in partnership with Dell Labs, which allows them to leverage digital volunteers and Dell’s social media monitoring platforms to reduce the noise-to-signal ratio. This is a novel “social media-based operation devoted to humanitarian relief, demonstrating the growing importance of social media in emergency situations.” As part of this center, the Red Cross also “announced a Digital Volunteer program to help respond to question from and provide information to the public during disasters.”

While important challenges do exist, there are many positive externalities to leveraging digital volunteers. As deputy high commissioner of UNHCR noted about this UNHCR-volunteer project in Somalia, these types of projects create more citizen-engagement and raises awareness of humanitarian organizations and projects. This in part explains why UNHCR wants more, not less, engage-ment with digital volunteers. Indeed, these volunteers also develop important skills that will be increasingly sought after by humanitarian organizations recruit-ing for junior full-time positions. Humanitarian organizations are likely to be come smarter and more up to speed on humanitarian technologies and digital humanitarian skills as a result. This change should be embraced.

So given the rise of “self-quantified” disaster-affected communities and digitally empowered volunteer communities, is there a future for traditional humani-tarian organizations? Of course, anyone who suggests otherwise is seriously misguided and out of touch with innovation in the humanitarian space. Twitter will not put the UN out of business. Humanitarian organizations will continue to play some very important roles, especially those relating to logistics and coor-dination. These organizations will continue outsourcing some roles but will also take on some new roles. The issue here is simply one of comparative advantage. Humanitarian organizations used to have a comparative advantage in some areas, but this has shifted for all the reasons described above. So outsourcing in some cases makes perfect sense.

Interestingly, organizations like UN OCHA are also changing some of their own internal information management processes as a result of their collaboration with volunteer networks like the SBTF, which they expect will lead to a number of efficiency gains. Furthermore, OCHA is behind the Digital Humanitarians initiative and has also been developing a check-in app for humanitarian pro-fessionals to use in disaster response—clear signs of innovation and change. Meanwhile, the UK’s Department for International Development (DfID) has just launched a $75+ million fund to leverage new technologies in support of humani-tarian response; this includes mobile phones, satellite imagery, Twitter as well as other social media technologies, digital mapping and gaming technologies. Given that crisis mapping integrates these new technologies and has been at the cutting edge of innovation in the humanitarian space, I’ve invited DfID to participate in this year’s International Conference on Crisis Mapping (ICCM 2012).

In conclusion, and as argued two years ago, the humanitarian industry is shifting towards a more multi-polar system. The rise of new actors, from digitally empowered disaster-affected communities to digital volunteer networks, has been driven by the rapid commercialization of communication technology—particularly the mobile phone and social networking platforms. These trends are unlikely to change soon and crises will continue to spur innovations in this space. This does not mean that traditional humanitarian organizations are becoming obsolete. Their roles are simply changing and this change is proof that they are not battlefield monuments. Of course, only time will tell whether they change fast enough.

Crowdsourcing Humanitarian Convoys in Libya

Many activists in Egypt donated food and medical supplies to support the Libyan revolution in early 2011. As a result, volunteers set up and coordinated humanitarian convoys from major Egyptian cities to Tripoli. But these convoys faced two major problems. First, volunteers needed to know where the convoys were in order to communicate this to Libyan revolutionists so they could wait for the fleet at the border and escort them to Tripoli. Second, because these volunteers were headed into a war zone, their friends and family wanted to keep track of them to make sure they were safe. The solution? IntaFeen.com.

Inta feen? means “where are you?” in Arabic and IntaFeen.com is a mobile check-in service like Foursquare but localized for the Arab World. Convoy drivers used IntaFeen to check-in at different stops along the way to Tripoli to provide regular updates on the situation. This is how volunteers back in Egypt who coordinated the convoy kept track of their progress and communicated updates in real-time to their Libyan counterparts. Volunteers who went along with the convoys also used IntaFeen and their check-in’s would also get posted on Twitter and Facebook, allowing families and friends in Egypt to track their whereabouts.

Al Amain Road is a highway between Alexandria and Tripoli. These tweets and check-in’s acted as a DIY fleet management system for volunteers and activists.

The use of IntaFeen combined with Facebook and Twitter also created an interesting side-effect in terms of social media marketing to promote activism. The sharing of these updates within and across various social networks galvanized more Egyptians to volunteer their time and resulted in more convoys.

I wonder whether these activists knew about another crowdsourced volunteer project taking place at exactly the same time in support of the UN’s humanitarian relief operations: Libya Crisis Map. Much of the content added to the map was sourced from social media. Could the #LibyaConvoy project have benefited from the real-time situational awareness provided by the Libya Crisis Map?

Will we see more convergence between volunteer-run crisis maps and volunteer-run humanitarian response in the near future?

Big thanks to Adel Youssef from IntaFeen.com who spoke about this fascinating project (and Ushahidi) at Where 2.0 this week. More information on #Libya Convoy is available here. See also my earlier blog posts on the use of check-in’s for activism and disaster response.

The Best of iRevolution: Four Years of Blogging

Many thanks to all iRevolution readers who have been following this blog, which turns four years old today. Thank you also for commenting, providing feedback, getting in touch and for sharing my writing! I have published over 500 blog posts to date and have relished the time and your input. To celebrate, I’ve listed below some of my favorite blog posts along with the most popular ones. Onwards!

2008

Operation Vula: ICT versus Apartheid

GSM versus People Power in Africa

Twitter Speed to the Rescue

UN World Food Program to use UAVs

Intellipedia for Humanitarian Warning/Response?

Crimson Hexagon: Early Warning 2.0?

Tracking Genocide by Remote Sensing

Crisis Mapping Kenya’s Election Violence

Digital Resistance: Between Digital Activism & Civil Resistance

2009

Crowdsourcing Honesty?

Project Cybersyn: Chile 2.0 in 1973

A Brief History of Crisis Mapping (Updated)

Threat and Risk Mapping Analysis in Sudan

Crisis Mapping and Agent Based Models

Folksomaps: Gold Standard for Community Mapping

Disaster Theory for Techies

How To Communicate Securely in Repressive Environments

What Does a Wasp Have To Do With Civil Resistance? Everything

US Calls for UN Aerial Surveillance to Detect Preparations for Attacks

Doctor Snow’s Health Map Propaganda

Promises and Pitfalls in the Spatial Prediction of Ethnic Violence

The Polymath Project & Crisis Mapping: Lessons in Collaborative Analysis

Why Dictators Love the Web or: How I Learned to Stop Worrying and Say So What?!

Digital Activism and the Puffy Clouds of Anecdote Heaven

2010

The Starfish and the Spider: 8 Principles of Decentralization

Breaking News: Repressive States Use Technologies to Repress!

Haiti and the Power of Crowdsourcing

Using Massive Multiplayer Games to Turksource Crisis Information

Rethinking the UN’s Global Pulse

Wag the Dog, or How Falsifying Crowdsourced Data Can Be a Pain

Failing Gracefully in Complex Systems: A Note on Resilience

Crowdsourcing and the Veil of Ignorance: A Question of Morality?

How to Run a Successful Crowdsourcing Project

Demystifying Crowdsourcing: An Introduction to Non-Probability Sampling

On Technology and Learning, Or Why Wright Brothers Did Not Create the 747

Wanted: Hyper Local Disruption

The Crowd is Always There: A Marketplace for Crowdsourcing Crisis Response

Calling 911: What Humanitarians Can Learn from 50 Years of Crowdsourcing

How Crowdsourced Data Can Predict Crisis Impact

2011

What is Crisis Mapping? An Update on the Field and Looking Ahead

Civil Resistance: Early Lessons Learned from Sudan’s #Jan30

Maps, Activism and Technology: Check-In’s with a Purpose

Using a Map to Bear Witness in Egypt #Jan25

How to Use Facebook if You Are a Repressive Regime

Civil Resistance Tactics Used in Egypt’s Revolution #Jan25

The Volunteers Behind the Libya Crisis Map: A True Story

Can Live Crisis Maps Help Prevent Mass Atrocities?

How To Use Technology To Counter Rumors During Crises

Mobile Banking and the Dictator’s Dilemma

Video: Changing the World, One Map at a Time

The Smart-Talk Trap in the Era of Social Media (and What to Do About It)

List of Completely Wrong Assumptions About Tech Use in Emerging Economies

On Tech and Building Resilient Societies to Mitigate the Impact of Disasters

Why Geo-Fencing Will Revolutionize Crisis Mapping

On Genghis Khan, Borneo and Galaxies

How to Crowdsource Happiness

Crowdsourcing and Crisis Mapping World War I

How to Crowdsource Crisis Response

Theorizing Ushahidi: An Academic Treatise

Democratizing ICT for Development: DIY Innovation & Open Data

Crowdsourcing Satellite Imagery Analysis for UNHCR-Somalia

Do “Liberation Technologies” Change the Balance of Power Between Repressive Regimes and Civil Society?

Beyond the Dot: Building Visual DNA for Crisis Mapping

How to Verify Crowdsourced Information from Social Media

Crowdsourcing vs Putin: “Mapping Dots is a Disease on the Map of Russia”

Why Bounded Crowdsourcing is Important for Crisis Mapping and Beyond

Amplifying Somali Voices Using SMS and a Live Map: #SomaliaSpeaks

SMS for Violence Prevention: PeaceTXT International Launches in Kenya

2012 

What do Travel Guides and Nazi Germany have to do with Crisis Mapping and Security?

Google Inc + World Bank = Empowering Citizen Cartographers?

The Use of Drones for Nonviolent Civil Resistance

Mobile Technologies, Crisis Mapping & Disaster Response

Innovation and Counter-Innovation: Digital Resistance in Russia

Truthiness as Probability: Moving Beyond the True or False Dichotomy when Verifying Social Media

#UgandaSpeaks: Al-Jazeera uses Ushahidi to Amplify Local Voices in Response to #Kony2012

Twitter, Crises and Early Detection: Why “Small Data” Still Matters

Crisis Mapping Syria: Automated Data Mining & Crowdsourced Human Intel

On Rumors, Repression and Digital Disruption in China: Opening Pandora’s Inbox of Truthiness?

The Economist recently published a brilliant piece on China entitled: “The Power of Microblogs: Zombie Followers and Fake Re-Tweets.” BBC News followed with an equally excellent article: “Damaging Coup Rumors Ricochet Across China.” Combined, these articles reveal just how profound the digital disruption in China is likely to be now that Pandora’s Inbox has been opened.

Credit: The Economist

The Economist article opens with an insightful historical comparison:

“In the year 15AD, during the short-lived Xin dynasty, a rumor spread that a yellow dragon, a symbol of the emperor, had inauspiciously crashed into a temple in the mountains of central China and died. Ten thousand people rushed to the site. The emperor Wang Mang, aggrieved by such seditious gossip, ordered arrests and interrogations to quash the rumor, but never found the source. He was dethroned and killed eight years later, and Han-dynasty rule was restored.”

“The next ruler, Emperor Guangwu, took a different approach, studying rumors as a barometer of public sentiment, according to a recent book Rumors in the Han Dynasty by Lu Zongli, a historian. Guangwu’s government compiled a ‘Rumors Report’, cataloguing people’s complaints about local officials, and making assessments that were passed to the emperor. The early Eastern Han dynasty became known for officials who were less corrupt and more attuned to the people.”

In present day China, a popular pastime among 250+ million Chinese users of microblogging platforms is to “spread news and rumors, both true and false, that challenge the official script of government officials and state-propaganda organs.” In Domination and the Arts of Resistance: Hidden Transcripts, James Scott distinguishes between public and hidden transcripts. The former describes the open, public discourse that take place between dominators and oppressed while hidden transcripts relate to the critique of power that “goes on offstage”, which the power elites cannot decode. Scott writes that when the oppressed classes publicize this “hidden transcript”, (the truthiness?) they become con-scious of its common status. Borrowing from Juergen Habermas (as interpreted by Clay Shirky), those who take on the tools of open expression become a public, and a synchronized public increasingly constrains undemocratic rulers while ex-panding the rights of that public. The result in China? “It is hard to overestimate how much the arrival of [microblogging platforms] has changed the dynamic between rulers and ruled over the past two years” (The Economist).

Chinese authorities have responded to this threat in two predictable ways, one repeating the ill-fated actions of the Xin Dynasty and the other reflecting the more open spirit of Emperor Guangwu. In the latter case, authorities are turning to microblogs as a “listening post” for public opinion and also as a publishing platform. Indeed, “government agencies, party organs and individual officials have set up more than 50,000 weibo accounts [Chinese equivalent of Twitter]” (The Economist). In the former case, the regime has sought to “combat rumors harshly and to tighten controls over the microblogs and their users, censoring posts and closely monitoring troublemakers.” The UK Guardian reports that China is now “taking the toughest steps yet against major microblogs and detain-ing six people for spreading rumors of a coup amid Beijing’s most serious political crisis for years.”

Beijing’s attempt to regulate microblogging companies by requiring users to sign up with their real names is unlikely to be decisive, however. “No matter how it is enforced, user verification seems unlikely to deter the spread of rumors and information that has so concerned authorities” (The Economist). To be sure, companies are already selling fake verification services for a small fee. Besides, verifying accounts for millions of users is simply too time-consuming and hence costly. Even Twitter gave up their verified account service a while back. The task of countering rumors is even more of a Quixotic dream.

Property tycoon Zhang Xin, who has more than 3 million followers, wrote: “What is the best way to stop ‘rumors’? It is transparency and openness. The more speech is discouraged, the more rumors there will be” (UK Guardian).

This may in part explains why Chinese authorities have shifted their approach to one of engagement as evidenced by those 50,000 new weibo accounts. With this second reaction, however, Beijing is possibly passing the point of no return. “This degree of online engagement can be awkward for authorities used to a comfortable buffer from public opinion,” writes The Economist. This is an understatement; Pandora’s (In)box is now open and the “hidden transcript” is cloaked no longer. The critique of power is decoded and elites are “forced” to devise a public reply as a result of this shared awareness lest they lose legitimacy vis-a-vis the broader population. But the regime doesn’t even have a “customer service” mechanism in place to deal with distributed and potentially high-volume complaints. Censorship is easy compared to engagement.

Recall the “Rumors Report” compiled by Emperor Guangwu’s government to catalogue people’s complaints about local officials. How will these 50,000 new weibo users deal with such complaints now that the report can be crowdsourced, especially given that fact that China’s “Internet users have become increasingly bold in their willingness to discuss current affairs and even sensitive political news [...]” (UK Guardian).

As I have argued in my dissertation, repressive regimes can react to real (or perceived)  threats posed by “liberation technologies” by either cracking down and further centralizing control and/or by taking on the same strategies as digital activists, which at times requires less centralization. Either way, they’re taking the first step on a slippery slope. By acknowledging the problem of rumors so publicly, the regime is actually calling more attention to how disruptive these simple speculations can be—the classic Streisand effect.

“By falsely packaging lies and speculation as ‘truth’ and ‘existence’, online rumours undermine the morale of the public, and, if out of control, they will seriously disturb the public order and affect social stability,” said a commentary in the People’s Daily, the official Communist party newspaper. (UK Guardian).

Practically speaking, how will those 50,000 new weibo users coordinate their efforts to counter rumors and spread state propaganda? “We have a saying among us: you only need to move your lips to start a rumor, but you need to run until your legs are broken to refute one,” says an employee of a state media outlet (The Economist). How will these new weibo users synchronize collective action in near real-time to counter rumors when any delay is likely to be interpreted as evidence of further guilt? Will they know how to respond to myriads of questions being bombarded at them in real-time by hundreds of thousands of Chinese microbloggers? This may lead to high-pressure situations that are rife for mistakes and errors, particularly if these government officials are new to microblogging. Indeed, If just one of these state-microbloggers slips, that slip could go viral with a retweet tsunami. Any retreat by authorities from this distributed engagement strategy will only lead to more rumors.

The rumors of the coup d’état continue to ricochet across China, gaining remarkable traction far and wide. Chinese microblogs were also alight last week with talk of corruption and power struggles within the highest ranks of the party, which may have fueled the rumor of an overthrow. This is damaging to China’s Communist Party which “likes to portray itself as unified and in control,” particularly as it prepares for it’s once-in-a-decade leadership shuffle. “The problem for China’s Communist Party is that it has no effective way of refuting such talk. There are no official spokesmen who will go on the record, no sources briefing the media on the background. Did it happen? Nobody knows. So the rumors swirl” (BBC News). Even the official media, which is “often found waiting for political guidance, can be slow and unresponsive.”

So if Chinese authorities and state media aren’t even equipped (beyond plain old censorship) to respond to national rumors of vis-a-vis an event as important as a coup (can it possibly get more important than that?), then how in the world will they deal with the undercurrent of rumors that continue to fill Chinese microblogs now that these can have 50,000 new targets online? Moreover, “many in China are now so cynical about the level of censorship that they will not believe what comes from the party’s mouthpieces even if it is true. Instead they will give credence to half-truths or fabrications on the web,” which is “corrosive for the party’s authority” (BBC News). This is a serious problem for China’s Communist elite who are obsessed with the task of projecting an image of total unity and stability.

In contrast, speculators on Chinese microblogging platforms don’t need a highly coordinated strategy to spread conspiracies. They are not handicapped by the centralization and collective action problem that Chinese authorities face; after all, it is clearly far easier to spread a rumor than to debunk one. As noted by The Economist, those spreading rumors have “at their disposal armies of zombie followers and fake re-tweets as well as marketing companies, which help draw attention to rumors until they are spread by a respected user with many real followers, such as a celebrity.” But there’s more at stake here than mere rumors. In fact, as noted by The Economist, the core of the problem has less to do with hunting down rumors of yellow dragons than with “the truth that they reflect: a nervous public. In the age of weibo, it may be that the wisps of truth prove more problematic for authorities than the clouds of falsehood.”

Fascinating epilogues:

China’s censorship can never defeat the internet
China’s censors tested by microbloggers who keep one step ahead of state media

Crisis Mapping Syria: Automated Data Mining and Crowdsourced Human Intelligence

The Syria Tracker Crisis Map is without doubt one of the most impressive crisis mapping projects yet. Launched just a few weeks after the protests began one year ago, the crisis map is spearheaded by a just handful of US-based Syrian activists have meticulously and systematically documented 1,529 reports of human rights violations including a total of 11,147 killings. As recently reported in this NewScientist article, “Mapping the Human Cost of Syria’s Uprising,” the crisis map “could be the most accurate estimate yet of the death toll in Syria’s uprising [...].” Their approach? “A combination of automated data mining and crowdsourced human intelligence,” which “could provide a powerful means to assess the human cost of wars and disasters.”

On the data-mining side, Syria Tracker has repurposed the HealthMap platform, which mines thousands of online sources for the purposes of disease detection and then maps the results, “giving public-health officials an easy way to monitor local disease conditions.” The customized version of this platform for Syria Tracker (ST), known as HealthMap Crisis, mines English information sources for evidence of human rights violations, such as killings, torture and detainment. As the ST Team notes, their data mining platform “draws from a broad range of sources to reduce reporting biases.” Between June 2011 and January 2012, for example, the platform collected over 43,o00 news articles and blog posts from almost 2,000 English-based sources from around the world (including some pro-regime sources).

Syria Tracker combines the results of this sophisticated data mining approach with crowdsourced human intelligence, i.e., field-based eye-witness reports shared via webform, email, Twitter, Facebook, YouTube and voicemail. This naturally presents several important security issues, which explains why the main ST website includes an instructions page detailing security precautions that need to be taken while sub-mitting reports from within Syria. They also link to this practical guide on how to protect your identity and security online and when using mobile phones. The guide is available in both English and Arabic.

Eye-witness reports are subsequently translated, geo-referenced, coded and verified by a group of volunteers who triangulate the information with other sources such as those provided by the HealthMap Crisis platform. They also filter the reports and remove dupli-cates. Reports that have a low con-fidence level vis-a-vis veracity are also removed. Volunteers use a dig-up or vote-up/vote-down feature to “score” the veracity of eye-witness reports. Using this approach, the ST Team and their volunteers have been able to verify almost 90% of the documented killings mapped on their platform thanks to video and/or photographic evidence. They have also been able to associate specific names to about 88% of those reported killed by Syrian forces since the uprising began.

Depending on the levels of violence in Syria, the turn-around time for a report to be mapped on Syria Tracker is between 1-3 days. The team also produces weekly situation reports based on the data they’ve collected along with detailed graphical analysis. KML files that can be uploaded and viewed using Google Earth are also made available on a regular basis. These provide “a more precisely geo-located tally of deaths per location.”

In sum, Syria Tracker is very much breaking new ground vis-a-vis crisis mapping. They’re combining automated data mining technology with crowdsourced eye-witness reports from Syria. In addition, they’ve been doing this for a year, which makes the project the longest running crisis maps I’ve seen in a hostile environ-ment. Moreover, they’ve been able to sustain these import efforts with just a small team of volunteers. As for the veracity of the collected information, I know of no other public effort that has taken such a meticulous and rigorous approach to documenting the killings in Syria in near real-time. On February 24th, Al-Jazeera posted the following estimates:

Syrian Revolution Coordination Union: 9,073 deaths
Local Coordination Committees: 8,551 deaths
Syrian Observatory for Human Rights: 5,581 deaths

At the time, Syria Tracker had a total of 7,901 documented killings associated with specific names, dates and locations. While some duplicate reports may remain, the team argues that “missing records are a much bigger source of error.” Indeed, They believe that “the higher estimates are more likely, even if one chooses to disregard those reports that came in on some of the most violent days where names were not always recorded.”

The Syria Crisis Map itself has been viewed by visitors from 136 countries around the world and 2,018 cities—with the top 3 cities being Damascus, Washington DC and, interestingly, Riyadh, Saudia Arabia. The witnessing has thus been truly global and collective. When the Syrian regime falls, “the data may help sub-sequent governments hold him and other senior leaders to account,” writes the New Scientist. This was one of the principle motivations behind the launch of the Ushahidi platform in Kenya over four years ago. Syria Tracker is powered by Ushahidi’s cloud-based platform, Crowdmap. Finally, we know for a fact that the International Criminal Court (ICC) and Amnesty International (AI) closely followed the Libya Crisis Map last year.

Communicating with Disaster Affected Communities (CDAC)

Communication is Aid: Curated tweets and commentary from the CDAC Network’s Media and Technology Fair, London 2012. My commentary in blue. This is the first time I’ve used Storify to curate content. (I bumped into the co-founder of the platform at SXSW which reminded me I really needed to get in on the action).

  1. Sha

    re
    “After the Japan earthquake, >20% of ALL web queries issued were on tsunamis.” @spangledrongo #commisaid

    Thu, Mar 22 2012 08:53:43
  2. Would be great to see how this type of search data compares to data from Tweets. Take this analysis of tweets following the earthquake in Chile, for example.
  3. Share
    RT @UNFPA: Professional systems are being replaced by consumer tools says @Google Crisis Response #commisaid

    Thu, Mar 22 2012 08:48:58
  4. And as a result, crisis-affected communities are increasingly becoming digital as I note in this blog post.
  5. Share
    Closed systems closed data will be left behind and unused:crisis response is social and collaboration is empowering @CDACNetwork #commisaid

    Thu, Mar 22 2012 08:49:58
  6. Share
    RT @catherinedem: Crisis response is #social – online social collaboration spikes during and after disaster @spangledrongo #commisaid

    Thu, Mar 22 2012 08:54:03
  7. Share
    “It is essential to have authoritative content” Nigel Snoad at @CDACNetwork’s Media & Tech Fair #commisaid

    Thu, Mar 22 2012 08:58:28
  8. Does this mean that all user-generated content should be ignored because said content does not necessarily come from a known and authoritative source? Who decides what is authoritative?
  9. Share
    we don’t empower communities by giving them info,they empower themselves by giving us info that we can act on-@komunikasikan #commisaid

    Thu, Mar 22 2012 11:31:02
  10. What if this information is not authoritative because it does not come from official sources?
  11. Share
    RT @ushahidi: “In a crisis, the mobile internet stays most resilient, even more than SMS.” #commisaid Nigel Snoad

    Thu, Mar 22 2012 09:02:53
  12. Share
    RT @jqg: Empower local communities to generate their own tools and figure out their own solutions #commisaid

    Thu, Mar 22 2012 09:10:43
  13. See this blog post on Democratizing ICT for Development Using DIY Innovation and Open Data.
  14. Share
    Through #Mission4636 SMS system, radio presenter @carelpedre was able to communicate directly with affected people in Haiti #commisaid

    Thu, Mar 22 2012 12:20:14
  15. This is rather interesting, I hadn’t realized that radio stations in Haiti actively used the information from the Ushahidi Haiti 4636 project.
  16. Share
    Fascinating talk with @carelpedre- many of #Haiti ‘s pre-earthquake twitter users came from @juno7′s lottery push notifications #Commisaid

    Thu, Mar 22 2012 11:11:43
  17. Share
    More about @Carelpedre using 4636 project after #Haiti earthquake bit.ly/plgzXJ #commisaid

    Thu, Mar 22 2012 12:17:52
  18. Share
    Internews: Innovation comes within, info comes from within, people will find ways to communicate no matter what #commisaid @CDACNetwork

    Thu, Mar 22 2012 09:26:25
  19. So best of luck to those who wish to regulate this space! As my colleague Tim McNamara has noted “Crisis mapping is not simply a technological shift, it is also a process of rapid decentralisation of power. With extremely low barriers to entry, many new entrants are appearing in the fields of emergency and disaster response. They are ignoring the traditional hierarchies, because the new entrants perceive that there is something that they can do which benefits others.”
  20. Share
    @souktel: “be simple, be creative and learn from the community around you” #commisaid

    Thu, Mar 22 2012 10:57:09
  21. Share
    Tools shouldn’t own data. RT @whiteafrican: “It’s not about the platform being open, it’s about the data being open”- @jcrowley #commisaid

    Thu, Mar 22 2012 11:56:08
  22. Share
    Humanitarians have to get their heads around media and tech or risk being left behind #m4d #media #tech #commisaid

    Thu, Mar 22 2012 11:56:58
  23. Share
    Cutting edge is to get the #crowd & the #algorithm to filter each other in filtering massive overload of information in a crisis #commisaid

    Thu, Mar 22 2012 12:01:17
  24. As Robert Kirkpatrick likes to say, “Use the hunch of the expert, machine algorithms and the wisdom of the crowd.”
  25. Share
    “How long until the disaster affected communities start analyzing the aid agencies?” #commisaid

    Thu, Mar 22 2012 12:14:50
  26. Yes! Sousveillance meets analysis of big data on the humanitarian sector.
  27. Share
    Why is it such a big deal, for the humanitarian industry to get feedback from a community? Companies have done it for decades. #commisaid

    Thu, Mar 22 2012 12:17:31
  28. Some of my thoughts on what the humanitarian community can learn from the private sector vis-a-vis customer support.
  29. Share
    People who are not traditional humanitarian actors are taking on humanitarian roles, driven by the democratisation of technology #commisaid

    Thu, Mar 22 2012 13:00:12
  30. Indeed, not only are disaster-affected communities increasingly digital, so are global volunteer networks like the Standby Volunteer Task Force (SBTF).
  31. Share
    Technology is shifting the power balance – it’s helping local communities to organise their own responses to disasters #commisaid

    Thu, Mar 22 2012 13:11:18
  32. Indeed, as a result of these mobile technologies, affected populations are increasingly able to source, share and generate a vast amount of information, which is completely transforming disaster response. More on this here.
  33. Share
    Like Paul Currion’s analogy – humanitarian sector risks obsolescence in the same way the record industry did. Watch out? #commisaid

    Thu, Mar 22 2012 13:17:36
  34. Share


    Thu, Mar 22 2012 14:57:13
  35. One of my favorite books, The Starfish and the Spider: The Unstoppable Power of Leaderless Organizations, has an excellent case study on the music industry. The above picture is taken from that chapter and charts the history of the industry from the perspective of hierarchies vs networks. I’ve argued a couple years ago that the same dynamic is taking place within humanitarian response. See this blog post on Disaster Relief 2.0: Toward a Multipolar System.
  36. Share
    A thousand flowers can bloom beautifully IF common data standards allow sharing. Right now no natural selection improving quality #commisaid

    Thu, Mar 22 2012 13:43:19
  37. Share
    @GSMADisasterRes :free phone numbers &short codes r not silver bullets 4 meaningful communication w/disaster affected communities #commisaid

    Thu, Mar 22 2012 13:58:44
  38. Share
    Scale horizontally, not vertically. The end of command and control?! #commisaid

    Thu, Mar 22 2012 14:07:54
  39. Share
    RT @reeniac: what now? communities have the solution, we need to listen. start by including them in the discussions – Dr Jamilah #commisaid

    Thu, Mar 22 2012 14:14:55