Category Archives: Social Media

Inferring International and Internal Migration Patterns from Twitter

My QCRI colleagues Kiran Garimella and Ingmar Weber recently co-authored an important study on migration patterns discerned from Twitter. The study was co-authored with  Bogdan State (Stanford)  and lead author Emilio Zagheni (CUNY). The authors analyzed 500,000 Twitter users based in OECD countries between May 2011 and April 2013. Since Twitter users are not representative of the OECD population, the study uses a “difference-in-differences” approach to reduce selection bias when in out-migration rates for individual countries. The paper is available here and key insights & results are summarized below.

Twitter Migration

To better understand the demographic characteristics of the Twitter users under study, the authors used face recognition software (Face++) to estimate both the gender and age of users based on their profile pictures. “Face++ uses computer vision and data mining techniques applied to a large database of celebrities to generate estimates of age and sex of individuals from their pictures.” The results are depicted below (click to enlarge). Naturally, there is an important degree of uncertainty about estimates for single individuals. “However, when the data is aggregated, as we did in the population pyramid, the uncertainty is substantially reduced, as overestimates and underestimates of age should cancel each other out.” One important limitation is that age estimates may still be biased if users upload younger pictures of themselves, which would result in underestimating the age of the sample population. This is why other methods to infer age (and gender) should also be applied.

Twitter Migration 3

I’m particularly interested in the bias-correction “difference-in-differences” method used in this study, which demonstrates one can still extract meaningful information about trends even though statistical inferences cannot be inferred since the underlying data does not constitute a representative sample. Applying this method yields the following results (click to enlarge):

Twitter Migration 2

The above graph reveals a number of interesting insights. For example, one can observe a decline in out-migration rates from Mexico to other countries, which is consistent with recent estimates from Pew Research Center. Meanwhile, in Southern Europe, the results show that out-migration flows continue to increase for  countries that were/are hit hard by the economic crisis, like Greece.

The results of this study suggest that such methods can be used to “predict turning points in migration trends, which are particularly relevant for migration forecasting.” In addition, the results indicate that “geolocated Twitter data can substantially improve our understanding of the relationships between internal and international migration.” Furthermore, since the study relies in publicly available, real-time data, this approach could also be used to monitor migration trends on an ongoing basis.

To which extent the above is feasible remains to be seen. Very recent mobility data from official statistics are simply not available to more closely calibrate and validate the study’s results. In any event, this study is an important towards addressing a central question that humanitarian organizations are also asking: how can we make statistical inferences from online data when ground-truth data is unavailable as a reference?

I asked Emilio whether techniques like “difference-in-differences” could be used to monitor forced migration. As he noted, there is typically little to no ground truth data available in humanitarian crises. He thus believes that their approach is potentially relevant to evaluate forced migration. That said, he is quick to caution against making generalizations. Their study focused on OECD countries, which represent relatively large samples and high Internet diffusion, which means low selection bias. In contrast, data samples for humanitarian crises tend to be far smaller and highly selected. This means that filtering out the bias may prove more difficult. I hope that this is a challenge that Emilio and his co-authors choose to take on in the near future.

bio

Social Media: The First 2,000 Years

What do Papyrus rolls and Twitter have in common? Both were used as a means of “instant” communication. Indeed, a careful reading of history reveals just how ancient social media really is. Further, the questions we pose about social media today have already been debated countless times over hundreds of years. Author Tom Standage traces this fascinating history of social media in his thought-provoking book Writing on the Wall: Social Media – The First 2,000 YearsIn so doing, Tom forces us to rethink our understanding and assumptions of social media use today. To be sure, this book will change the way you think about social media. I highlight some of the most intriguing insights below.

Marcus Tullius Cicero (106 BC to 43 BC) was a Roman philosopher, politician and lawyer. When Julius Caesar relocated him from Rome to a distant output, Cicero drew on an elaborate communication system and social network to stay abreast of events in the capital. Printing presses did not exist at the time, nor did paper for that matter. So papyrus rolls were used to exchange letters and other documents, which were in turn copied, commented on and shared. In this way, Cicero received timely updates on politics and gossip coming from Rome, having asked his contacts in the capital to write him daily. Common abbreviations were soon used to save space and time, much like today’s acronyms on social media (e.g., BTW, AFAIK) . SVBEEV (si vales, bene est, ego valeo), for example, was a popular acronym for “if you are well, that is good, I am well.” Often, letters were also quoted in other letters, much like blog posts today. In fact, some letters during Cicero’s time were “addressed to several people and were written [....] to be posted in public for general consumption.”

The enabling infrastructure of this information system was slavery—many of the scribes and messengers who copied and delivered messages were slaves. In short, “slaves were the Roman equivalent of broadband.” Friends were also used to carry letters across cities, countries and indeed continents. “One advantage of getting friends to pass on the news was that they could highlight items of interest and add their own comments or background information in the covering letters they sent along with the copied letters. The combination of personal letters and impersonal news was more valuable then either in isolation, because each provided additional context for the other. And then, as now, one was far more likely to pay attention if a friend said it was important or expressed an opinion about it.”

wax tablets

Not all letters during Cicero’s time were sent via papyrus rolls. Wax tablets (pictured above) mounted in wooden frames “that fold together like a book” were used for messages sent over a short distance, which required a quick reply. “To modern eyes, these tablets [...] look strikingly similar to tablet computers. The recipient’s response could be scratched onto the same tablet, and the messenger who had delivered it would then take it straight back to the sender.” Earlier, in Mesopotamia, “letters were written in cuneiform on small clay tablets that fit neatly into the palm of the hand. Letters almost always fit onto a single tablet, which imposed a limit on the length of the message.” One can’t help but draw parallels with smartphones and Twitter.

Graffiti was also served as a social media some 2,000 years ago. In Pompeii, for example, ancient graffiti was found on the streets, in bars and also in private houses. Writing graffiti was not regarded as defacement at the time. “The most prominent messages, painted in large letters, were political slogans expressing support for candidates running for election [...].” Criticisms of political candidates were also found in Pompeii’s ancient graffiti, as were advertisements for events and even rental vacancies. As author Tom Standage notes, “the great merit of graffiti is that one did not have to be a magistrate [...] to add one’s voice to the conversation; the walls were open to everyone.”

The following graffiti messages found in Pompeii reveal what ordinary people were thinking about:

“I won 8,522 denarii by gaming, fair play!”

“I made bread”

“The man I am having dinner with is a barbarian”

“Atimetus got me pregnant”

These provided “glimpses of everyday activities, rather like status updates on modern social networks.” In addition, graffiti messages were left near inns as advice to potential customers, serving both positive and negative reviews, much like today’s Yelp and related websites. “More practical still were the messages addressed to specific people.” Examples include:

“Samius to Cornelius: go hang yourself!”

“Gaius Sabinus says a fond hello to Statius. Traveler, eat bread in Pompeii but go to Nuceria to drink. At Nuceria, the drinking is better”

According to Tom, there are “even a few examples of dialogues, where an inscription inspired comments or responses.” Not surprisingly, perhaps, “the sexual boasts and scatological humor familiar from modern graffiti in public lavatories can also be found in Pompeii [...].” Like much of social media today, a lot of the ancient graffiti that appeared on the walls of Pompeii was of no interest to anyone. As one graffiti message, which appeared four times in the ancient city laments: “Oh wall, I am amazed you haven’t fallen down, since you bear the tedious scribblings of so many writers.”

Pompeii-street-graffiti-631

In the 1500’s, as printing presses multiplied and religious pamphlets vent viral, there was growing anxiety about the potential spread of false information (and seditious books). This ignited a heated debate on whether printing should be tightly regulated. In England, the company that had been given draconian powers to destroy unregistered printing presses and books that painted the Monarchy in bad light. This company argued the following:

“If every man may print, that is so disposed, it may be a means, that heresies, treasons, and seditious libelles shall be too often dispersed, whereas if only known men do print this inconvenience is avoided.”

Again, the parallels with today’s debate over the reliability of crowdsourcing and user-generated content are clear. But the growing desire for new in Europe during the Thirty Year’s War (early 1600’s) made crowdsourcing a compelling approach for the collection of news reports. Indeed, the increasing demand for “news about the war led to the appearance of a new type of publication: the coranto. This was a single sheet, printed on both sides, with a compilation of items, usually letters or eyewitness accounts of battles or other notable events. [...]. Being anonymous, corantos were regarded as less trustworthy than handwritten news letters, which often related news at first hand.” So a backlash against corantos was inevitable, with criticism that we hear today about social media. For example, one critic at the time “thought it dangerous for ordinary people to have greater access to news, because printing allowed rumors and falsehoods to spread, causing social and political instability.”

In any event, “the pamphlets of the 1640’s existed in an interconnected web, constantly referring to, quoting, or in dialogue with each other, like blog posts today.” As this information web continued continued to scale, “the bewildering variety of new voices and formats made it very difficult to work out what was going on. As one observer put it, ‘oft times we have much more printed than is true.” But John Milton didn’t buy the arguments for regulating written speech. Milton countered that no one is truly capable of acting as a reasonable censor since humans are susceptible to  error or bias. While press freedom would allow “bad or erroneous works to be printed,” Milton argued that this was actually a good thing. “If more readers came into contact with bad ideas because of printing, those ideas could be more swiftly and easily disproved.” In essence, Milton was making the case for crowdsourced verification of information. Similar arguments have recently been made.

coffee-house

Meanwhile, at the coffee house. The first caffeinated drink reached Europe around the 1600’s. “And along with the coffee bean itself came the institution of the coffeehouse, which had become an important meeting place and source of news in the Arab world.” The same was to happen in Europe, where coffeehouses served the same function as today’s co-working spaces and innovation hubs/labs. Some coffee houses were “thronged with businessmen, who would keep regular hours at particular coffee houses so that their associates would know where to find them, and who used coffee houses as offices, meeting rooms, and venues for trade.” Indeed, “the main business of coffee houses was the sharing and discussion of news and opinion [...].” In sum, “coffee houses were an alluring social platform for sharing information.”

There’s a lot more to “Writing on the Wall” than summarized above, such as the tension between press regulation and freedom, how the era of centralized, mass media dominance was a two-century anomaly in the natural course of social media, the origins of the political economy of mass media, etc. So I highly recommend this book to iRevolution readers. I, for one, relished it.


bio

Yes, I’m Writing a Book (on Digital Humanitarians)

I recently signed a book deal with Taylor & Francis Press. The book, which is tentatively titled “Digital Humanitarians: How Big Data is Changing the Face of Disaster Response,” is slated to be published next year. The book will chart the rise of digital humanitarian response from the Haiti Earthquake to 2015, highlighting critical lessons learned and best practices. To this end, the book will draw on real-world examples of digital humanitarians in action to explain how they use new technologies and crowdsourcing to make sense of “Big (Crisis) Data”. In sum, the book will describe how digital humanitarians & humanitarian technologies are together reshaping the humanitarian space and what this means for the future of disaster response. The purpose of this book is to inspire and inform the next generation of (digital) humanitarians while serving as a guide for established humanitarian organizations & emergency management professionals who wish to take advantage of this transformation in humanitarian response.

2025

The book will thus consolidate critical lessons learned in digital humanitarian response (such as the verification of social media during crises) so that members of the public along with professionals in both international humanitarian response and domestic emergency management can improve their own relief efforts in the face of “Big Data” and rapidly evolving technologies. The book will also be of interest to academics and students who wish to better understand methodological issues around the use of social media and user-generated content for disaster response; or how technology is transforming collective action and how “Big Data” is disrupting humanitarian institutions, for example. Finally, this book will also speak to those who want to make a difference; to those who of you who may have little to no experience in humanitarian response but who still wish to help others affected during disasters—even if you happen to be thousands of miles away. You are the next wave of digital humanitarians and this book will explain how you can indeed make a difference.

The book will not be written in a technical or academic writing style. Instead, I’ll be using a more “storytelling” form of writing combined with a conversational tone. This approach is perfectly compatible with the clear documentation of critical lessons emerging from the rapidly evolving digital humanitarian space. This conversational writing style is not at odds with the need to explain the more technical insights being applied to develop next generation humanitarian technologies. Quite on the contrary, I’ll be using intuitive examples & metaphors to make the most technical details not only understandable but entertaining.

While this journey is just beginning, I’d like to express my sincere thanks to my mentors for their invaluable feedback on my book proposal. I’d also like to express my deep gratitude to my point of contact at Taylor & Francis Press for championing this book from the get-go. Last but certainly not least, I’d like to sincerely thank the Rockefeller Foundation for providing me with a residency fellowship this Spring in order to accelerate my writing.

I’ll be sure to provide an update when the publication date has been set. In the meantime, many thanks for being an iRevolution reader!

bio

Video: Humanitarian Response in 2025

I gave a talk on “The future of Humanitarian Response” at UN OCHA’s Global Humanitarian Policy Forum (#aid2025) in New York yesterday. More here for context. A similar version of the talk is available in the video presentation below.

Some of the discussions that ensued during the Forum were frustrating albeit an important reality check. Some policy makers still think that disaster response is about them and their international humanitarian organizations. They are still under the impression that aid does not arrive until they arrive. And yet, empirical research in the disaster literature points to the fact that the vast majority of survivals during disasters is the result of local agency, not external intervention.

In my talk (and video above), I note that local communities will increasingly become tech-enabled first responders, thus taking pressure off the international humanitarian system. These tech savvy local communities already exit. And they already respond to both “natural” (and manmade) disasters as noted in my talk vis-a-vis the information products produced by tech-savvy local Filipino groups. So my point about the rise of tech-enabled self-help was a more diplomatic way of conveying to traditional humanitarian groups that humanitarian response in 2025 will continue to happen with or without them; and perhaps increasingly without them.

This explains why I see OCHA’s Information Management (IM) Team increasingly taking on the role of “Information DJ”, mixing both formal and informal data sources for the purposes of both formal and informal humanitarian response. But OCHA will certainly not be the only DJ in town nor will they be invited to play at all “info events”. So the earlier they learn how to create relevant info mixes, the more likely they’ll still be DJ’ing in 2025.

Bio

Opening Keynote Address at CrisisMappers 2013

Screen Shot 2013-11-18 at 1.58.07 AM

Welcome to Kenya, or as we say here, Karibu! This is a special ICCM for me. I grew up in Nairobi; in fact our school bus would pass right by the UN every day. So karibu, welcome to this beautiful country (and continent) that has taught me so much about life. Take “Crowdsourcing,” for example. Crowdsourcing is just a new term for the old African saying “It takes a village.” And it took some hard-working villagers to bring us all here. First, my outstanding organizing committee went way, way above and beyond to organize this village gathering. Second, our village of sponsors made it possible for us to invite you all to Nairobi for this Fifth Annual, International Conference of CrisisMappers (ICCM).

I see many new faces, which is really super, so by way of introduction, my name is Patrick and I develop free and open source next generation humanitarian technologies with an outstanding team of scientists at the Qatar Computing Research Institute (QCRI), one of this year’s co-sponsors.

We’ve already had an exciting two-days of pre-conference site visits with our friends from Sisi ni Amani and our co-host Spatial Collective. ICCM participants observed first-hand how GIS, mobile technology and communication projects operate in informal settlements, covering a wide range of topics that include governance, civic education and peacebuilding. In addition, our friend Heather Leson from the Open Knowledge Foundation (OKF) coordinated an excellent set of trainings at the iHub yesterday. So a big thank you to Heather, Sisi ni Amani and Spatial Collective for these outstanding pre-conference events.

Screen Shot 2013-11-19 at 10.48.30 AM

This is my 5th year giving opening remarks at ICCM, so some of you will know from previous years that I often take this moment to reflect on the past 12 months. But just reflecting on the past 12 days alone requires it’s own separate ICCM. I’m referring, of course, to the humanitarian and digital humanitarian response to the devastating Typhoon in the Philippines. This response, which is still ongoing, is unparalleled in terms of the level of collaboration between members of the Digital Humanitarian Network (DHN) and formal humanitarian organizations like UN OCHA and WFP. All of these organizations, both formal and digital, are also members of the CrisisMapper Network.

Screen Shot 2013-11-18 at 2.07.59 AM

The Digital Humanitarian Network, or DHN, serves as the official interface between formal humanitarian organizations and global networks of tech-savvy digital volunteers. These digital volunteers provide humanitarian organizations with the skill and surge capacity they often need to make timely sense of “Big (Crisis) Data” during major disasters. By Big Crisis Data, I mean social media content and satellite imagery, for example. This overflow of such information generated during disasters can be as paralyzing to humanitarian response as the absence of information. And making sense of this overflow in response to Yolanda has required all hands on deck—i.e., an unprecedented level of collaboration between many members of the DHN.

So I’d like to share with you 2 initial observations from this digital humanitarian response to Yolanda; just 2 points that may be signs of things to come. Local Digital Villages and World Wide (good) Will.

Screen Shot 2013-11-18 at 2.09.42 AM

First, there were numerous local digital humanitarians on the ground in the Philippines. These digitally-savvy Filipinos were rapidly self-organizing and launching crisis maps well before any of us outside the Philippines had time to blink. One such group is Rappler, for example.

Screen Shot 2013-11-18 at 2.10.37 AM

We (the DHN) reached out to them early on, sharing both our data and volunteers. Remember that “Crowdsourcing” is just a new word for the old African saying that “it takes a village…” and sometimes, it takes a digital village to support humanitarian efforts on the ground. And Rappler is hardly the only local digital community that mobilizing in response to Yolanda, there are dozens of digital villages spearheading similar initiatives across the country.

The rise of local digital villages means that the distant future (or maybe not too distant future) of humanitarian operations may become less about the formal “brick-and-mortar” humanitarian organizations and, yes, also less about the Digital Humanitarian Network. Disaster response is and has always have been about local communities self-organizing and now local digital communities self-organizing. The majority of lives saved during disasters is attributed to this local agency, not international, external relief. Furthermore, these local digital villages are increasingly the source of humanitarian innovation, so we should pay close attention; we have a lot to learn from these digital villages. Naturally, they too are learning a lot from us.

The second point that struck me occurred when the Standby Volunteer Task Force (SBTF) completed its deployment of MicroMappers on behalf of OCHA. The response from several SBTF volunteers was rather pointed—some were disappointed that the deployment had closed; others were downright upset. What happened next was very interesting; you see, these volunteers simply kept going, they used (hacked) the SBTF Skype Chat for Yolanda (which already had over 160 members) to self-organize and support other digital humanitarian efforts that were still ongoing. So the SBTF Team sent an email to it’s 1,000+ volunteers with the following subject header: “Closing Yolanda Deployment, Opening Other Opportunities!”

Screen Shot 2013-11-18 at 2.11.28 AM

The email provided a list of the most promising ongoing digital volunteer opportunities for the Typhoon response and encouraged volunteers to support whatever efforts they were most drawn to. This second reveals that a “World Wide (good) Will” exists. People care. This is good! Until recently, when disasters struck in faraway lands, we would watch the news on television wishing we could somehow help. That private wish—that innate human emotion—would perhaps translate into a donation. Today, not only can you donate cash to support those affected by disasters, you can also donate a few minutes of your time to support the relief efforts on the ground thanks to new humanitarian technologies and platforms. In other words, you, me, all of us can now translate our private wishes into direct, online public action, which can support those working in disaster-affected areas including local digital villages.

Screen Shot 2013-11-18 at 2.12.21 AM

This surge of World Wide (good) Will explains why SBTF volunteers wanted to continue volunteering for as long as they wished even if our formal digital humanitarian network had phased out operations. And this is beautiful. We should not seek to limit or control this global goodwill or play the professional versus amateur card too quickly. Besides, who are we kidding? We couldn’t control this flood of goodwill even if we wanted to. But, we can embrace this goodwill and channel it. People care, they want to offer their time to help others thousands of miles away. This is beautiful and the kind of world I want to live in. To paraphrase the philosopher Hannah Arendt, the greatest harm in the world is caused not by evil but apathy. So we should cherish the digital goodwill that springs during disasters. This spring is the digital equivalent of mutual aid, of self-help. The global village of digital Good Samaritans is growing.

At the same time, this goodwill, this precious human emotion and the precious time it freely offers can cause more harm than good if it is not channeled responsibly. When international volunteers poor into disaster areas wanting to help, their goodwill can have the opposite effect, especially when they are inexperienced. This is also true of digital volunteers flooding in to help online.

We in the CrisisMappers community have the luxury of having learned a lot about digital humanitarian response since the Haiti Earthquake; we have learned important lessons about data privacy and protection, codes of conduct, the critical information needs of humanitarian organizations and disaster-affected populations, standardizing operating procedures, and so on. Indeed we now (for the first time) have data protection protocols that address crowdsourcing, social media and digital volunteers thanks to our colleagues at the ICRC. We also have an official code of conduct on the use of SMS for disaster response thanks to our colleagues at GSMA. This year’s World Disaster Report (WDR 2013) also emphasizes the responsible use of next generation humanitarian technologies and the crisis data they manage.

Screen Shot 2013-11-18 at 2.13.03 AM

Now, this doesn’t mean that we the formal (digital) humanitarian sector have figured it all out—far from it. This simply means that we’ve learned a few important and difficult lessons along the way. Unlike newcomers to the digital humanitarian space, we have the benefit of several years of hard experience to draw on when deploying for disasters like Typhoon Yolanda. While sharing these lessons and disseminating them as widely as possible is obviously a must, it is simply not good enough. Guidebooks and guidelines just won’t cut it. We also need to channel the global spring of digital goodwill and distribute it to avoid  “flash floods” of goodwill. So what might these goodwill channels look like? Well they already exist in the form of the Digital Humanitarian Network—more specifically the members of the DHN.

These are the channels that focus digital goodwill in support of the humanitarian organizations that physically deploy to disasters. These channels operate using best practices, codes of conduct, protocols, etc., and can be held accountable. At the same time, however, these channels also block the upsurge of goodwill from new digital volunteers—those outside our digital villages. How? Our channels block this World Wide (good) Will by requiring technical expertise to engage with us and/or  by requiring an inordinate amount of time commitment. So we should not be surprised if the “World Wide (Good) Will” circumvents our channels altogether, and in so doing causes more harm than good during disasters. Our channels are blocking their engagement and preventing them from joining our digital villages. Clearly we need different channels to focus the World Wide (Good) Will.

Screen Shot 2013-11-18 at 2.14.21 AM

Our friends at Humanitarian OpenStreetMap already figured this out two years ago when they set up their microtasking server, making it easier for less tech-savvy volunteers to engage. We need to democratize our humanitarian technologies to responsibly channel the huge surplus global goodwill that exists online. This explains why my team and I at QCRI are developing MicroMappers and why we deployed the platform in response to OCHA’s request within hours of Typhoon Yolanda making landfall in the Philippines.

Screen Shot 2013-11-18 at 2.15.21 AM

This digital humanitarian operation was definitely far from perfect, but it was super simple to use and channeled 208 hours of global goodwill in just a matter days. Those are 208 hours that did not cause harm. We had volunteers from dozens of countries around the world and from all ages and walks of life offering their time on MicroMappers. OCHA, which had requested this support, channeled the resulting data to their teams on the ground in the Philippines.

These digital volunteers all cared and took the time to try and help others thousands of miles away. The same is true of the remarkable digital volunteers supporting the Humanitarian OpenStreetMap efforts. This is the kind of world I want to live in; the world in which humanitarian technologies harvest the global goodwill and channels it to make a difference to those affected by disasters.

Screen Shot 2013-11-18 at 2.09.42 AM

So these are two important trends I see moving forward, the rise of well-organized, local digital humanitarian groups, like Rappler, and the rise of World Wide (Good) Will. We must learn from the former, from the local digital villages, and when asked, we should support them as best we can. We should also channel, even amplify the World Wide (Good) Will by democratizing humanitarian technologies and embracing new ways to engage those who want to make a difference. Again, Crowdsourcing is simply a new term for the old African proverb, that it takes a village. Let us not close the doors to that village.

So on this note, I thank *you* for participating in ICCM and for being a global village that cares, both on and offline. Big thanks as well to our current team of sponsors for caring about this community and making sure that our village does continue to meet in person every year. And now for the next 3 days, we have an amazing line-up of speakers, panelists & technologies for you. So please use these days to plot, partner and disrupt. And always remember: be tough on ideas, but gentle on people.

Thanks again, and keep caring.

#Westgate Tweets: A Detailed Study in Information Forensics

My team and I at QCRI have just completed a detailed analysis of the 13,200+ tweets posted from one hour before the attacks began until two hours into the attack. The purpose of this study, which will be launched at CrisisMappers 2013 in Nairobi tomorrow, is to make sense of the Big (Crisis) Data generated during the first hours of the siege. A summary of our results are displayed below. The full results of our analysis and discussion of findings are available as a GoogleDoc and also PDF. The purpose of this public GoogleDoc is to solicit comments on our methodology so as to inform the next phase of our research. Indeed, our aim is to categorize and study the entire Westgate dataset in the coming months (730,000+ tweets). In the meantime, sincere appreciation go to my outstanding QCRI Research Assistants, Ms. Brittany Card and Ms. Justine MacKinnon for their hard work on the coding and analysis of the 13,200+ tweets. Our study builds on this preliminary review.

The following 7 figures summarize the main findings of our study. These are discussed in more detail in the GoogleDoc/PDF.

Figure 1: Who Authored the Most Tweets?

Figure 2: Frequency of Tweets by Eyewitnesses Over Time?

Figure 3: Who Were the Tweets Directed At?

Figure 4: What Content Did Tweets Contain?

Figure 5: What Terms Were Used to Reference the Attackers?

Figure 6: What Terms Were Used to Reference Attackers Over Time?

Figure 7: What Kind of Multimedia Content Was Shared?

Early Results of MicroMappers Response to Typhoon Yolanda (Updated)

We have completed our digital humanitarian operation in the Philippines after five continuous days with MicroMappers. Many, many thanks to all volunteers from all around the world who donated their time by clicking on tweets and images coming from the Philippines. Our UN OCHA colleagues have confirmed that the results are being shared widely with their teams in the field and with other humanitarian organizations on the ground. More here.

ImageClicker

In terms of preliminary figures (to be confirmed):

  • Tweets collected during first 48 hours of landfall = ~230,000
  • Tweets automatically filtered for relevancy/uniqueness = ~55,000
  • Tweets clicked using the TweetClicker = ~ 30,000
  • Relevant tweets triangulated using TweetClicker = ~3,800
  • Triangulated tweets published on live Crisis Map = ~600
  • Total clicks on TweetClicker = ~ 90,000
  • Images clicked using the ImageClicker = ~ 5,000
  • Relevant images triangulated using TweetClicker = ~1,200
  • Triangulated images published on live Crisis Map = ~180
  • Total clicks on ImageClicker = ~15,000
  • Total clicks on MicroMappers (Image + Tweet Clickers) = ~105,000

Since each single tweet and image uploaded to the Clickers was clicked on by (at least) three individual volunteers for quality control purposes, the number of clicks is three times the total number of tweets and images uploaded to the respective clickers. In sum, digital humanitarian volunteers have clocked a grand total of ~105,000 clicks to support humanitarian operations in the Philippines.

While the media has largely focused on the technology angle of our digital humanitarian operation, the human story is for me the more powerful message. This operation succeeded because people cared. Those ~105,000 clicks did not magically happen. Each and every single one of them was clocked by humans, not machines. At one point, we had over 300 digital volunteers from the world over clicking away at the same time on the TweetClicker and more than 200 on the ImageClicker. This kind of active engagement by total strangers—good “digital Samaritans”—explains why I find the human angle of this story to be the most inspiring outcome of MicroMappers. “Crowdsourcing” is just a new term for the old saying “it takes a village,” and sometimes it takes a digital village to support humanitarian efforts on the ground.

Until recently, when disasters struck in faraway lands, we would watch the news on television wishing we could somehow help. That private wish—that innate human emotion—would perhaps translate into a donation. Today, not only can you donate cash to support those affected by disasters, you can also donate a few minutes of your time to support the operational humanitarian response on the ground by simply clicking on MicroMappers. In other words, you can translate your private wish into direct, online public action, which in turn translates into supporting offline collective action in the disaster-affected areas.

Clicking is so simple that anyone with Internet access can help. We had high schoolers in Qatar clicking away, fire officers in Belgium, graduate students in Boston, a retired couple in Kenya and young Filipinos clicking away. They all cared and took the time to try and help others, often from thousands of miles away. That is the kind of world I want to live in. So if you share this vision, then feel free to join the MicroMapper list-serve.

Yolanda TweetClicker4

Considering that MicroMappers is still very much under development, we are all pleased with the results. There were of course many challenges; the most serious was the CrowdCrafting server which hosts our Clickers. Unfortunately, that server was not able to handle the load and traffic generated by digital volunteers. So their server crashed twice and also slowed our Clickers to a complete stop at least a dozen times during the past five days. At times, it would take 10-15 seconds for a new tweet or image to load, which was frustrating. We were also limited by the number of tweets and images we could upload at any given time, usually ~1,500 at most. Any larger load would seriously slow down the Clickers. So it is rather remarkable that digital volunteers managed to clock more than 100,000 clicks given the repeated interruptions. 

Besides the server issue, the other main bottleneck was the geo-location of the ~30,000 tweets and ~5,000 images tagged using the Clickers. We do have a Tweet and Image GeoClicker but these were not slated to launch until next week at CrisisMappers 2013, which meant they weren’t ready for prime time. We’ll be sure to launch them soon. Once they are operational, we’ll be able to automatically push triangulated tweets and images from the Tweet and Image Clickers directly to the corresponding GeoClickers so volunteers can also aid humanitarian organizations by mapping important tweets and images directly.

There’s a lot more that we’ve learned throughout the past 5 days and much room for improvement. We have a long list of excellent suggestions and feedback from volunteers and partners that we’ll be going through starting tomorrow. The most important next step is to get a more powerful server that can handle a lot more load and traffic. We’re already taking action on that. I have no doubt that our clicks would have doubled without the server constraints.

For now, though, BIG thanks to the SBTF Team and in particular Jus McKinnon, the QCRI et al team, in particular Ji Lucas, Hemant Purohit and Andrew Ilyas for putting in very, very long hours, day in and day out on top of their full-time jobs and studies. And finally, BIG thanks to the World Wide Crowd, to all you who cared enough to click and support the relief operations in the Philippines. You are the heroes of this story.

bio

Live Crisis Map of Disaster Damage Reported on Social Media

Update: See early results of MicroMappers deployment here

Digital humanitarian volunteers have been busing tagging images posted to social media in the aftermath of Typhoon Yolanda. More specifically, they’ve been using the new MicroMappers ImageClicker to rate the level of damage they see in each image. Thus far, they have clicked over 7,000 images. Those that are tagged as “Mild” and “Severe” damage are then geolocated by members of the Standby Volunteer Task Force (SBTF) who have partnered with GISCorps and ESRI to create this live Crisis Map of the disaster damage tagged using the ImageClicker. The map takes a few second to load, so please be patient.

YolandaPH Crisis Map 1

The more pictures are clicked using the ImageClicker, the more populated this crisis map will become. So please help out if you have a few seconds to spare—that’s really all it takes to click an image. If there are no picture left to click or the system is temporarily offline, then please come back a while later as we’re uploading images around the clock. And feel free to join our list-serve in the meantime if you wish to be notified when humanitarian organizations need your help in the future. No prior experience or training necessary. Anyone who knows how to use a computer mouse can become a digital humanitarian.

The SBTF, GISCorps and ESRI are members of the Digital Humanitarian Network (DHN), which my colleague Andrej Verity and I co-founded last year. The DHN serves as the official interface for direct collaboration between traditional “brick-and-mortar” humanitarian organizations and highly skilled digital volunteer networks. The SBTF Yolanda Team, spearheaded by my colleague Justine Mackinnon, for example, has also produced this map based on the triangulated results of the TweetClicker:

YolandaPH Crisis Map 2
There’s a lot of hype around the use of new technologies and social media for disaster response. So I want to be clear that our digital humanitarian operations in the Philippines have not been perfect. This means  that we’re learning (a lot) by doing (a lot). Such is the nature of innovation. We don’t have the luxury of locking ourselves up in a lab for a year to build the ultimate humanitarian technology platform. This means we have to work extra, extra hard when deploying new platforms during major disasters—because not only do we do our very best to carry out Plan A, but we often have to carry out  Plans B and C in parallel just in case Plan A doesn’t pan out. Perhaps Samuel Beckett summed it up best: “Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.”

bio

Digital Humanitarians: From Haiti Earthquake to Typhoon Yolanda

We’ve been able to process and make sense of a quarter of a million tweets in the aftermath of Typhoon Yolanda. Using both AIDR (still under development) and Twitris, we were able to collect these tweets in real-time and use automated algorithms to filter for both relevancy and uniqueness. The resulting ~55,000 tweets were then uploaded to MicroMappers (still under development). Digital volunteers from the world over used this humanitarian technology platform to tag tweets and now images from the disaster (click image below to enlarge). At one point, volunteers tagged some 1,500 tweets in just 10 minutes. In parallel, we used machine learning classifiers to automatically identify tweets referring to both urgent needs and offers of help. In sum, the response to Typhoon Yolanda is the first to make full use of advanced computing, i.e., both human computing and machine computing to make sense of Big (Crisis) Data.

ImageClicker YolandaPH

We’ve come a long way since the tragic Haiti Earthquake. There was no way we would’ve been able to pull off the above with the Ushahidi platform. We weren’t able to keep up with even a few thousand tweets a day back then, not to mention images. (Incidentally, MicroMappers can also be used to tag SMS). Furthermore, we had no trained volunteers on standby back when the quake struck. Today, not only do we have a highly experienced network of volunteers from the Standby Volunteer Task Force (SBTF) who serve as first (digital) responders, we also have an ecosystem of volunteers from the Digital Humanitarian Network (DHN). In the case of Typhoon Yolanda, we also had a formal partner, the UN Office for the Coordination of Humanitarian Affairs (OCHA), that officially requested digital humanitarian support. In other words, our efforts are directly in response to clearly articulated information needs. In contrast, the response to Haiti was “supply based” in that we simply pushed out all information that we figured might be of use to humanitarian responders. We did not have a formal partner from the humanitarian sector going into the Haiti operation.

Yolanda Prezi

What this new digital humanitarian operation makes clear is that preparedness, partnerships & appropriate humanitarian technology go a long way to ensuring that our efforts as digital humanitarians add value to the field-based operations in disaster zones. The above Prezi by SBTF co-founder Anahi (click on the image to launch the presentation) gives an excellent overview of how these digital humanitarian efforts are being coordinated in response to Yolanda. SBTF Core Team member Justine Mackinnon is spearheading the bulk of these efforts.

While there are many differences between the digital response to Haiti and Yolanda, several key similarities have also emerged. First, neither was perfect, meaning that we learned a lot in both deployments; taking a few steps forward, then a few steps back. Such is the path of innovation, learning by doing. Second, like our use of Skype in Haiti, there’s no way we could do this digital response work without Skype. Third, our operations were affected by telecommunications going offline in the hardest hit areas. We saw an 18.7% drop in relevant tweets on Saturday compared to the day before, for example. Fourth, while the (very) new technologies we are deploying are promising, they are still under development and have a long way to go. Fifth, the biggest heroes in response to Haiti were the volunteers—both from the Haitian Diaspora and beyond. The same is true of Yolanda, with hundreds of volunteers from the world over (including the Philippines and the Diaspora) mobilizing online to offer assistance.

A Filipino humanitarian worker in Quezon City, Philippines, for example, is volunteering her time on MicroMappers. As is customer care advisor from Eurostar in the UK and a fire officer from Belgium who recruited his uniformed colleagues to join the clicking. We have other volunteer Clickers from Makati (Philippines), Cape Town (South Africa), Canberra & Gold Coast (Australia), Berkeley, Brooklyn, Citrus Heights & Hinesburg (US), Kamloops (Canada), Paris & Marcoussis (France), Geneva (Switzerland), Sevilla (Spain), Den Haag (Holland), Munich (Germany) and Stokkermarke (Denmark) to name just a few! So this is as much a human story is it is one about technology. This is why online communities like MicroMappers are important. So please join our list-serve if you want to be notified when humanitarian organizations need your help.

Bio

Typhoon Yolanda: UN Needs Your Help Tagging Crisis Tweets for Disaster Response (Updated)

Final Update 14 [Nov 13th @ 4pm London]: Thank you for clicking to support the UN’s relief operations in the Philippines! We have now completed our mission as digital humanitarian volunteers. The early results of our collective online efforts are described here. Thank you for caring and clicking. Feel free to join our list-serve if you want to be notified when humanitarian organizations need your help again during the next disaster—which we really hope won’t be for a long, long time. In the meantime, our hearts and prayers go out to those affected by this devastating Typhoon.

-

The United Nations Office for the Coordination of Humanitarian Affairs (OCHA) just activated the Digital Humanitarian Network (DHN) in response to Typhoon Yolanda, which has already been described as possibly one of the strongest Category 5 storms in history. The Standby Volunteer Task Force (SBTF) was thus activated by the DHN to carry out a rapid needs & damage assessment by tagging reports posted to social media. So Ji Lucas and I at QCRI (+ Hemant & Andrew) and Justine Mackinnon from SBTF have launched MicroMappers to microtask the tagging of tweets & images. We need all the help we can get given the volume we’ve collected (and are continuing to collect). This is where you come in!

TweetClicker_PH2

You don’t need any prior experience or training, nor do you need to create an account or even login to use the MicroMappers TweetClicker. If you can read and use a computer mouse, then you’re all set to be a Digital Humanitarian! Just click here to get started. Every tweet will get tagged by 3 different volunteers (to ensure quality control) and those tweets that get identical tags will be shared with our UN colleagues in the Philippines. All this and more is explained in the link above, which will give you a quick intro so you can get started right away. Our UN colleagues need these tags to better understand who needs help and what areas have been affected.

ImageClicker YolandaPH

It only takes 3 seconds to tag a tweet or image, so if that’s all the time you have then that’s plenty! And better yet, if you also share this link with family, friends, colleagues etc., and invite them to tag along. We’ll soon be launching We have also launched the ImageClicker to tag images by level of damage. So please stay tuned. What we need is the World Wide Crowd to mobilize in support of those affected by this devastating disaster. So please spread the word. And keep in mind that this is only the second time we’re using MicroMappers, so we know it is not (yet) perfect : ) Thank you!

bio

p.s. If you wish to receive an alert next time MicroMappers is activated for disaster response, then please join the MicroMappers list-serve here. Thanks!

Previous updates:

Update 1: If you notice that all the tweets (tasks) have been completed, then please check back in 1/2 hour as we’re uploading more tweets on the fly. Thanks!

Update 2: Thanks for all your help! We are getting lots of traffic, so the Clicker is responding very slowly right now. We’re working on improving speed, thanks for your patience!

Update 3: We collected 182,000+ tweets on Friday from 5am-7pm (local time) and have automatically filtered this down to 35,175 tweets based on relevancy and uniqueness. These 35K tweets are being uploaded to the TweetClicker a few thousand tweets at a time. We’ll be repeating all this for just one more day tomorrow (Saturday). Thanks for your continued support!

Update 4: We/you have clicked through all of Friday’s 35K tweets and currently clicking through today’s 28,202 tweets, which we are about 75% of the way through. Many thanks for tagging along with us, please keep up the top class clicking, we’re almost there! (Sunday, 1pm NY time)

Update 5: Thanks for all your help! We’ll be uploading more tweets tomorrow (Monday, November 11th). To be notified, simply join this list-serve. Thanks again! [updated post on Sunday, November 10th at 5.30pm New York]

Update 6: We’ve uploaded more tweets! This is the final stretch, thanks for helping us on this last sprint of clicks!  Feel free to join our list-serve if you want to be notified when new tweets are available, many thanks! If the system says all tweets have been completed, please check again in 1/2hr as we are uploading new tweets around the clock. [updated Monday, November 11th at 9am London]

Update 7 [Nov 11th @ 1pm London]We’ve just launched the ImageClicker to support the UN’s relief efforts. So please join us in tagging images to provide rapid damage assessments to our humanitarian partners. Our TweetClicker is still in need of your clicks too. If the Clickers are slow, then kindly be patient. If all the tasks are done, please come back in 1/2hr as we’re uploading content to both clickers around the clock. Thanks for caring and helping the relief efforts. An update on the overall digital humanitarian effort is available here.

Update 8 [Nov 11th @ 6.30pm NY]We’ll be uploading more tweets and images to the TweetClicker & ImageClicker by 7am London on Nov 12th. Thank you very much for supporting these digital humanitarian efforts, the results of which are displayed here. Feel free to join our list-serve if you want to be notified when the Clickers have been fed!

Update 9 [Nov 12th @ 6.30am London]: We’ve fed both our TweetClicker and ImageClicker with new tweets and images. So please join us in clicking away to provide our UN partners with the situational awareness they need to coordinate their important relief efforts on the ground. The results of all our clicks are displayed here. Thank you for helping and for caring. If the Clickers or empty or offline temporarily, please check back again soon for more clicks.

Update 10 [Nov 12th @ 10am New York]: Were continuing to feed both our TweetClicker and ImageClicker with new tweets and images. So please join us in clicking away to provide our UN partners with the situational awareness they need to coordinate their important relief efforts on the ground. The results of all our clicks are displayed here. Thank you for helping and for caring. If the Clickers or empty or offline temporarily, please check back again soon for more clicks. Try different browsers if the tweets/images are not showing up.

Update 11 [Nov 12th @ 5pm New York]: Only one more day to go! We’ll be feeding our TweetClicker and ImageClicker with new tweets and images by 7am London on the 13th. We will phase out operations by 2pm London, so this is the final sprint. The results of all our clicks are displayed here. Thank you for helping and for caring. If the Clickers are empty or offline temporarily, please check back again soon for more clicks. Try different browsers if the tweets/images are not showing up.

Update 12 [Nov 13th @ 9am London]: This is the last stretch, Clickers! We’ve fed our TweetClicker and ImageClicker with new tweets and images. We’ll be refilling them until 2pm London (10pm Manila) and phasing out shortly thereafter. Given that MicroMappers is still under development, we are pleased that this deployment went so well considering. The results of all our clicks are displayed here. Thank you for helping and for caring. If the Clickers are empty or offline temporarily, please check back again soon for more clicks. Try different browsers if the tweets/images are not showing up.

Update 13 [Nov 13th @ 11am London]: Just 3 hours left! Our UN OCHA colleagues have just asked us to prioritize the ImageClicker, so please focus on that Clicker. We’ll be refilling the ImageClicker until 2pm London (10pm Manila) and phasing out shortly thereafter. Given that MicroMappers is still under development, we are pleased that this deployment went so well considering. The results of all our clicks are displayed here. Thank you for helping and for caring. If the ImageClicker is empty or offline temporarily, please check back again soon for more clicks. Try different browsers if images are not showing up.