Category Archives: Ushahidi

Over 2 Million Tweets from Oklahoma Tornado Automatically Processed (Updated)

Update: We have now processed a total of 2 million tweets (up from 1 million).

My colleague Hemant Purohit at QCRI has been working with us on automatically extracting needs and offers of help posted on Twitter during disasters. When the 2-mile wide, Category 4 Tornado struck Moore, Oklahoma, he immediately began to collect relevant tweets about the Tornado’s impact and applied the algorithms he developed at QCRI to extract needs and offers of help.

tornado_ok

As long-time readers of iRevolution will know, this is an approach I’ve been advocating for and blogging about for years, including the auto-matching of needs and offers. These algorithms (classifiers) will also be made available as part of our Artificial Intelligence for Disaster Response (AIDR) platform. In the meantime, we have contacted our colleagues at the American Red Cross’s Digital Operations Center (DigiOps) to offer the results of the processed data, i.e., 1,000+ tweets requesting & offering help. If you are an established organization engaged in relief efforts following the Tornado, please feel free to get in touch with us (patrick@iRevolution.net) so we can make the data available to you. 

Bio

Automatically Classifying Crowdsourced Election Reports

As part of QCRI’s Artificial Intelligence for Monitoring Elections (AIME) project, I liaised with Kaggle to work with a top notch Data Scientist to carry out a proof of concept study. As I’ve blogged in the past, crowdsourced election monitoring projects are starting to generate “Big Data” which cannot be managed or analyzed manually in real-time. Using the crowdsourced election reporting data recently collected by Uchaguzi during Kenya’s elections, we therefore set out to assess whether one could use machine learning to automatically tag user-generated reports according to topic, such as election-violence. The purpose of this post is to share the preliminary results from this innovative study, which we believe is the first of it’s kind.

uchaguzi

The aim of this initial proof-of-concept study was to create a model to classify short messages (crowdsourced election reports) into several predetermined categories. The classification models were developed by applying a machine learning technique called gradient boosting on word features extracted from the text of the election reports along with their titles. Unigrams, bigrams and the number of words in the text and title were considered in the model development. The tf-idf weighting function was used following internal validation of the model.

The results depicted above confirm that classifiers can be developed to automatically classify short election observation reports crowdsourced from the public. The classification was generated by 10-fold cross validation. Our classifier is able to correctly predict whether a report is related to violence with an accuracy of 91%, for example. We can also accurately predict  89% of reports that relate to “Voter Issues” such as registration issues and reports that indicate positive events, “Fine” (86%).

The plan for this Summer and Fall is to replicate this work for other crowdsourced election datasets from Ghana, Liberia, Nigeria and Uganda. We hope the insights gained from this additional research will reveal which classifiers and/or “super classifiers” are portable across certain countries and election types. Our hypothesis, based on related crisis computing research, is that classifiers for certain types of events will be highly portable. However, we also hypothesize that the application of most classifiers across countries will result in lower accuracy scores. To this end, our Artificial Intelligence for Monitoring Elections platform will allow election monitoring organizations (end users) to create their own classifiers on the fly and thus meet their own information needs.

bio

Big thanks to Nao for his excellent work on this predictive modeling project.

Crisis Mapping, Neogeography and the Delusion of Democratization

Professor Muki Haklay kindly shared with me this superb new study in which he questions the alleged democratization effects of Neogeography. As my colleague Andrew Turner explained in 2006, “Neogeography means ‘new geography’ and consists of a set of techniques and tools that fall outside the realm of traditional GIS, Geographic Information Systems. [...] Essentially, Neogeography is about people using and creating their own maps, on their own terms and by combining elements of an existing toolset. Neogeography is about sharing location information with friends & visitors, helping shape context, and conveying under-standing through knowledge of place.” To this end, as Muki writes, “it is routinely argued that the process of producing and using geographical information has been fundamentally democratized.” For example, as my colleague Nigel Snoad argued in 2011, “[...] Google, Microsoft and OpenStreetMap have really demo-cratized mapping.” Other CrisisMappers, including myself, have made similar arguments over the years.

neogeo1

Muki explores this assertion by delving into the various meanings of demo-cratization. He adopts the specific notion of democratization that “evokes ideas about participation, equality, the right to influence decision making, support to individual and group rights, access to resources and opportunities, etc.” With this definition in hand, Muki argues that “using this stronger interpretation of democratization reveals the limitation of current neogeographic practices and opens up the possibility of considering alternative development of technologies that can, indeed, be considered democratizing.” To explore this further, he turns to Andrew Feenberg‘s critical philosophy of technology. Feenberg identifies “four main streams of thought on the essence of technology and its linkage to society: instrumentalism, determinism, substantivism & critical theory.”

Screen Shot 2013-03-16 at 6.19.43 PM

Feenberg’s own view is constructivist, “emphasizing that technology development is humanly controlled and encapsulates values and politics; it should thus be open to democratic control and intervention.” In other words, “technology can and should be seen as a result of political negotiations that lead to its production and use. In too many cases, the complexities of technological systems are used to concentrate power within small groups of technological, financial, and political elites and to prevent the wider body of citizens from meaningful participation in shaping it and deciding what role it should have in the everyday.” Furthermore, “Feenberg highlights that technology encapsulates an ambivalence between the ‘conservation of hierarchy’, which most technologies promote and reproduce—hence the continuity in power structures in advanced capitalist societies despite technological upheaval—and ‘democratic rationalisation’, which are the aspects of new technologies that undermine existing power structures and allow new opportunities for marginalized or ignored groups to assert themselves.”

To this end, Feenberg calls for a “deep democratization” of technology as an alternative to technocracy. “Instead of popular agency appearing as an anomaly and an interference, it would be normalized and incorporated into the standard procedures of technical design.” In other words, deep democratization is about empowerment: “providing the tools that will allow increased control over the technology by those in disadvantaged and marginalized positions in society.” Muki contrasts this with neogeography, which is “mostly represented in a decon-textualised way—as the citation in the introduction from Turner’s (2006) Intro-duction to Neogeography demonstrates: it does not discuss who the people are who benefit and whether there is a deeper purpose, beyond fun, for their engage-ment in neogeography.” And so, as neogeographers would have it, since “there is nothing that prevents anyone, anytime, and anywhere, and for any purpose from using the system, democratization has been achieved.” Or maybe not. Enter the Digital Divides.

digidivide

Yes, there are multiple digital divides. Differential access to computers & comm-unication technology is just one. “Beyond this, there is secondary digital ex-clusion, which relates to the skills and abilities of people to participate in online activities beyond rudimentary browsing.” Related to this divide is the one between the “Data Haves” and the “Data Have Nots”. There is also an important divide in speed—as anyone who has worked in say Liberia will have experienced—it takes a lot longer to upload/download/transfer content than in Luxembourg. “In summary, the social, economic, structural, and technical evidence should be enough to qualify and possibly withdraw the democratization claims that are attached to neogeographic practices.”

That said, the praxis of neogeography still has democratic potential. “To address the potential of democratization within neogeographic tools, we need to return to Feenberg’s idea of deep democratization  and the ability of ordinary citizens to direct technical codes and influence them so that they can include alternative meanings and values. By doing so, we can explore the potential of neogeographic practices to support democratisation in its fuller sense. At the very least, citizens should be able to reuse existing technology and adapt it so that it can be used to their own goals and to represent their own values.” So Muki adds a “Hierarchy of Hacking” to Feeberg’s conceptual framework, i.e., the triangle below.

Screen Shot 2013-03-16 at 7.03.49 PM

While the vast majority can participate in a conversation about what to map (Meaning), only a “small technical elite within society” can contribute to “Deep Technical Hacking,” which “requires very significant technical knowledge in creating new geographic data collection tools, setting up servers, and configuring database management systems.” Muki points to Map Kibera as an example of Deep Technical Hacking. I would add that “Meaning Hacking” is often hijacked by “Deep Technical Hackers” who tend to be the ones introducing-and-controlling local neogeography projects despite their “best” intentions. But the fact is this: Deep Tech Hackers typically have little to no actual experience in community development and are often under pressure to hype up blockbuster-like successes at fancy tech conferences in the US. This may explain why most take full owner-ship over all decisions having to do with Meaning- and Use-Hacking right from the start of a project. See this blog post’s epilogue, for more on this dynamic.

One success story, however, is Liberia’s Innovation Lab (iLab). My field visit to Monrovia in 2011 made me realize just how many completely wrong assumptions I had about the use of neogeography platforms in developing countries. Instead of parachuting in and out, the co-founders of iLab became intimately familiar with the country by spending a considerable amount of time in Monrovia and outside the capital city to understand the social, political and historical context in which they were introducing neogeography. And so, while they initially expected to provide extensive training on neogeography platforms right off the bat, they quickly realized that this was the wrong approach entirely for several reasons. As Muki observers, “Because of the reduced barriers, neogeography does offer some increased level of democratization but, to fulfill this potential, it requires careful implementation that takes into account social and political aspects,” which is precisely what the team at the iLab have done and continue to do impressively well. Note that one of the co-founders is a development expert, not a technology hacker. And while the other is a hacker, he spent several years working in Liberia. (Another equally impressive success story is this one from Brazil’s Mare shantytown).

blank

I thus fully subscribe to Muki’s hacking approach and made a very similar ar-gument in this 2011 blog post: “Democratizing ICT for Development with DIY Innovation and Open Data.” I directly challenged the “participatory” nature of these supposedly democratizing technologies and in effect questioned whether Deep Technical Hackers really do let go of control vis-a-vis the hacking of “Meaning” and “Use”. While I used Ushahidi as an example of a DIY platform, it is clear from Muki’s study that Ushahidi like other neogeography platforms also falls way short of deep democratization and hack-ability. That said, as I wrote then, “it is worth remembering that the motivations driving this shift [towards neogeography] are more important than any one technology. For example, recall the principles behind the genesis of the Ushahidi platform: Democratizing information flows and access; promoting Open Data and Do it Yourself (DIY) Innovation with free, highly hackable (i.e., open source) technology; letting go of control.” In other words, the democratizing potential should not be dismissed outright even if we’re not quite there yet (or ever).

As I noted in 2011,  hackable and democratizing technologies ought to be like a “choose your own adventure game. The readers, not the authors, finish the story. They are the main characters who bring the role playing games and stories to life.” This explains why I introduced the notion a “Fischer Price Theory of Tech-nology” five years ago at this meeting with Andrew Turner and other colleagues. As argued then, “What our colleagues in the tech-world need to keep in mind is that the vast majority of our partners in the field have never taken a computer science or software engineering course. [...] The onus thus falls on the techies to produce the most simple, self-explanatory, intuitive interfaces.”

I thus argued that neogeography platforms ought to be as easy to use (and yes hack) as simple as computer games, which is why I was excited to see the latest user interface (UI) developments for OpenStreetMap (image below). Of course, as Muki has ably demonstrated, UI design is just the tip of the iceberg vis-a-vis democratization effects. But democratization is both relative and a process, and neogeography platforms are unlikely to become less democratizing over time, for instance. While some platforms still have a long road ahead with respect to reaching their perceived potential (if ever), a few instances may already have made in-roads in terms of their local political effects as argued here and in my doctoral dissertation.

OSMneogeo

Truly hackable technology, however, needs to go beyond the adventure story and Fischer Price analogies described above. The readers should have the choice of becoming authors before they even have a story in mind, while gamers should have the option of creating their own games in the first place. In other words, as Muki argues, “the artful alteration of technology beyond the goals of its original design or intent,” enables “Deep Democratization.” To this end, “Freely pro-viding the hackable building blocks for DIY Innovation is one way to let go of control and democratize [neogeography platforms],” not least if the creators can make a business out of their buildings. 

Muki concludes by noting that, “the main error in the core argument of those who promote [neogeography] as a democratic force is the assumption that, by increasing the number of people who utilise geographic information in different ways and gain access to geographic technology, these users have been em-powered and gained more political and social control. As demonstrated in this paper, neogeography has merely opened up the collection and use of this information to a larger section of the affluent, educated, and powerful part of society.”  What’s more, “The control over the information is kept, by and large, by major corporations and the participant’s labor is enrolled in the service of these corporations, leaving the issue of payback for this effort a moot point. Significantly, the primary intention of the providers of the tools is not to empower communities or to include marginalized groups, as they do not re-present a major source of revenue.” I argued this exact point here a year ago.

bio

Analyzing Tweets Posted During Mumbai Terrorist Attacks

Over 1 million unique users posted more than 2.7 million tweets in just 3 days following the triple bomb blasts that struck Mumbai on July 13, 2011. Out of these, over 68,000 tweets were “original tweets” (in contrast to retweets) and related to the bombings. An analysis of these tweets yielded some interesting patterns. (Note that the Ushahidi Map of the bombings captured ~150 reports; more here).

One unique aspect of this study (PDF) is the methodology used to assess the quality of the Twitter dataset. The number of tweets per user was graphed in order to test for a power law distribution. The graph below shows the log distri-bution of the number of tweets per user. The straight lines suggests power law behavior. This finding is in line with previous research done on Twitter. So the authors conclude that the quality of the dataset is comparable to the quality of Twitter datasets used in other peer-reviewed studies.

I find this approach intriguing because Professor Michael Spagat, Dr. Ryan Woodard and I carried out related research on conflict data back in 2006. One fascinating research question that emerges from all this, and which could be applied to twitter datasets, is whether the slope of the power law says anything about the type of conflict/disaster being tweeted about, the expected number of casualties or even the propagation of rumors.  If you’re interested in pursuing this research question (and have worked with power laws before), please do get in touch. In the meantime, I challenge the authors’ suggestion that a power law distribution necessarily says anything about the quality or reliability of the underlying data. Using the casualty data from SyriaTracker (which is also used by USAID in their official crisis maps), my colleague Dr. Ryan Woodard showed that this dataset does not follow a power law distribution—even thought it is one of the most reliable on Syria.

Syria_PL

Moving on to the content analysis of the Mumbai blast tweets:  ”The number of URLs and @-mentions in tweets increase during the time of the crisis in com-parison to what researchers have exhibited for normal circumstances.” The table below lists the top 10 URLs shared on Twitter. Inter-estingly, the link to a Google Spreadsheet was amongst the most shared resource. Created by Twitter user Nitin Sagar, the spreadsheet was used to “coordinate relief operation among people. Within hours hundreds of people registered on the sheet via Twitter. People asked for or off ered help on that spreadsheet for many hours.”

The analysis also reveals that ”the number of tweets or updates by authority users (those with large number of followers) are very less, i.e., majority of content generated on Twitter during the crisis comes from non authority users.”  In addition, tweets generated by authority users have a high level of retweets. The results also indicate that “the number of tweets generated by people with large follower base (who are generally like government owned accounts, cele-brities, media companies) were very few. Thus, the majority of content generated at the time of crisis was from unknown users. It was also observed that, though the number of posts were less by users with large number of followers, these posts registered high numbers of retweets.”

Rumors related to the blasts also spread through Twitter. For example, rumors began to circulate about a fourth bomb going off. “Some tweets even speci fied locations of 4th blast as Lemington street, Colaba and Charni. Around 500+ tweets and retweets were posted about this.” False rumors about hospital blood banks needing donations were also propagated via Twitter. “They were initiated by a user, @KapoorChetan and around 2,000 tweets and retweets were made regarding this by Twitter users.” The authors of the study believe that such false rumors and can be prevented if credible sources like the mainstream media companies and the government post updates on social media more frequently.

I did a bit of research on this and found that NDTV did use their twitter feed (which has over half-a-million followers) to counter these rumors. For example, “RT @ndtv: Mumbai police: Don’t believe rumours of more bombs. False rumours being spread deliberately.” Journalist Sonal Kalra also acted to counter rumors: “RT @sonalkalra: BBMs about bombs found in Delhi are FALSE. Pls pls don’t spread rumours. #mumbaiblasts.”

In conclusion, the study considers the “privacy threats during the Twitter activity after the blasts. People openly tweeted their phone numbers on social media websites like Twitter, since at such moment of crisis people wished to reach out to help others. But, long after the crisis was over, such posts still remained publicly available on the Internet.” In addition, “people also openly posted their blood group, home address, etc. on Twitter to off er help to victims of the blasts.” The Ushahidi Map also includes personal information. These data privacy and security issues continue to pose major challenges vis-a-vis the use of social media for crisis response.

Bio

See also: Did Terrorists Use Twitter to Increase Situational Awareness? [Link]

Haiti: Lies, Damned Lies and Crisis Mapping

You’d think there was some kind of misinformation campaign going on about the Ushahidi-Haiti Crisis Map given the number of new lies that are still being manu-factured even though it has been over three years since the earthquake. Please, if you really want a professional, independent and rigorous account of the project, read this evaluation. The findings are mixed but the report remains the only comprehensive, professional and independent evaluation of the Ushahidi-Haiti and 4636 efforts. So if you have questions about the project, please read the report and/or contact the evaluators directly.

Screen Shot 2013-02-25 at 2.10.47 AM

In the meantime, I’ve decided to collect the most ridiculous lies & rumors and post my all-time favorites below.

1. “Mission 4636  & Haitian volunteers very strongly opposed the publishing of 4636 SMS’s on the Ushahidi-Haiti Crisis Map given data privacy concerns.”

Robert, the person responsible for Mission 4636, agreed (in writing) to publish the SMS’s after two lawyers noted that there was implied consent to make these messages public. The screenshot of the email below clearly proves this. Further-more, he and I co-authored this peer-reviewed study several months after the earthquake to document the lessons learned from the SMS response in Haiti. Surely if one of us had heard about these concerns from the Diaspora, we would have known this and reconsidered the publishing of the SMS’s. We would also have written this up as a major issue in our study. Moreover, the independent and professional evaluators referred to above would also have documented this major issue if it were true.

Screen Shot 2013-02-26 at 9.35.59 AM

I, for one, did not receive a single email from anyone involved in Mission 4636 demanding that the SMS’s not be made public. None of the Boston-based Haitian volunteers who I met in person ever asked for the messages to remain con-fidential; nor did Haitian Diaspora journalists who interviewed us or the many Haitians who called into the radio interviews we participated in ask for the messages to remain secret. Also, the joint decision to (only) map the most urgent and actionable life-and-death messages was supported by a number of humani-tarian colleagues who agreed that the risks of making this information public were minimal vis-à-vis the Do No Harm principle.

On a practical note, time was a luxury we did not have; an entire week had already passed since the earthquake and we were already at the tail end of the search and rescue phase. This meant that literally every hour counted for potential survivors still trapped under the rubble. There was no time to second-guess the lawyers or to organize workshops on the question. Making the most urgent and actionable life-and-death text messages public meant that the Haitian Diaspora, which was incredibly active in the response, could use that information to help coordinate efforts. NGOs in Haiti could also make use of this information—not to mention the US Marine Corps, which claimed to have saved hundreds of lives thanks to the Ushahidi-Haiti Crisis Map.

Crisis Mapping can be risky business, there’s no doubt about that. Sometimes tough-but-calculated decisions are needed. If one of the two lawyers had opined that the messages should not be made public, then the SMS’s would not have been published, end of story. In any case, the difficulties we faced during this crisis mapping response to Haiti is precisely why I’ve been working hard with GSMA’s Disaster Response Program to create this SMS Code of Conduct. I have also been collaborating directly with the International Committee of the Red Cross (ICRC) to update Data Privacy and Protection Protocols so they include guidelines on social media use and crisis mapping. This new report will be officially launched in Geneva this April followed by a similar event in DC.

2. “Mission 4636 was a completely separate and independent initiative to the Ushahidi Haiti Crisis Map.”

Then why was Josh Nesbit looking for an SMS solution specifically for Ushahidi? The entire impetus for 4636 was the Haiti Crisis Map. Thanks to his tweet, Josh was put in touch with a contact at Digicel Haiti in Port-au-Prince. Several days later, the 4636 short code was set up and integrated with the Ushahidi platform.

jn4636

3. “The microtasking platform developed by Ushahidi to translate the text messages during the first two weeks of operation was built by Tim Schwartz, i.e., not Ushahidi.”

Tim Schwartz is a good friend and wonderful colleague. So when I came across this exciting new rumor, I emailed him right away to thank him: “I’m super surprised since no one ever told me this before. If it is indeed true, then I owe you a huge huge thanks!!” His reply: “Well… not exactly:) Brian [from Ushahidi] took our code from the haitianquake.com and modified it to make the base of 4636. Then I came in and wrote the piece that let volunteers translate missing persons messages and put them into Google Person Finder. Brian definitely wrote the original volunteer part for 4636. He’s the rockstar:)”

4. “Digital Democracy (Dd) developed all the workflows for the Ushahidi-Haiti Crisis Map and also trained the majority of volunteers.”

Dd’s co-founder Emily Jacobi is a close friend and trusted colleague. So I emailed her about this fun new rumor back in October to see if I had somehow missed something. Emily replied: “It’s totally ludicrous to claim that Dd solely set up any of those processes. I do think we played an important role in helping to inform, document & systematize those workflows, which is a world away from claiming sole or even lead ownership of any of it.” Indeed, the workflows kept changing on a daily basis and hundreds of volunteers were trained in person or online–often several times a day. That said, Dd absolutely took the lead in crafting the work-flows & training the bulk of volunteers who spearheaded the Chile Crisis Map. I recommend reading up on Dd’s awesome projects in Haiti and worldwide here.

5. “FEMA Administrator Craig Fugate’s comment below about the Ushahidi Haiti Crisis Map was actually not about the Ushahidi project. Craig was confused and was actually referring to the Humanitarian OpenStreet Map (OSM) of Haiti.”

Again, I was stunned, but in a good way. Kate Chapman, the director of Humani-tarian OpenStreetMap, is a good friend and trusted colleague, so I emailed her the following: “I still hear all kinds of rumors about Haiti but this is the *first* time I’ve come across this one and if this is indeed true then goodness gracious I really need to know so I can give credit where credit is due!” Her reply? She too had never heard this claim before. I trust her 100% so if she ever does tell me that this new rumor is true, I’ll be the first to blog and tweet about it. I’m a huge fan of Humanitarian OpenStreetMap, they really do remarkable work, which is why I included 3 of their projects as case studies in a book chapter I just sub-mitted for publication. In any event, I fully share Kate’s feelings on the rumors: “My feelings on anything that had to do with Haiti is it doesn’t really matter anymore. It has been 2 and a half years. Let’s look on to preparedness and how to improve.” Wise words from a wise woman.

CraigFEMAtweet

6. “Sabina Carlson who acted as the main point of contact between the Ushahidi Haiti project and the Haitian Diaspora also spearheaded the translation efforts and is critical of her Ushahidi Haiti Team members and in particular Patrick Meier for emphasizing the role of international actors and ignoring the Haitian Diaspora.”

This is probably one of the strangest lies yet. Everyone in Boston knows full well that Sabina was not directly focused on translation but rather on outreach and partnership building with the Haitian Diaspora. Sabina, who is a treasured friend, emailed me (out of the blue) when she heard about some of the poisonous rumors circulating. “This was a shock to me,” she wrote, “I would never say anything to put you down, Patrick, and I’m upset that my words were mis-interpreted and used to do just that.” She then detailed exactly how the lie was propagated and by whom (she has the entire transcript).

The fact is this: none of us in Boston ever sought to portray the Diaspora as insignificant or to downplay their invaluable support. Why in the world would we ever do that? Robert and I detailed the invaluable role played by the Diaspora in our peer-reviewed study, for example. Moreover, I invited Sabina to join our Ushahidi-Haiti team precisely because the Diaspora were already responding in amazing ways and I knew they’d stay the course after the end of the emergency phase—we wanted to transfer full ownership of the Haiti Crisis Map to Haitian hands.  In sum, it was crystal clear to every single one of us that Sabina was the perfect person to take on this very important responsibility. She represented the voice and interests of Haitians with incredible agility, determination and intell-igence throughout our many months of work together, both in Boston and Haiti.

bio

Launching: SMS Code of Conduct for Disaster Response

Shortly after the devastating Haiti Earthquake of January 12, 2010, I published this blog post on the urgent need for an SMS code of conduct for disaster response. Several months later, I co-authored this peer-reviewed study on the lessons learned from the unprecedented use of SMS following the Haiti Earth-quake. This week, at the Mobile World Congress (MWC 2013) in Barcelona, GSMA’s Disaster Response Program organized two panels on mobile technology for disaster response and used the event to launch an official SMS Code of Conduct for Disaster Response (PDF). GSMA members comprise nearly 800 mobile operators based in more than 220 countries.

Screen Shot 2013-02-18 at 2.27.32 PM

Thanks to Kyla Reid, Director for Disaster Response at GSMA, and to Souktel’s Jakob Korenblummy calls for an SMS code of conduct were not ignored. The three of us spent a considerable amount of time in 2012 drafting and re-drafting a detailed set of principles to guide SMS use in disaster response. During this process, we benefited enormously from many experts on the mobile operators side and the humanitarian community; many of whom are at MWC 2013 for the launch of the guidelines. It is important to note that there have been a number of parallel efforts that our combined work has greatly benefited from. The Code of Conduct we launched this week does not seek to duplicate these important efforts but rather serves to inform GSMA members about the growing importance of SMS use for disaster response. We hope this will help catalyze a closer relationship between the world’s leading mobile operators and the international humanitarian community.

Since the impetus for this week’s launch began in response to the Haiti Earth-quake, I was invited to reflect on the crisis mapping efforts I spearheaded at the time. (My slides for the second panel organized by GSMA are available here. My more personal reflections on the 3rd year anniversary of the earthquake are posted here). For several weeks, digital volunteers updated the Ushahidi-Haiti Crisis Map (pictured above) with new information gathered from hundreds of different sources. One of these information channels was SMS. My colleague Josh Nesbit secured an SMS short code for Haiti thanks to a tweet he posted at 1:38pm on Jan 13th (top left in image below). Several days later, the short code (4636) was integrated with the Ushahidi-Haiti Map.

Screen Shot 2013-02-18 at 2.40.09 PM

We received about 10,000 text messages from the disaster-affected population during the during the Search and Rescue phase. But we only mapped about 10% of these because we prioritized the most urgent and actionable messages. While mapping these messages, however, we had to address a critical issue: data privacy and protection. There’s an important trade-off here: the more open the data, the more widely useable that information is likely to be for professional disaster responders, local communities and the Diaspora—but goodbye privacy.

Time was not a luxury we had; an an entire week had already passed since the earthquake. We were at the tail end of the search and rescue phase, which meant that literally every hour counted for potential survivors still trapped under the rubble. So we immediately reached out to 2 trusted lawyers in Boston, one of them a highly reputable Law Professor at The Fletcher School of Law and Diplomacy who also a specialist on Haiti. You can read the lawyers’ written email replies along with the day/time they were received on the right-hand side of the slide. Both lawyers opined that consent was implied vis-à-vis the publishing of personal identifying information. We shared this opinion with all team members and partners working with us. We then made a joint decision 24 hours later to move ahead and publish the full content of incoming messages. This decision was supported by an Advisory Board I put together comprised of humanitarian colleagues from the Harvard Humanitarian Initiative who agreed that the risks of making this info public were minimal vis-à-vis the principle of Do No HarmUshahidi thus launched a micro-tasking platform to crowdsource the translation efforts and hosted this on 4636.Ushahidi.com [link no longer live], which volunteers from the Diaspora used to translate the text messages.

I was able to secure a small amount of funding in March 2010 to commission a fully independent evaluation of our combined efforts. The project was evaluated a year later by seasoned experts from Tulane University. The results were mixed. While the US Marine Corps publicly claimed to have saved hundreds of lives thanks to the map, it was very hard for the evaluators to corroborate this infor-mation during their short field visit to Port-au-Prince more than 12 months after the earthquake. Still, this evaluation remains the only professional, independent and rigorous assessment of Ushahidi and 4636 to date.

Screen Shot 2013-02-25 at 2.10.47 AM

The use of mobile technology for disaster response will continue to increase for years to come. Mobile operators and humanitarian organizations must therefore be pro-active in managing this increase demand by ensuring that the technology is used wisely. I, for one, never again want to spend 24+ precious hours debating whether or not urgent life-and-death text messages can or cannot be mapped because of uncertainties over data privacy and protection—24 hours during a Search and Rescue phase is almost certain to make the difference between life and death. More importantly, however, I am stunned that a bunch of volunteers with little experience in crisis response and no affiliation whatsoever to any established humanitarian organization were able to secure and use an official SMS short code within days of a major disaster. It is little surprise that we made mistakes. So a big thank you to Kyla and Jakob for their leadership and perseverance in drafting and launching GSMA’s official SMS Code of Conduct to make sure the same mistakes are not made again.

While the document we’ve compiled does not solve every possible challenge con-ceivable, we hope it is seen as a first step towards a more informed and responsible use of SMS for disaster response. Rest assured that these guidelines are by no means written in stone. Please, if you have any feedback, kindly share them in the comments section below or privately via email. We are absolutely committed to making this a living document that can be updated.

To connect this effort with the work that my CrisisComputing Team and I are doing at QCRI, our contact at Digicel during the Haiti response had given us the option of sending out a mass SMS broadcast to their 2 million subscribers to get the word out about 4636. (We had thus far used local community radio stations). But given that we were processing incoming SMS’s manually, there was no way we’d be able to handle the increased volume and velocity of incoming text messages following the SMS blast. So my team and I are exploring the use of advanced computing solutions to automatically parse and triage large volumes of text messages posted during disasters. The project, which currently uses Twitter, is described here in more detail.

bio

The World at Night Through the Eyes of the Crowd

Ushahidi has just uploaded the location of all CrowdMap reports to DevSeed’s awesome MapBox and the result looks gorgeous. Click this link to view the map below in an interactive, full-browser window. Ushahidi doesn’t disclose the actual number of reports depicted, only the number of maps that said reports have been posted to and the number of countries that CrowdMaps have been launched for. But I’m hoping they’ll reveal that figure soon as well. (Update from Ushahidi: This map shows the 246,323 unique locations used for reports from the launch of Crowdmap on Aug 9, 2010 to Jan 18, 2013).

Screen Shot 2013-02-06 at 3.10.38 AM

In any event, I’ve just emailed my colleagues at Ushahidi to congratulate them and ask when their geo-dataset will be made public since they didn’t include a link to said dataset in their recent blog post. I’ll be sure to let readers know in the comments section as soon as I get a reply. There are a plethora of fascinating research questions that this dataset could potentially help us answer. I’m really excited and can’t wait for my team and I at QCRI to start playing with the data. I’d also love to see this static map turned into a live map; one that allows users to actually click on individual reports as they get posted to a CrowdMap and to display the category (or categories) they’ve been tagged with. Now that would be just be so totally über cool—especially if/when Ushahidi opens up that data to the public, even if at a spatially & temporally aggregated level.

For more mesmerizing visualizations like this one, see my recent blog post entitled “Social Media: Pulse of the Planet?” which is also cross-posted on the National Geographic blog here. In the meantime, I’m keeping my fingers crossed that Ushahidi will embrace an Open Data policy from here on out and highly recommend the CrowdGlobe Report to readers interested in learning more about CrowdMap and Ushahidi.

bio

Why Ushahidi Should Embrace Open Data

“This is the report that Ushahidi did not want you to see.” Or so the rumors in certain circles would have it. Some go as far as suggesting that Ushahidi tried to burry or delay the publication. On the other hand, some rumors claim that the report was a conspiracy to malign and discredit Ushahidi. Either way, what is clear is this: Ushahidi is an NGO that prides itself in promoting transparency & accountability; an organization prepared to take risks—and yes fail—in the pursuit of this  mission.

The report in question is CrowdGlobe: Mapping the Maps. A Meta-level Analysis of Ushahidi & Crowdmap. Astute observers will discover that I am indeed one of the co-authors. Published by Internews in collaboration with George Washington University, the report (PDF) reveals that 93% of 12,000+ Crowdmaps analyzed had fewer than 10 reports while a full 61% of Crowdmaps had no reports at all. The rest of the findings are depicted in the infographic below (click to enlarge) and eloquently summarized in the above 5-minute presentation delivered at the 2012 Crisis Mappers Conference (ICCM 2012).

Infographic_2_final (2)

Back in 2011, when my colleague Rob Baker (now with Ushahidi) generated the preliminary results of the quantitative analysis that underpins much of the report, we were thrilled to finally have a baseline against which to measure and guide the future progress of Ushahidi & Crowdmap. But when these findings were first publicly shared (August 2012), they were dismissed by critics who argued that the underlying data was obsolete. Indeed, much of the data we used in the analysis dates back to 2010 and 2011. Far from being obsolete, however, this data provides a baseline from which the use of the platform can be measured over time. We are now in 2013 and there are apparently 36,000+ Crowdmaps today rather than just 12,000+.

To this end, and as a member of Ushahidi’s Advisory Board, I have recommended that my Ushahidi colleagues run the same analysis on the most recent Crowdmap data in order to demonstrate the progress made vis-a-vis the now-outdated public baseline. (This analysis takes no more than an hour a few days to carry out). I also strongly recommend that all this anonymized meta-data be made public on a live dashboard in the spirit of open data and transparency. Ushahidi, after all, is a public NGO funded by some of the biggest proponents of open data and transparency in the world.

Embracing open data is one of the best ways for Ushahidi to dispel the harmful rumors and conspiracy theories that continue to swirl as a result of the Crowd-Globe report. So I hope that my friends at Ushahidi will share their updated analysis and live dashboard in the coming weeks. If they do, then their bold support of this report and commitment to open data will serve as a model for other organizations to emulate. If they’ve just recently resolved to make this a priority, then even better.

In the meantime, I look forward to collaborating with the entire Ushahidi team on making the upcoming Kenyan elections the most transparent to date. As referenced in this blog post, the Standby Volunteer Task Force (SBTF) is partnering with the good people at PyBossa to customize an awesome micro-tasking platform that will significantly facilitate and accelerate the categorization and geo-location of reports submitted to the Ushahidi platform. So I’m working hard with both of these outstanding teams to make this the most successful, large-scale microtasking effort for election monitoring yet. Now lets hope for everyone’s sake that the elections remain peaceful. Onwards!

Personal Reflections: 3 Years After the Haiti Earthquake

The devastating earthquake that struck Port-au-Prince on January 12, 2010 killed as many as 200,000 people. My fiancée and five close friends were in Haiti at the time and narrowly escaped a collapsing building. They were some of the lucky few survivors. But I had no knowledge that they had survived until 8 hours or so after the earthquake because we were unable get any calls through. The Haiti Crisis Map I subsequently spearheaded still stands as the most psycho-logically and emotionally difficult project I’ve ever been a part of.

The heroes of this initiative and the continuing source of my inspiration today were the hundreds and hundreds of volunteers who ensured the Haiti Crisis Map remained live for so many weeks. The majority of these volunteers were of course the Haitian Diaspora as well as Haitians in country. I had the honor of meeting and working with one of these heroes while in Port-au-Prince, Kurt Jean-Charles, the CEO of the Haitian software company Solutions.ht. I invited Kurt to give the Keynote at the 2010 International Crisis Mappers Conference (ICCM 2010) and highly recommend watching the video above. Kurt speaks directly from the heart.

HaitianDiaspora

Another personal hero of mine (pictured above) is Sabina Carlson—now Sabina Carlson Robillard following her recent wedding to Louino in Port-au-Prince! She volunteered as the Haitian Diaspora Liaison for the Haiti Crisis Map and has been living in Cité Soleil ever since. Needless to say, she continues to inspire all of us who have had the honor of working with her and learning from her.

Finally, but certainly not (!) least, the many, many hundreds of amazing volun-teers who tirelessly translated tens of thousands of text messages for this project. Thanks to you, some 1,500 messages from the disaster-affected population were added to the live crisis map of Haiti. This link points to the only independent, rigorous and professional evaluation of the project that exists. I highly reco-mmend reading this report as it comprises a number of important lessons learned in crisis mapping and digital humanitarian response.

Fonkoze

In the meantime, please consider making a donation to Fonkoze, an outstanding local organization committed to the social and economic improvement of the Haitian poor. Fonkoze is close to my heart not only because of the great work that they do but also because its staff and CEO were the ones who ensured the safe return of my fiancée and friends after the earthquake. In fact, my fiancée has continued to collaborate with them ever since and still works on related projects in Haiti. She is headed back to Port-au-Prince this very weekend. To make a tax deductible donation to Fonkoze, please visit this link. Thank you.

My thoughts & prayers go out to all those who lost loved ones in Haiti years ago.

Could Social Media Have Prevented the Largest Mass Poisoning of a Population in History?

I just finished reading a phenomenal book. Resilience: Why Things Bounce Back, was co-authored by my good friend Andrew Zolli of PopTech fame and his won-derful colleague Ann Marie Healey. I could easily write several dozen blog posts on this brilliant book. Consider this the first of possibly many more posts to follow. Some will summarize and highlight insights that really resonated with me while others like the one below will use the book as a spring board to explore related questions and themes.

In one of the many interesting case studies that Andrew and Ann discuss in their book, the following one may very well be the biggest #FAIL in all of development history. The vast majority of Bangladeshis did not have access to clean water during the early 1970s, which contributed to numerous diseases that claimed hundreds of thousands of lives every year. So UNICEF launched a “nationwide program to sink shallow tube wells across the country. Once a small hand pump was installed to the top of the tube, clean water rose quickly to the surface.”

By the end of the 1970s, over 300,000 tube wells had been installed and some 10 million more went into operation by the late 1990s. With access to clean water, the child mortality rate dropped by more than half, from 24% to less than 10%. UNICEF’s solution was thus “touted as a model for South Asia and the world.” In the early 1980s, however, signs of widespread arsenic poising began to appear across the country. “UNICEF had mistaken deep water for clean water and never tested its tube wells for this poison.” WHO soon predicted that “one in a hundred Bangladeshis drinking from the contaminated wells would die from an arsenic-related cancer.” The government estimated that about half of the 10 million wells were contaminated. A few years later, WHO announced that Bangladesh was “facing the largest mass poisoning of a population in history.”

In a typical move that proves James Scott’s thesis Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed, the Bangladeshi government partnered with the World Bank to paint the sprout of each well red if the water was contaminated and green if safe to drink. Five years and over $40 million later, the project had only been able to test half of the 10 million wells. “Officially, this intervention was hailed as almost instantaneous success.” But the widespread negative socio-economic impact and community-based conflicts that resulted from this one-off, top-down intervention calls into question the purported success of this intervention.

As Andrew and Ann explain, water use in Bangladesh (like many other countries) starts and ends with women and girls. “They are the ones who will determine if a switch to a green well is warranted because they are the ones who fetch the water in water numerous times a day.” The location of these green wells will largely determine “whether or not women and girls can access them in a way that is deemed socially appropriate.” As was the case with many of these wells, “the religious and cultural norms impeded a successful switch.”

In addition, “negotiating use of someone else’s green well was an act fraught with potential conflict.” As a result, some still used water from red-painted wells. In fact, “reports started to come in of families and communities chipping away at the red paint on their wells,” with some even repainting theirs with green. Such was the stigma of being a family linked to a red well. Indeed, “young girls living within the vicinity of contaminated wells [recall that there were an estimated 5 million such wells] suffered from diminishing marriage prospects, if they were able to marry at all.” In addition, because the government was unable to provide alternative sources of clean water for half of the communities with a red well, “many women and girls returned to surface water sources like ponds and lakes, significantly more likely to be contaminated with fecal pathogens.” As a result, “researchers estimated that abandonment of shallow tube wells increased a household’s risk of diarrheal disease by 20%.”

In 2009, a water quality survey carried out by the government found that “approximately 20 million people were still being exposed to excessive quantities of arsenic.” And so, “while the experts and politicians discuss how to find a solution for the unintended consequences of the intervention, the people of Bangladesh continue bringing their buckets to the wells while crossing their fingers behind their backs.”

I have several questions (and will omit the ones that start with WTF?). Could social media have mitigated this catastrophic disaster? It took an entire decade for UNICEF and the Bangladeshi government to admit that massive arsenic poisoning was taking place. And even then, when UNICEF finally responded to the crisis in 1998, they said “We are wedded to safe water, not tube wells, but at this time tube wells remain a good, affordable idea and our program will go on.” By then it was too late anyway since arsenic in the wells had “found their way into the food supply. Rice irrigated with the tube wells was found to contain more than nine times the normal amount of arsenic. Rice concentrated the poison, even if one managed to avoid drinking contaminated well water, concentrated amounts would just up in one’s food.”

Could social media—had they existed in the 1980s—been used to support the early findings published by local scientists 15 years before UNICEF publicly recognized (but still ignored) the crisis? Could scientists and activists have launched a public social media campaign to name and shame? Could hundreds of pictures posted on Flickr and videos uploaded to YouTube made a difference by directly revealing the awful human consequences of arsenic poisoning?

Could an Ushahidi platform powered by FrontlineSMS have been used to create a crowdsourced complaints mechanism? Could digital humanitarian volunteers from the Standby Volunteer Task Force (SBTF) have worked with local counterparts to create a live country-wide map of concerns posted anonymously by girls and women across thousands of communities in Bangladesh? Could an interactive voice response (IVR) system like this one been set up to address concerns and needs of illiterate individuals? Could a PeaceTXT approach have been used to catalyze behavior change? Can these technologies build more resilient societies that allow them to bounce back from crises like these?

And since mass arsenic poisoning is still happening in Bangladesh today, 40 years after UNICEF’s first intervention, are initiatives like the ones described above being tried at all?