Tag Archives: Haiti

Digital Humanitarians: From Haiti Earthquake to Typhoon Yolanda

We’ve been able to process and make sense of a quarter of a million tweets in the aftermath of Typhoon Yolanda. Using both AIDR (still under development) and Twitris, we were able to collect these tweets in real-time and use automated algorithms to filter for both relevancy and uniqueness. The resulting ~55,000 tweets were then uploaded to MicroMappers (still under development). Digital volunteers from the world over used this humanitarian technology platform to tag tweets and now images from the disaster (click image below to enlarge). At one point, volunteers tagged some 1,500 tweets in just 10 minutes. In parallel, we used machine learning classifiers to automatically identify tweets referring to both urgent needs and offers of help. In sum, the response to Typhoon Yolanda is the first to make full use of advanced computing, i.e., both human computing and machine computing to make sense of Big (Crisis) Data.

ImageClicker YolandaPH

We’ve come a long way since the tragic Haiti Earthquake. There was no way we would’ve been able to pull off the above with the Ushahidi platform. We weren’t able to keep up with even a few thousand tweets a day back then, not to mention images. (Incidentally, MicroMappers can also be used to tag SMS). Furthermore, we had no trained volunteers on standby back when the quake struck. Today, not only do we have a highly experienced network of volunteers from the Standby Volunteer Task Force (SBTF) who serve as first (digital) responders, we also have an ecosystem of volunteers from the Digital Humanitarian Network (DHN). In the case of Typhoon Yolanda, we also had a formal partner, the UN Office for the Coordination of Humanitarian Affairs (OCHA), that officially requested digital humanitarian support. In other words, our efforts are directly in response to clearly articulated information needs. In contrast, the response to Haiti was “supply based” in that we simply pushed out all information that we figured might be of use to humanitarian responders. We did not have a formal partner from the humanitarian sector going into the Haiti operation.

Yolanda Prezi

What this new digital humanitarian operation makes clear is that preparedness, partnerships & appropriate humanitarian technology go a long way to ensuring that our efforts as digital humanitarians add value to the field-based operations in disaster zones. The above Prezi by SBTF co-founder Anahi (click on the image to launch the presentation) gives an excellent overview of how these digital humanitarian efforts are being coordinated in response to Yolanda. SBTF Core Team member Justine Mackinnon is spearheading the bulk of these efforts.

While there are many differences between the digital response to Haiti and Yolanda, several key similarities have also emerged. First, neither was perfect, meaning that we learned a lot in both deployments; taking a few steps forward, then a few steps back. Such is the path of innovation, learning by doing. Second, like our use of Skype in Haiti, there’s no way we could do this digital response work without Skype. Third, our operations were affected by telecommunications going offline in the hardest hit areas. We saw an 18.7% drop in relevant tweets on Saturday compared to the day before, for example. Fourth, while the (very) new technologies we are deploying are promising, they are still under development and have a long way to go. Fifth, the biggest heroes in response to Haiti were the volunteers—both from the Haitian Diaspora and beyond. The same is true of Yolanda, with hundreds of volunteers from the world over (including the Philippines and the Diaspora) mobilizing online to offer assistance.

A Filipino humanitarian worker in Quezon City, Philippines, for example, is volunteering her time on MicroMappers. As is customer care advisor from Eurostar in the UK and a fire officer from Belgium who recruited his uniformed colleagues to join the clicking. We have other volunteer Clickers from Makati (Philippines), Cape Town (South Africa), Canberra & Gold Coast (Australia), Berkeley, Brooklyn, Citrus Heights & Hinesburg (US), Kamloops (Canada), Paris & Marcoussis (France), Geneva (Switzerland), Sevilla (Spain), Den Haag (Holland), Munich (Germany) and Stokkermarke (Denmark) to name just a few! So this is as much a human story is it is one about technology. This is why online communities like MicroMappers are important. So please join our list-serve if you want to be notified when humanitarian organizations need your help.

Bio

Haiti: Lies, Damned Lies and Crisis Mapping

You’d think there was some kind of misinformation campaign going on about the Ushahidi-Haiti Crisis Map given the number of new lies that are still being manu-factured even though it has been over three years since the earthquake. Please, if you really want a professional, independent and rigorous account of the project, read this evaluation. The findings are mixed but the report remains the only comprehensive, professional and independent evaluation of the Ushahidi-Haiti and 4636 efforts. So if you have questions about the project, please read the report and/or contact the evaluators directly.

Screen Shot 2013-02-25 at 2.10.47 AM

In the meantime, I’ve decided to collect the most ridiculous lies & rumors and post my all-time favorites below.

1. “Mission 4636  & Haitian volunteers very strongly opposed the publishing of 4636 SMS’s on the Ushahidi-Haiti Crisis Map given data privacy concerns.”

Robert, the person responsible for Mission 4636, agreed (in writing) to publish the SMS’s after two lawyers noted that there was implied consent to make these messages public. The screenshot of the email below clearly proves this. Further-more, he and I co-authored this peer-reviewed study several months after the earthquake to document the lessons learned from the SMS response in Haiti. Surely if one of us had heard about these concerns from the Diaspora, we would have known this and reconsidered the publishing of the SMS’s. We would also have written this up as a major issue in our study. Moreover, the independent and professional evaluators referred to above would also have documented this major issue if it were true.

Screen Shot 2013-02-26 at 9.35.59 AM

I, for one, did not receive a single email from anyone involved in Mission 4636 demanding that the SMS’s not be made public. None of the Boston-based Haitian volunteers who I met in person ever asked for the messages to remain con-fidential; nor did Haitian Diaspora journalists who interviewed us or the many Haitians who called into the radio interviews we participated in ask for the messages to remain secret. Also, the joint decision to (only) map the most urgent and actionable life-and-death messages was supported by a number of humani-tarian colleagues who agreed that the risks of making this information public were minimal vis-à-vis the Do No Harm principle.

On a practical note, time was a luxury we did not have; an entire week had already passed since the earthquake and we were already at the tail end of the search and rescue phase. This meant that literally every hour counted for potential survivors still trapped under the rubble. There was no time to second-guess the lawyers or to organize workshops on the question. Making the most urgent and actionable life-and-death text messages public meant that the Haitian Diaspora, which was incredibly active in the response, could use that information to help coordinate efforts. NGOs in Haiti could also make use of this information—not to mention the US Marine Corps, which claimed to have saved hundreds of lives thanks to the Ushahidi-Haiti Crisis Map.

Crisis Mapping can be risky business, there’s no doubt about that. Sometimes tough-but-calculated decisions are needed. If one of the two lawyers had opined that the messages should not be made public, then the SMS’s would not have been published, end of story. In any case, the difficulties we faced during this crisis mapping response to Haiti is precisely why I’ve been working hard with GSMA’s Disaster Response Program to create this SMS Code of Conduct. I have also been collaborating directly with the International Committee of the Red Cross (ICRC) to update Data Privacy and Protection Protocols so they include guidelines on social media use and crisis mapping. This new report will be officially launched in Geneva this April followed by a similar event in DC.

2. “Mission 4636 was a completely separate and independent initiative to the Ushahidi Haiti Crisis Map.”

Then why was Josh Nesbit looking for an SMS solution specifically for Ushahidi? The entire impetus for 4636 was the Haiti Crisis Map. Thanks to his tweet, Josh was put in touch with a contact at Digicel Haiti in Port-au-Prince. Several days later, the 4636 short code was set up and integrated with the Ushahidi platform.

jn4636

3. “The microtasking platform developed by Ushahidi to translate the text messages during the first two weeks of operation was built by Tim Schwartz, i.e., not Ushahidi.”

Tim Schwartz is a good friend and wonderful colleague. So when I came across this exciting new rumor, I emailed him right away to thank him: “I’m super surprised since no one ever told me this before. If it is indeed true, then I owe you a huge huge thanks!!” His reply: “Well… not exactly:) Brian [from Ushahidi] took our code from the haitianquake.com and modified it to make the base of 4636. Then I came in and wrote the piece that let volunteers translate missing persons messages and put them into Google Person Finder. Brian definitely wrote the original volunteer part for 4636. He’s the rockstar:)”

4. “Digital Democracy (Dd) developed all the workflows for the Ushahidi-Haiti Crisis Map and also trained the majority of volunteers.”

Dd’s co-founder Emily Jacobi is a close friend and trusted colleague. So I emailed her about this fun new rumor back in October to see if I had somehow missed something. Emily replied: “It’s totally ludicrous to claim that Dd solely set up any of those processes. I do think we played an important role in helping to inform, document & systematize those workflows, which is a world away from claiming sole or even lead ownership of any of it.” Indeed, the workflows kept changing on a daily basis and hundreds of volunteers were trained in person or online–often several times a day. That said, Dd absolutely took the lead in crafting the work-flows & training the bulk of volunteers who spearheaded the Chile Crisis Map. I recommend reading up on Dd’s awesome projects in Haiti and worldwide here.

5. “FEMA Administrator Craig Fugate’s comment below about the Ushahidi Haiti Crisis Map was actually not about the Ushahidi project. Craig was confused and was actually referring to the Humanitarian OpenStreet Map (OSM) of Haiti.”

Again, I was stunned, but in a good way. Kate Chapman, the director of Humani-tarian OpenStreetMap, is a good friend and trusted colleague, so I emailed her the following: “I still hear all kinds of rumors about Haiti but this is the *first* time I’ve come across this one and if this is indeed true then goodness gracious I really need to know so I can give credit where credit is due!” Her reply? She too had never heard this claim before. I trust her 100% so if she ever does tell me that this new rumor is true, I’ll be the first to blog and tweet about it. I’m a huge fan of Humanitarian OpenStreetMap, they really do remarkable work, which is why I included 3 of their projects as case studies in a book chapter I just sub-mitted for publication. In any event, I fully share Kate’s feelings on the rumors: “My feelings on anything that had to do with Haiti is it doesn’t really matter anymore. It has been 2 and a half years. Let’s look on to preparedness and how to improve.” Wise words from a wise woman.

CraigFEMAtweet

6. “Sabina Carlson who acted as the main point of contact between the Ushahidi Haiti project and the Haitian Diaspora also spearheaded the translation efforts and is critical of her Ushahidi Haiti Team members and in particular Patrick Meier for emphasizing the role of international actors and ignoring the Haitian Diaspora.”

This is probably one of the strangest lies yet. Everyone in Boston knows full well that Sabina was not directly focused on translation but rather on outreach and partnership building with the Haitian Diaspora. Sabina, who is a treasured friend, emailed me (out of the blue) when she heard about some of the poisonous rumors circulating. “This was a shock to me,” she wrote, “I would never say anything to put you down, Patrick, and I’m upset that my words were mis-interpreted and used to do just that.” She then detailed exactly how the lie was propagated and by whom (she has the entire transcript).

The fact is this: none of us in Boston ever sought to portray the Diaspora as insignificant or to downplay their invaluable support. Why in the world would we ever do that? Robert and I detailed the invaluable role played by the Diaspora in our peer-reviewed study, for example. Moreover, I invited Sabina to join our Ushahidi-Haiti team precisely because the Diaspora were already responding in amazing ways and I knew they’d stay the course after the end of the emergency phase—we wanted to transfer full ownership of the Haiti Crisis Map to Haitian hands.  In sum, it was crystal clear to every single one of us that Sabina was the perfect person to take on this very important responsibility. She represented the voice and interests of Haitians with incredible agility, determination and intell-igence throughout our many months of work together, both in Boston and Haiti.

bio

Using CrowdFlower to Microtask Disaster Response

Cross-posted from CrowdFlower blog

A devastating earthquake struck Port-au-Prince on January 12, 2010. Two weeks later, on January 27th, a CrowdFlower was used to translate text messages from Haitian Creole to English. Tens of thousands of messages were sent by affected Haitians over the course of several months. All of these were heroically translated by hundreds of dedicated Creole-speaking volunteers based in dozens of countries across the globe. While Ushahidi took the lead by developing the initial translation platform used just days after the earthquake, the translation efforts were eventually rerouted to CrowdFlower. Why? Three simple reasons:

  1. CrowdFlower is one of the leading and most highly robust micro-tasking platforms there is;
  2. CrowdFlower’s leadership is highly committed to supporting digital humanitarian response efforts;
  3. Haitians in Haiti could now be paid for their translation work.

While the CrowdFlower project was launched 15 days after the earthquake, i.e., following the completion of search and rescue operations, every single digital humanitarian effort in Haiti was reactive. The key takeaway here was the proof of concept–namely that large-scale micro-tasking could play an important role in humanitarian information management. This was confirmed months later when devastating floods inundated much of Pakistan. CrowdFlower was once again used to translate incoming messages from the disaster affected population. While still reactive, this second use of CrowdFlower demonstrated replicability.

The most recent and perhaps most powerful use of CrowdFlower for disaster response occurred right after Typhoon Pablo devastated the Philippines in early December 2012. The UN Office for the Coordination of Humanitarian Affairs (OCHA) activated the Digital Humanitarian Network (DHN) to rapidly deliver a detailed dataset of geo-tagged pictures and video footage (posted on Twitter) depicting the damage caused by the Typhoon. The UN needed this dataset within 12 hours, which required that 20,000 tweets to be analyzed as quickly as possible. The Standby Volunteer Task Force (SBTF), a member of Digital Huma-nitarians, immediately used CrowdFlower to identify all tweets with links to pictures & video footage. SBTF volunteers subsequently analyzed those pictures and videos for damage and geographic information using other means.

This was the most rapid use of CrowdFlower following a disaster. In fact, this use of CrowdFlower was pioneering in many respects. This was the first time that a member of the Digital Humanitarian Network made use of CrowdFlower (and thus micro-tasking) for disaster response. It was also the first time that Crowd-Flower’s existing workforce was used for disaster response. In addition, this was the first time that data processed by CrowdFlower contributed to an official crisis map produced by the UN for disaster response (see above).

These three use-cases, Haiti, Pakistan and the Philippines, clearly demonstrate the added value of micro-tasking (and hence CrowdFlower) for disaster response. If CrowdFlower had not been available in Haiti, the alternative would have been to pay a handful of professional translators. The total price could have come to some $10,000 for 50,000 text messages (at 0.20 cents per word). Thanks to CrowdFlower, Haitians in Haiti were given the chance to make some of that money by translating the text messages themselves. Income generation programs are absolutely critical to rapid recovery following major disasters. In Pakistan, the use of CrowdFlower enabled Pakistani students and the Diaspora to volunteer their time and thus accelerate the translation work for free. Following Typhoon Pablo, paid CrowdFlower workers from the Philippines, India and Australia categorized several thousand tweets in just a couple hours while the volunteers from the Standby Volunteer Task Force geo-tagged the results. Had CrowdFlower not been available then, it is highly, highly unlikely that the mission would have succeeded given the very short turn-around required by the UN.

While impressive, the above use-cases were also reactive. We need to be a lot more pro-active, which is why I’m excited to be collaborating with CrowdFlower colleagues to customize a standby platform for use by the Digital Humanitarian Network. Having a platform ready-to-go within minutes is key. And while digital volunteers will be able to use this standby platform, I strongly believe that paid CrowdFlower workers also have a key role to play in the digital huma-nitarian ecosystem. Indeed, CrowdFlower’s large, multinational and multi-lingual global workforce is simply unparalleled and has the distinct advantage of being very well versed in the CrowdFlower platform.

In sum, it is high time that the digital humanitarian space move from crowd-sourcing to micro-tasking. It has been three years since the tragic earthquake in Haiti but we have yet to adopt micro-tasking more widely. CrowdFlower should thus play a key role in promoting and enabling this important shift. Their con-tinued important leadership in digital humanitarian response should also serve as a model for other private sector companies in the US and across the globe.

bio

Launching: SMS Code of Conduct for Disaster Response

Shortly after the devastating Haiti Earthquake of January 12, 2010, I published this blog post on the urgent need for an SMS code of conduct for disaster response. Several months later, I co-authored this peer-reviewed study on the lessons learned from the unprecedented use of SMS following the Haiti Earth-quake. This week, at the Mobile World Congress (MWC 2013) in Barcelona, GSMA’s Disaster Response Program organized two panels on mobile technology for disaster response and used the event to launch an official SMS Code of Conduct for Disaster Response (PDF). GSMA members comprise nearly 800 mobile operators based in more than 220 countries.

Screen Shot 2013-02-18 at 2.27.32 PM

Thanks to Kyla Reid, Director for Disaster Response at GSMA, and to Souktel’s Jakob Korenblummy calls for an SMS code of conduct were not ignored. The three of us spent a considerable amount of time in 2012 drafting and re-drafting a detailed set of principles to guide SMS use in disaster response. During this process, we benefited enormously from many experts on the mobile operators side and the humanitarian community; many of whom are at MWC 2013 for the launch of the guidelines. It is important to note that there have been a number of parallel efforts that our combined work has greatly benefited from. The Code of Conduct we launched this week does not seek to duplicate these important efforts but rather serves to inform GSMA members about the growing importance of SMS use for disaster response. We hope this will help catalyze a closer relationship between the world’s leading mobile operators and the international humanitarian community.

Since the impetus for this week’s launch began in response to the Haiti Earth-quake, I was invited to reflect on the crisis mapping efforts I spearheaded at the time. (My slides for the second panel organized by GSMA are available here. My more personal reflections on the 3rd year anniversary of the earthquake are posted here). For several weeks, digital volunteers updated the Ushahidi-Haiti Crisis Map (pictured above) with new information gathered from hundreds of different sources. One of these information channels was SMS. My colleague Josh Nesbit secured an SMS short code for Haiti thanks to a tweet he posted at 1:38pm on Jan 13th (top left in image below). Several days later, the short code (4636) was integrated with the Ushahidi-Haiti Map.

Screen Shot 2013-02-18 at 2.40.09 PM

We received about 10,000 text messages from the disaster-affected population during the during the Search and Rescue phase. But we only mapped about 10% of these because we prioritized the most urgent and actionable messages. While mapping these messages, however, we had to address a critical issue: data privacy and protection. There’s an important trade-off here: the more open the data, the more widely useable that information is likely to be for professional disaster responders, local communities and the Diaspora—but goodbye privacy.

Time was not a luxury we had; an an entire week had already passed since the earthquake. We were at the tail end of the search and rescue phase, which meant that literally every hour counted for potential survivors still trapped under the rubble. So we immediately reached out to 2 trusted lawyers in Boston, one of them a highly reputable Law Professor at The Fletcher School of Law and Diplomacy who also a specialist on Haiti. You can read the lawyers’ written email replies along with the day/time they were received on the right-hand side of the slide. Both lawyers opined that consent was implied vis-à-vis the publishing of personal identifying information. We shared this opinion with all team members and partners working with us. We then made a joint decision 24 hours later to move ahead and publish the full content of incoming messages. This decision was supported by an Advisory Board I put together comprised of humanitarian colleagues from the Harvard Humanitarian Initiative who agreed that the risks of making this info public were minimal vis-à-vis the principle of Do No HarmUshahidi thus launched a micro-tasking platform to crowdsource the translation efforts and hosted this on 4636.Ushahidi.com [link no longer live], which volunteers from the Diaspora used to translate the text messages.

I was able to secure a small amount of funding in March 2010 to commission a fully independent evaluation of our combined efforts. The project was evaluated a year later by seasoned experts from Tulane University. The results were mixed. While the US Marine Corps publicly claimed to have saved hundreds of lives thanks to the map, it was very hard for the evaluators to corroborate this infor-mation during their short field visit to Port-au-Prince more than 12 months after the earthquake. Still, this evaluation remains the only professional, independent and rigorous assessment of Ushahidi and 4636 to date.

Screen Shot 2013-02-25 at 2.10.47 AM

The use of mobile technology for disaster response will continue to increase for years to come. Mobile operators and humanitarian organizations must therefore be pro-active in managing this increase demand by ensuring that the technology is used wisely. I, for one, never again want to spend 24+ precious hours debating whether or not urgent life-and-death text messages can or cannot be mapped because of uncertainties over data privacy and protection—24 hours during a Search and Rescue phase is almost certain to make the difference between life and death. More importantly, however, I am stunned that a bunch of volunteers with little experience in crisis response and no affiliation whatsoever to any established humanitarian organization were able to secure and use an official SMS short code within days of a major disaster. It is little surprise that we made mistakes. So a big thank you to Kyla and Jakob for their leadership and perseverance in drafting and launching GSMA’s official SMS Code of Conduct to make sure the same mistakes are not made again.

While the document we’ve compiled does not solve every possible challenge con-ceivable, we hope it is seen as a first step towards a more informed and responsible use of SMS for disaster response. Rest assured that these guidelines are by no means written in stone. Please, if you have any feedback, kindly share them in the comments section below or privately via email. We are absolutely committed to making this a living document that can be updated.

To connect this effort with the work that my CrisisComputing Team and I are doing at QCRI, our contact at Digicel during the Haiti response had given us the option of sending out a mass SMS broadcast to their 2 million subscribers to get the word out about 4636. (We had thus far used local community radio stations). But given that we were processing incoming SMS’s manually, there was no way we’d be able to handle the increased volume and velocity of incoming text messages following the SMS blast. So my team and I are exploring the use of advanced computing solutions to automatically parse and triage large volumes of text messages posted during disasters. The project, which currently uses Twitter, is described here in more detail.

bio

Personal Reflections: 3 Years After the Haiti Earthquake

The devastating earthquake that struck Port-au-Prince on January 12, 2010 killed as many as 200,000 people. My fiancée and five close friends were in Haiti at the time and narrowly escaped a collapsing building. They were some of the lucky few survivors. But I had no knowledge that they had survived until 8 hours or so after the earthquake because we were unable get any calls through. The Haiti Crisis Map I subsequently spearheaded still stands as the most psycho-logically and emotionally difficult project I’ve ever been a part of.

The heroes of this initiative and the continuing source of my inspiration today were the hundreds and hundreds of volunteers who ensured the Haiti Crisis Map remained live for so many weeks. The majority of these volunteers were of course the Haitian Diaspora as well as Haitians in country. I had the honor of meeting and working with one of these heroes while in Port-au-Prince, Kurt Jean-Charles, the CEO of the Haitian software company Solutions.ht. I invited Kurt to give the Keynote at the 2010 International Crisis Mappers Conference (ICCM 2010) and highly recommend watching the video above. Kurt speaks directly from the heart.

HaitianDiaspora

Another personal hero of mine (pictured above) is Sabina Carlson—now Sabina Carlson Robillard following her recent wedding to Louino in Port-au-Prince! She volunteered as the Haitian Diaspora Liaison for the Haiti Crisis Map and has been living in Cité Soleil ever since. Needless to say, she continues to inspire all of us who have had the honor of working with her and learning from her.

Finally, but certainly not (!) least, the many, many hundreds of amazing volun-teers who tirelessly translated tens of thousands of text messages for this project. Thanks to you, some 1,500 messages from the disaster-affected population were added to the live crisis map of Haiti. This link points to the only independent, rigorous and professional evaluation of the project that exists. I highly reco-mmend reading this report as it comprises a number of important lessons learned in crisis mapping and digital humanitarian response.

Fonkoze

In the meantime, please consider making a donation to Fonkoze, an outstanding local organization committed to the social and economic improvement of the Haitian poor. Fonkoze is close to my heart not only because of the great work that they do but also because its staff and CEO were the ones who ensured the safe return of my fiancée and friends after the earthquake. In fact, my fiancée has continued to collaborate with them ever since and still works on related projects in Haiti. She is headed back to Port-au-Prince this very weekend. To make a tax deductible donation to Fonkoze, please visit this link. Thank you.

My thoughts & prayers go out to all those who lost loved ones in Haiti years ago.

Surprising Findings: Using Mobile Phones to Predict Population Displacement After Major Disasters

Rising concerns over the consequences of mass refugee flows during several crises in the late 1970′s is what prompted the United Nations (UN) to call for the establishment of early warning systems for the first time. “In 1978-79 for example, the United Nations and UNHCR were clearly overwhelmed by and unprepared for the mass influx of Indochinese refugees in South East Asia. The number of boat people washed onto the beaches there seriously challenged UNHCR’s capability to cope. One of the issues was the lack of advance information. The result was much human suffering, including many deaths. It took too long for emergency assistance by intergovernmental and non-governmental organizations to reach the sites” (Druke 2012 PDF).

Forty years later, my colleagues at Flowminder are using location data from mobile phones to nowcast and predict population displacement after major disasters. Focusing on the devastating 2010 Haiti earthquake, the team analyzed the movement of 1.9 million mobile users before and after the earthquake. Naturally, the Flowminder team expected that the mass exodus from Port-au-Prince would be rather challenging to predict. Surprisingly, however, the predictability of people’s movements remained high and even increased during the three-month period following the earthquake.

The team just released their findings in a peer-reviewed study entitled: “Predictability of population displacement after the 2010 Haiti earthquake” (PNAS 2012). As the analysis reveals, “the destinations of people who left the capital during the first three weeks after the earthquake was highly correlated with their mobility patterns during normal times, and specifically with the locations in which people had significant social bonds, as measured by where they spent Christmas and New Year holidays” (PNAS 2012).

For the people who left Port-au-Prince, the duration of their stay outside the city, as well as the time for their return, all followed a skewed, fat-tailed distribution. The findings suggest that population movements during disasters may be significantly more predictable than previously thought” (PNAS 2012). Intriguingly, the analysis also revealed the period of time that people in Port-au-Prince waited to leave the city (and then return) was “power-law distributed, both during normal days and after the earthquake, albeit with different exponents (PNAS 2012).” Clearly then, “[p]eople’s movements are highly influenced by their historic behavior and their social bonds, and this fact remained even after one of the most severe disasters in history” (PNAS 2012).

 

I wonder how this approach could be used in combination with crowdsourced satellite imagery analysis on the one hand and with Agent Based Models on the other. In terms of crowdsourcing, I have in mind the work carried out by the Standby Volunteer Task Force (SBTF) in partnership with UNHCR and Tomnod in Somalia last year. SBTF volunteers (“Mapsters”) tagged over a quarter million features that looked liked IDP shelters in under 120 hours, yielding a triangulated country of approximately 47,500 shelters.

In terms of Agent Based Models (ABMs), some colleagues and I  worked on “simulating population displacements following a crisis”  back in 2006 while at the Santa Fe Institute (SFI). We decided to use an Agent Based Model because the data on population movement was simply not within our reach. Moreover, we were particularly interested in modeling movements of ethnic populations after a political crisis and thus within the context of a politically charged environment.

So we included a preference for “safety in numbers” within the model. This parameter can easily be tweaked to reflect a preference for moving to locations that allow for the maintenance of social bonds as identified in the Flowminder study. The figure above lists all the parameters we used in our simple decision theoretic model.

The output below depicts the Agent Based Model in action. The multi-colored panels on the left depict the geographical location of ethnic groups at a certain period of time after the crisis escalates. The red panels on the right depict the underlying social networks and bonds that correspond to the geographic distribution just described. The main variable we played with was the size or magnitude of the sudden onset crisis to determine whether and how people might move differently around various ethnic enclaves. The study long with the results are available in this PDF.

In sum, it would be interesting to carry out Flowminder’s analysis in combination with crowdsourced satellite imagery analysis and live sensor data feeding into an Agent Base Model. Dissertation, anyone?

Disaster Response, Self-Organization and Resilience: Shocking Insights from the Haiti Humanitarian Assistance Evaluation

Tulane University and the State University of Haiti just released a rather damming evaluation of the humanitarian response to the 2010 earthquake that struck Haiti on January 12th. The comprehensive assessment, which takes a participatory approach and applies a novel resilience framework, finds that despite several billion dollars in “aid”, humanitarian assistance did not make a detectable contribution to the resilience of the Haitian population and in some cases increased certain communities’ vulnerability and even caused harm. Welcome to supply-side humanitarian assistance directed by external actors.

“All we need is information. Why can’t we get information?” A quote taken from one of many focus groups conducted by the evaluators. “There was little to no information exchange between the international community tasked with humanitarian response and the Haitian NGOs, civil society or affected persons / communities themselves.” Information is critical for effective humanitarian assistance, which should include two objectives: “preventing excess mortality and human suffering in the immediate, and in the longer term, improving the community’s ability to respond to potential future shocks.” This longer term objective thus focuses on resilience, which the evaluation team defines as follows:

“Resilience is the capacity of the affected community to self-organize, learn from and vigorously recover from adverse situations stronger than it was before.”

This link between resilience and capacity for self-organization is truly profound and incredibly important. To be sure, the evaluation reveals that “the humani-tarian response frequently undermined the capacity of Haitian individuals and organizations.” This completely violates the Hippocratic Oath of Do No Harm. The evaluators thus “promote the attainment of self-sufficiency, rather than the ongoing dependency on standard humanitarian assistance.” Indeed, “focus groups indicated that solutions to help people help themselves were desired.”

I find it particularly telling that many aid organizations interviewed for this assessment were reluctant to assist the evaluators in fully capturing and analyzing resource flows, which are critical for impact evaluation. “The lack of transparency in program dispersal of resources was a major constraint in our research of effective program evaluation.” To this end, the evaluation team argue that “by strengthening Haitian institutions’ ability to monitor and evaluate, Haitians will more easily be able to track and monitor international efforts.”

I completely disagree with this remedy. The institutions are part of the problem, and besides, institution-building takes years if not decades. To assume there is even political will and the resources for such efforts is at best misguided. If resilience is about strengthening the capacity of affected communities to self-organize, then I would focus on just that, applying existing technologies and processes that both catalyze and facilitate demand-side, people-centered self-organization. My previous blog post on “Technology and Building Resilient Societies to Mitigate the Impact of Disasters” elaborates on this point.

In sum, “resilience is the critical link between disaster and development; monitoring it will ensure that relief efforts are supporting, and not eroding, household and community capabilities.” This explains why crowdsourcing and data mining efforts like those of Ushahidi, HealthMap and UN Global Pulse are important for disaster response, self-organization and resilience.

Twitter, Crises and Early Detection: Why “Small Data” Still Matters

My colleagues John Brownstein and Rumi Chunara at Harvard Univer-sity’s HealthMap project are continuing to break new ground in the field of Digital Disease Detection. Using data obtained from tweets and online news, the team was able to identify a cholera outbreak in Haiti weeks before health officials acknowledged the problem publicly. Meanwhile, my colleagues from UN Global Pulse partnered with Crimson Hexagon to forecast food prices in Indonesia by carrying out sentiment analysis of tweets. I had actually written this blog post on Crimson Hexagon four years ago to explore how the platform could be used for early warning purposes, so I’m thrilled to see this potential realized.

There is a lot that intrigues me about the work that HealthMap and Global Pulse are doing. But one point that really struck me vis-a-vis the former is just how little data was necessary to identify the outbreak. To be sure, not many Haitians are on Twitter and my impression is that most humanitarians have not really taken to Twitter either (I’m not sure about the Haitian Diaspora). This would suggest that accurate, early detection is possible even without Big Data; even with “Small Data” that is neither representative or indeed verified. (Inter-estingly, Rumi notes that the Haiti dataset is actually larger than datasets typically used for this kind of study).

In related news, a recent peer-reviewed study by the European Commi-ssion found that the spatial distribution of crowdsourced text messages (SMS) following the earthquake in Haiti were strongly correlated with building damage. Again, the dataset of text messages was relatively small. And again, this data was neither collected using random sampling (i.e., it was crowdsourced) nor was it verified for accuracy. Yet the analysis of this small dataset still yielded some particularly interesting findings that have important implications for rapid damage detection in post-emergency contexts.

While I’m no expert in econometrics, what these studies suggests to me is that detecting change-over–time is ultimately more critical than having a large-N dataset, let alone one that is obtained via random sampling or even vetted for quality control purposes. That doesn’t mean that the latter factors are not important, it simply means that the outcome of the analysis is relatively less sensitive to these specific variables. Changes in the baseline volume/location of tweets on a given topic appears to be strongly correlated with offline dynamics.

What are the implications for crowdsourced crisis maps and disaster response? Could similar statistical analyses be carried out on Crowdmap data, for example? How small can a dataset be and still yield actionable findings like those mentioned in this blog post?

Some Thoughts on Real-Time Awareness for Tech@State

I’ve been invited to present at Tech@State in Washington DC to share some thoughts on the future of real-time awareness. So I thought I’d use my blog to brainstorm and invite feedback from iRevolution readers. The organizers of the event have shared the following questions with me as a way to guide the conver-sation: Where is all of this headed?  What will social media look like in five to ten years and what will we do with all of the data? Knowing that the data stream can only increase in size, what can we do now to prepare and prevent being over-whelmed by the sheer volume of data?

These are big, open-ended questions, and I will only have 5 minutes to share some preliminary thoughts. I shall thus focus on how time-critical crowdsourcing can yield real-time awareness and expand from there.

Two years ago, my good friend and colleague Riley Crane won DARPA’s $40,000 Red Balloon Competition. His team at MIT found the location of 10 weather balloons hidden across the continental US in under 9 hours. The US covers more than 3.7 million square miles and the balloons were barely 8 feet wide. This was truly a needle-in-the-haystack kind of challenge. So how did they do it? They used crowdsourcing and leveraged social media—Twitter in particular—by using a “recursive incentive mechanism” to recruit thousands of volunteers to the cause. This mechanism would basically reward individual participants financially based on how important their contributions were to the location of one or more balloons. The result? Real-time, networked awareness.

Around the same time that Riley and his team celebrated their victory at MIT, another novel crowdsourcing initiative was taking place just a few miles away at The Fletcher School. Hundreds of students were busy combing through social and mainstream media channels for actionable and mappable information on Haiti following the devastating earthquake that had struck Port-au-Prince. This content was then mapped on the Ushahidi-Haiti Crisis Map, providing real-time situational awareness to first responders like the US Coast Guard and US Marine Corps. At the same time, hundreds of volunteers from the Haitian Diaspora were busy translating and geo-coding tens of thousands of text messages from disaster-affected communities in Haiti who were texting in their location & most urgent needs to a dedicated SMS short code. Fletcher School students filtered and mapped the most urgent and actionable of these text messages as well.

One year after Haiti, the United Nation’s Office for the Coordination of Humanitarian Affairs (OCHA) asked the Standby Volunteer Task Force (SBTF) , a global network of 700+ volunteers, for a real-time map of crowdsourced social media information on Libya in order to improve their own situational awareness. Thus was born the Libya Crisis Map.

The result? The Head of OCHA’s Information Services Section at the time sent an email to SBTF volunteers to commend them for their novel efforts. In this email, he wrote:

“Your efforts at tackling a difficult problem have definitely reduced the information overload; sorting through the multitude of signals on the crisis is no easy task. The Task Force has given us an output that is manageable and digestible, which in turn contributes to better situational awareness and decision making.”

These three examples from the US, Haiti and Libya demonstrate what is already possible with time-critical crowdsourcing and social media. So where is all this headed? You may have noted from each of these examples that their success relied on the individual actions of hundreds and sometimes thousands of volunteers. This is primarily because automated solutions to filter and curate the data stream are not yet available (or rather accessible) to the wider public. Indeed, these solutions tend to be proprietary, expensive and/or classified. I thus expect to see free and open source solutions crop up in the near future; solutions that will radically democratize the tools needed to gain shared, real-time awareness.

But automated natural language processing (NLP) and machine learning alone are not likely to succeed, in my opinion. The data stream is actually not a stream, it is a massive torent of non-indexed information, a 24-hour global firehose of real-time, distributed multi-media data that continues to outpace our ability to produce actionable intelligence from this torrential downpour of 0′s and 1′s. To turn this data tsunami into real-time shared awareness will require that our filtering and curation platforms become more automated and collaborative. I believe the key is thus to combine automated solutions with real-time collabora-tive crowdsourcing tools—that is, platforms that enable crowds to collaboratively filter and curate real-time information, in real-time.

Right now, when we comb through Twitter, for example, we do so on our own, sitting behind our laptop, isolated from others who may be seeking to filter the exact same type of content. We need to develop free and open source platforms that allow for the distributed-but-networked, crowdsourced filtering and curation of information in order to democratize the sense-making of the firehose. Only then will the wider public be able to win the equivalent of Red Balloon competitions without needing $40,000 or a degree from MIT.

I’d love to get feedback from readers about what other compelling cases or arguments I should bring up in my presentation tomorrow. So feel free to post some suggestions in the comments section below. Thank you!

Tracking Population Movements using Mobile Phones and Crisis Mapping: A Post-Earthquake Geospatial Study in Haiti

I’ve been meaning to blog about this project since it was featured on BBC last month: “Mobile Phones Help to Target Disaster Aid, says Study.” I’ve since had the good fortune of meeting Linus Bengtsson and Xin Lu, the two lead authors of this study (PDF), at a recent strategy meeting organized by GSMA. The authors are now launching “Flowminder” in affiliation with the Karolinska Institutet in Stockholm to replicate their excellent work beyond Haiti. If “Flowminder” sounds familiar, you may be thinking of Hans Rosling’s “Gapminder” which also came out of the Karolinska Institutet. Flowminder’s mission: “Providing priceless information for free for the benefit of those who need it the most.”

As the authors note, “population movements following disasters can cause important increases in morbidity and mortality.” That is why the UN sought to develop early warning systems for refugee flows during the 1980′s and 1990′s. These largely didn’t pan out; forecasting is not a trivial challenge. Nowcasting, however, may be easier. That said, “no rapid and accurate method exists to track population movements after disasters.” So the authors used “position data of SIM cards from the largest mobile phone company in Haiti (Digicel) to estimate the magnitude and trends of population movements following the Haiti 2010 earthquake and cholera outbreak.”

The geographic locations of SIM cards were determined by the location of the mobile phone towers that SIM cards were connecting to when calling. The authors followed the daily positions of 1.9 million SIM cards for 42 days prior to the earthquake and 158 days following the quake. The results of the analysis reveal that an estimated 20% of the population in Port-au-Prince left the city within three weeks of the earthquake. These findings corresponded well with of a large, retrospective population based survey carried out by the UN.

“To demonstrate feasibility of rapid estimates and to identify areas at potentially increased risk of outbreaks,” the authors “produced reports on SIM card move-ments from a cholera outbreak area at its immediate onset and within 12 hours of receiving data.” This latter analysis tracked close to 140,000 SIM cards over an 8 day period. In sum, the “results suggest that estimates of population movements during disasters and outbreaks can be delivered rapidly and with potentially high validity in areas with high mobile phone use.”

I’m really keen to see the Flowminder team continue their important work in and beyond Haiti. I’ve invited them to present at the International Conference of Crisis Mappers (ICCM 2011) in Geneva next month and hope they’ll be able to join us. I’m interested to explore the possibilities of combining this type of data and analysis with crowdsourced crisis information and satellite imagery analysis. In addition, mobile phone data can also be used to estimate the hardest hit areas after a disaster. For more on this, please see my previous blog post entitled “Analyzing Call Dynamics to Assess the Impact of Earthquakes” and this post on using mobile phone data to assess the impact of building damage in Haiti.