Monthly Archives: October 2011

Democratizing ICT for Development with DIY Innovation and Open Data

The recent Net Impact conference in Portland proved to be an ideal space to take a few steps back and reflect on the bigger picture. There was much talk of new and alternative approaches to traditional development. The word “participatory” in particular was a trending topic among both presenters and participants. But exactly how “participatory” are these “participatory” approaches to develop-ment? Do they fundamentally democratize the development process? And do these “novel” participatory approaches really let go of control? Should they? The following thoughts and ideas were co-developed in follow-up conversations with my colleague Chrissy Martin who also attended Net Impact. She blogs at Innovate.Inclusively.

I haven’t had the space recently to think through some of these questions or reflect about how the work I’ve been doing with Ushahidi fits (or doesn’t) within the traditional development paradigm—a paradigm which many at the confer-ence characterized as #fail. Some think that perhaps technology can help change this paradigm, hence the burst of energy around the ICT for Development (ICT4D) field. That said, it is worth remembering that the motivations driving this shift are more important than any one technology. For example, recall the principles behind the genesis of the Ushahidi platform: Democratizing information flows and access; promoting Open Data and Do it Yourself (DIY) Innovation with free, highly hackable (i.e., open source) technology; letting go of control.

The Ushahidi platform is not finished. It will never be finished. This is deliberate, not an error in the code. Free and open source software (FOSS) is by definition in a continual phase of co-Research and Development (co-R&D). The Ushahidi platform is not a solution, it is a platform on top of which others build their own solutions. These solutions remain open source and some are folded back into the core Ushahidi code. This type of “open protocol” can reverse “innovation cascades” leading to “reverse innovation” from developing to indus-trialized countries (c.f. information cascades). FOSS acts like a virus, it self-propagates. The Ushahidi platform, for example, has propagated to over 130 countries since it was first launched during Kenya’s post-election violence almost four years ago.

In some ways, the Ushahidi platform can be likened to a “choose your own adventure” game. The readers, not the authors, finish the story. They are the main characters who bring the role playing games and stories to life. But FOSS goes beyond this analogy. The readers can become the authors and vice versa. Welcome to co-creation. Perhaps one insightful analogy is the comparison between Zipcar and RelayRides.

I’ve used the Zipcar for over five years now and love it. But what would a “democratized” Zipcar look like? You guessed it: RelayRides turns every car owner into their own mini-DIY-Zipcar company. You basically get your own “Zipcar-in-a-box” kit and rent out your own car in the same way that Zipcar does with their cars. RelayRides is basically an open source version of Zipcar, a do-it-yourself innovation. A good friend of mine, Becca, is an avid RelayRides user. The income from lending her car out lets her cover part of her rent, and if she needs a car while hers is rented out, she’ll get online and look for available RelayRides in her neighborhood. She likes the “communal ownership” spirit that the technology facilitates. Indeed, she is getting to know her neighbors better as a result. In this case, DIY Innovation is turning strangers, a crowd, into a comm-unity. Perhaps DIY Innovation can facilitate community building in the long run.

The Ushahidi platform shares this same spirit. The motivation behind Ushahidi’s new “Check-In’s” feature, for example, is to democratize platforms like Foursquare. There’s no reason why others can’t have their own Foursquares and customize them for their own projects along with the badges, etc. That’s not to imply that the Ushahidi platform is perfect. There’s a long way to go, but again, it will never be perfect nor is that the intention. Sure, the technology will become more robust, stable and extensible, but not perfect. Perfection denotes an endstate. There is no endstate in co-R&D. The choose your own adventure story continues for as long as the reader, the main character decides to read on.

I’m all for “participatory development” but I’m also interested in allowing indivi-duals to innovate for themselves first and then decide how and who to participate with. I’d call that self-determination. This explains why the Ushahidi team is no longer the only “game in town” so-to-speak. Our colleagues at DISC have customized the Ushahidi platform in more innovative and relevant ways than we could have for the Egyptian context. Not only that, they’re making a business out of customizing the platform and training others in the Arab World. The Ushahidi code is out of our hands and it has been since 2008. We’re actively promoting and supporting partners like DISC. Some may say we’re nurturing our own competition. Well then, even better.

Freely providing the hackable building blocks for DIY Innovation is one way to let go of control and democratize ICT4D. Another complementary way is to democratize information access by promoting automated Open Data generation, i.e., embedded real-time sensors for monitoring purposes. Equal and public access to Open Data levels the playing field, prevents information arbitrage and disrupts otherwise entrenched flows of information. Participatory development without Open Data is unlikely to hold institutions accountable or render the quality of their services (or lack thereof) more transparent. But by Open Data here I don’t only mean data generated via participatory surveys or crowdsourcing.

The type of public-access Open Data generation I’m interested in could be called “Does-It-Itself” Open Data, or DII Data. Take “The Internet of Things” idea and apply this to traditional development. Let non-intrusive, embedded and real-time sensors provide direct, empirical and open data on the status of develop-ment projects without any “middle man” who may have an interest in skewing the data. In other words, hack the Monitoring and Evaluation process (M&E) by letting the sensors vote for themselves and display the “election results” publicly and in real time. Give the sensors a voice. Meet Evan Thomas, a young professor at Portland State, who spends his time doing just this at SweetLab, and my colleague Rose Goslinga who is taking the idea of DII Data to farmers in Kenya.

Evan embeds customized sensors to monitor dozens of development projects in several countries. These sensors generate real-time, high-resolution data that is otherwise challenging, expensive and time-consuming to collect via the tradi-tional survey-based approach. Evan’s embedded sensors generate behavior and usage data for projects like the Mercy Corps Water and Sanitation Program and Bridges to Prosperity Program. Another example of DII Data is Rose’s weather index insurance (WII) project in Kenya called Kilimo Salama. This initiative uses atmospheric data automatically transmitted via local weather towers to determine insurance payouts for participating farmers during periods of drought or floods. Now, instead of expensive visits to farms and subjective assessments, this data-driven approach to feedback loops lowers program costs and renders the process more objective and transparent.

There is of course more to the development field than the innovative processes described above. Development means a great many things to different people. The same is true of the words “Democracy”, “Participatory” and “Crowd-sourcing.” For me, crowdsourcing, like democracy, is a methodology that can catalyze greater participation and civic engagement. Some liken this to demo-cratizing the political process. Elections, in a way, are crowdsourced. Obviously, however, crowdsourced elections in no way imply that they are free, open or fair. Moreover, elections are but one of the ingredients in the recipe for  a democratic, political process.

In the same way, democratizing ICT4D is not a sufficient condition to ensure that the traditional development space obtains a new hashtag: #success. Letting go of control and allowing for self-determination can of course lead to unexpected outcomes. At this point, however, given the #fail hashtag associated with traditional development, perhaps unexpected outcomes driven by democratic, bottom-up innovation processes that facilitate self-organization, determination and participation, are more respectful to human dignity and ingenuity.

Microtasking Advocacy and Humanitarian Response in Somalia

I’ve been working on bridging the gap between the technology innovation sector and the humanitarian & human rights communities for years now. One area that holds great promise is the use of microtasking for advocacy and humanitarian response. So I’d like to share two projects I’m spearheading with the support of several key colleagues. I hope these pilot projects will further demonstrate the value of mainstreaming microtasking. Both initiatives are focused on Somalia.

The first pilot project plans to leverage Souktel‘s large SMS subscriber base in Somalia to render local Somali voices and opinions more visibile in the mainstream media. This initiative combines the efforts of a Somali celebrity, members of the Somali Diaspora, a major international news organization, Ushahidi and CrowdFlower. In order to translate, categorize and geolocate incoming text messages, I reached out to my colleagues at CrowdFlower, a San Francisco-based company specializing in microtasking.

I had catalyzed a partnership with Crowdflower during the PakReport deploy-ment last year and wanted to repeat this successful collaboration for Somalia. To my delight, the team at Crowdflower was equally interested in contri-buting to this initiative. So we’ve started to customize a Crowdflower plugin for Somalia. This interface will allow members of the Somali Diaspora to use a web-based platform to translate, categorize and geolocate incoming SMS’s from the Horn of Africa. The text messages processed by the Diaspora will then be published on a public Ushahidi map.

Our international media partner will help promote this initiative and invite comments in response to the content shared via SMS. The media group will then select the most compelling replies and share these (via SMS) with the authors of the original text messages in Somalia.  The purpose of this project is to catalyze more media and world attention on Somalia, which is slowly slipping from the news. We hope that the content and resulting interaction will generate the kind of near real-time information that advocacy groups and the Diaspora can leverage in their lobbying efforts.

The second pilot project is a partnership between the Standby Volunteer Task Force (SBTF), UNHCR, DigitalGlobe and Tomnod. The purpose of this project, is to build on this earlier trial run and microtask the tagging of informal shelters in a certain region of the country to identify where IDPs are located and also esti-mate the total IDP population size. The microtasking part of this project is possible thanks to the Tomnod platform, which I’ve already blogged about in the context of this recent Syria project. The project will use a more specialized rule-set and feature-key developed with UNHCR to maximize data quality.

We are also partnering with the European Commission’s Joint Research Center (JRC) on this UNCHR project. The JRC team will run their automated shelter-detection algorithms on the same set of satellite images. The goal is to compare and triangulate crowdsource methods with automated approaches to satellite imagery analysis.

There are several advantages to using microtasking solutions for advocacy and humanitarian purposes. The first is that the tasks can easily be streamlined and distributed far and wide. Secondly, this approach to microtasking is highly scalable, rapid and easily modifiable. Finally, microtasking allows for quality control via triangulation, accountability and statistical analysis. For example, only when two volunteers translate an incoming text message from Somalia in a similar way does that text message get pushed to an Ushahidi map of local Somali voices. The same kind of triangulation can be applied to the categorization and geolocation of text messages, and indeed shelters in satellite imagery.

Microtasking is no silver bullet for advocacy and humanitarian response. But it is an important new tool in the tool box that can provide substantial support in times of crisis, especially when leveraged with other traditional approaches. I really hope the two projects described above take off. In the meantime, feel free to browse through my earlier blog posts below for further information on related applications of microtasking:

·  Combining Crowdsourced Satellite Imagery Analysis with Crisis Reporting
·  OpenStreetMap’s Microtasking Platform for Satellite Imagery Tracing
·  Crowdsourcing Satellite Imagery Analysis for Somalia
· Crowdsourcing the Analysis of Satellite Imagery for Disaster Response
· Wanted for Pakistan: A Turksourcing Plugin for Crisis Mapping
· Using Massive Multiplayer Games to Turksource Crisis Information
· From Netsourcing to Crowdsourcing to Turksourcing Crisis Information
· Using Mechanical Turk to Crowdsource Humanitarian Response


Applying Earthquake Physics to Conflict Analysis

I really enjoyed speaking with Captain Wayner Porter whilst at PopTech 2011 last week. We both share a passion for applying insights from complexity science to different disciplines. I’ve long found the analogies between earthquakes and conflicts intriguing. We often talk of geopolitical fault lines, mounting tensions and social stress. “If this sounds at all like the processes at work in the Earth’s crust, where stresses build up slowly to be released in sudden earthquakes … it may be no coincidence” (Buchanan 2001).

To be sure, violent conflict is “often like an earthquake: it’s caused by the slow accumulation of deep and largely unseen pressures beneath the surface of our day-to-day affairs. At some point these pressures release their accumulated energy with catastrophic effect, creating shock waves that pulverize our habitual and often rigid ways of doing things…” (Homer-Dixon 2006).

But are fore shocks and aftershocks in social systems really as discernible as well? Like earthquakes, both inter-state and internal wars actually occur with the same statistical pattern (see my previous blog post on this). Since earthquakes and conflicts are complex systems, they also exhibit emergent features associated with critical states. In sum, “the science of earthquakes […] can help us understand sharp and sudden changes in types of complex systems that aren’t geological–including societies…” (Homer-Dixon 2006).

Back in 2006, I collaborated with Professor Didier Sornette and Dr. Ryan Woodard from the Swiss Federal Institute of Technology (ETHZ) to assess whether a mathematical technique developed for earthquake prediction might shed light on conflict dynamics. I presented this study along with our findings at the American Political Science Association (APSA) convention last year (PDF). This geophysics technique, “superposed epoch analysis,” is used to identify statistical signatures before and after earthquakes. In other words, this technique allows us to discern whether any patterns are discernible in the data during foreshocks and aftershocks. Earthquake physicists work from global spatial time series data of seismic events to develop models for earthquake prediction. We used a global time series dataset of conflict events generated from newswires over a 15-year period. The graph below explains the “superposed epoch analysis” technique as applied to conflict data.

eqphysics

The curve above represents a time series of conflict events (frequency) over a particular period of time. We select arbitrary threshold, such as “threshold A” denoted by the dotted line. Every peak that crosses this threshold is then “copied” and “pasted” into a new graph. That is, the peak, together with the data points 25 days prior to and following the peak is selected.

The peaks in the new graph are then superimposed and aligned such that the peaks overlap precisely. With “threshold A”, two events cross the threshold, five for “threshold B”. We then vary the thresholds to look for consistent behavior and examine the statistical behavior of the 25 days before and after the “extreme” conflict event. For this study, we performed the computational technique described above on the conflict data for the US, UK, Afghanistan, Columbia and Iraq.

Picture 4Picture 5Picture 6

The foreshock and aftershock behaviors in Iraq and Afghanistan appear to be similar. Is this because the conflicts in both countries were the result of external intervention, i.e., invasion by US forces (exogenous shock)?

In the case of Colombia, an internal low intensity and protracted conflict, the statistical behavior of foreshocks and aftershocks are visibly different from those of Iraq and Afghanistan. Do the different statistical behaviors point to specific signature associated with exogenous and endogenous causes of extreme events? Does one set of behavior contrast with another one in the same way that old wars and new wars differ?

Are certain extreme events endogenous or exogenous in nature? Can endogenous or exogenous signatures be identified? In other words, are extreme events just part of the fat tail of a power law due to self-organized criticality (endogeneity)? Or is catastrophism in action, extreme events require extreme causes outside the system (exogeneity)?

Another possibility still is that extreme events are the product of both endo-genous and exogenous effects. How would this dynamic unfold? To answer these questions, we need to go beyond political science. The distinction between responses to endogenous and exogenous processes is a fundamental property of physics and is quantified as the fluctuation-dissipation theorem in statistical mechanics. This theory has been successfully applied to social systems (such as books sales) as a way to help understand different classes of causes and effects.

Questions for future research: Do conflict among actors in social systems display measurable endogenous and exogenous behavior? If so, can a quantitative signature of precursory (endogenous) behavior be used to help recognize and then reduce growing conflict? The next phase of this research will be to apply the above techniques to the conflict dataset already used to examine the statistical behavior of foreshocks and aftershocks.

The Mathematics of War: On Earthquakes and Conflicts

A conversation with my colleague Sinan Aral at PopTech 2011 reminded me of some earlier research I had carried out on the mathematics of war. So this is a good time to share some of the findings from this research. The story begins some 60 years ago, when British physicist Lewis Fry Richardson found that international wars follow what is called a power law distribution. A power law distribution relates the frequency and “magnitude” of events. For example, the Richter scale, relates the size of earthquakes to their frequency. Richardson found that the frequency of international wars and the number of causalities each produced followed a power law.

More recently, my colleague Erik-Lars Cederman sought to explain Richardson’s findings in his 2003 peer-reviewed publication “Modeling the Size of Wars: From Billiard Balls to Sandpiles.” However, Lars used an invalid statistical technique to test for power law distributions. In 2005, I began collaborating with Pro-fessors Neil Johnson and Michael Spagat on related research after I came across their fascinating co-authored study that tested casualty distributions in new wars (internal conflicts) for power laws. Though he was not a co-author on the 2005 study, my colleague Sean Gourely presented this research at TED in 2009.

In any case, I invited Michael to present his research at The Fletcher School in the Fall of 2005 to generate interest here. Shortly after, I suggested to Michael that we test whether conflict events, in addition to casualties, followed a power law distribution. I had access to an otherwise proprietary dataset on conflict events that spanned a longer time period than the casualty datasets that he and Neils were working off. I also suggested we try to test whether casualties from natural disasters follow a power law distribution.

We chose to pursue the latter first and I submitted an abstract to the 2006 American Political Science Association (APSA) conference to present our findings. Soon after, I was accepted to the Santa Fe Institute’s Complex Systems Summer Institute for PhD students and took the opportunity to pursue my original research in testing conflict events for power law distributions with my colleague Dr. Ryan Woodard.

The APSA paper, presented in August 2006, was entitled “Natural Disasters, Casualties and Power Laws:  A Comparative Analysis with Armed Conflict” (PDF). Here is the paper’s abstract and findings:

Power-law relationships, relating events with magnitudes to their frequency, are common in natural disasters and violent conflict. Compared to many statistical distributions, power laws drop off more gradually, i.e. they have “fat tails”. Existing studies on natural disaster power laws are mostly confined to physical measurements, e.g., the Richter scale, and seldom cover casualty distributions. Drawing on the Center for Research on the Epidemiology of Disasters (CRED) International Disaster Database, 1980 to 2005, we find strong evidence for power laws in casualty distributions for all disasters combined, both globally and by continent except for North America and non-EU Europe. This finding is timely and gives useful guidance for disaster preparedness and response since natural catastrophes are increasing in frequency and affecting larger numbers of people.  We also find that the slopes of the disaster casualty power laws are much smaller than those for modern wars and terrorism, raising an open question of how to explain the differences. We show that many standard risk quantification methods fail in the case of natural disasters.

apsa1

Dr. Woodard and I presented our research on power laws and conflict events at SFI in June 2006. We produced a paper in August of that year entitled “Concerning Critical Correlations in Conflict, Cooperation and Casualties” (PDF). As the title implies, we also tested whether cooperative events followed a power law. As far as I know, we were the first to test conflict events not to mention cooperative events for power laws. In addition, we looked at conflict/cooperation (C/C) events in Western countries.

The abstract and some findings are included below:

Knowing that the number of casualties of war are distributed as a power law and given a rich data set of conflict and cooperation (C/C) events, we ask: Are there correlations among C/C events? Is there a correlation between C/C events and war casualties? Can C/C data be used as proxy for (potentially) less reliable casualty data? Can C/C data be used in conflict early warning systems? To begin to answer these questions we analyze the distribution of C/C event data for the period 1990–2004 in Afghanistan, Colombia, Iran, Iraq, North Korea, Switzerland, UK and USA. We find that the distributions of individual C/C event types scale as power laws, but only over approximately a single decade, leaving open the possibility of a more appropriate fit (for which we have not yet tested). However, the average exponent of the power law (2.5) is the same as that found in recent studies of casualties of war. We find low levels of correlations between C/C events in Iraq and Afghanistan but not in the other countries studied. We find that the distribution of the sum of all conflict or cooperation events scales exponentially. Finally, we find low levels of correlations between a two year time series of casualties in Afghanistan and the corresponding conflict events.

sfi1sfi2sfi3

I’m looking to discuss all this further with Sinan and learning more about his fascinating area of research.

The Best of PopTech2011 in Tweets and Pics

@CauseGlobal: Zolli opens PopTech2011 , A World ReBalancing:
“We are not in Kansas, nor are we in Oz.
We are in the whirlwind”

@eileenlambert: Tablet in US $300, tablet in India $35.
Eastern countries are innovating
for radical affordability.
@andrew_zolli

@storylaura: the rural poor only exist as numbers.
by taking pictures they are removed from anonymity.
Shahidul Alam

@rperezzz: At PopTech2011 @shahidul: introduced term “The Majority World” – better than third world. I don’t want to be third of anything.

@SarahNelson: Check out majorityworld.com to see the work of photographers from developing nations

@frogdesign: American dream is alive and well -
just not in U.S -it is in India. -Anand G.

@dgilford: “Destiny is something you make rather than inherit.” @AnandWrites on India’s revolution against tradition of “know your place

@rperezzz: PopTech2011: @AnandWrites :In India, we-centric societies
are moving toward me-centric societies.

@thedelk: “China’s economic dominance is more imminent, larger in magnitude, and broader in scope than is currently believed.”

@wlabar: “By 2030 there will be a G1 – China” – Arvind Subramanian

@priyaparker: Cover of @arvindsubraman‘s book #eclipse is photo of
Obama bowing to a fully-standing Hu.

@brainpicker: Ooh! @PopTech launches fantastic new iPad app,
visualizing the World Rebalancing theme j.mp/oUY853

@artate: check out @unglobalpulse piece in the new @poptech ipad app:
mobile surveys to get a pulse of the planet http://bit.ly/pdwKRm

 

@poptech: “Rebalancing is not something you do once,
it’s a way of life” -@StephanieCoontz

@rperezzz: Countries most resistant to women’s rights are countries where women have least access to labor force, says @StephanieCoontz

@xtinem: Coontz #PopTech2011 “US is dead last of all western countries
in work family policies”

@storylaura: I used to say US neanderthal in family/work policy. I’ve studied Neanderthals & they took great family care. Stephanie Coontz 

@storylauraIf we redefine gender to include women’s right to work,
we must redefine work to include workers’ rights to family life.

@bookpickings: Stealth of Nations: The Global Rise of the Informal Economy – intriguing new book by Robert Neuwirth http://j.mp/pIXUh0

@AnandWrites: 1.8 billion people, half of world workers,
work off the books in informal economy: Robert Neuwirth

@dgilford: Informal economy is profitable: avg.
Lagos street shoe seller has higher margin than Payless Shoe Stores.
-Robert Neuwith

priyaparker: @nils_gilman predicts rise of “survival entrepreneurship”
in places like Greece, Egypt, Syria

@poptech: “Reminders that we’re not masters of the universe.”
– President of Iceland on the events of past decade

@AnandWrites: How Iceland dealt with its crisis so different from US,
acc. to president. He says they purged all the people responsible

@brainpicker: “Bank failure should not become the responsibility of
the people.” The president of Iceland tells it like it is

@AnandWrites: “What we are now seeing is people power on its purest form”: Iceland president on power of social media

@storylaura: protests and action in Iceland successful because
1) mobilized thru internet
2) demands concrete and measurable.
Prez Grimson

@JMathewAllen: Pres of Iceland Social Media has empowered the people
and made institutions a “sideshow”

@AnandWritesPrez: When Iceland had financial crisis, China was more helpful than West. He hosts more delegations from China than Europe.

@poptechAnnouncing PopTech Reykjavik 2012: Toward Resilience,
June 27-29, 2012 http://poptech.org/iceland

@PatrickMeier: On the Role of Technology in Building Resilient Societies http://tinyurl.com/3aw4tsb

@wlabar: Best estimate by IPCC is an increase in temp. of 1.8 to 4C

@AnandWrites2.7 billion humans have no access to financial services,
even merely to save: Bhagwan Chowdhry http://pic.twitter.com/beNS0bot

@AnandWritesCellphones will become the banks for the poor

@rperezzz: PopTech2011 Social Innovation Fellow Rose Goslinga takes stage.
“I insure the rains” – micro-insurance for Kenyan farmers.

@brainpicker: “850,000 girls in Kenya miss school because
they don’t have sanitary pads.”

@ZanaAfrica: Spread the word: pads + health education
can break cycles of poverty for girls.

@rperezzz: I <3 Rothberg’s PopTech2011 preso title: High Speed DNA Sequencing: Outbreaks, Honey Bees, Neanderthals, Watson, Moore and Your Genome.

@deliciousblur: High speed genome sequencing offers a new way
to develop therapeutic drugs

@storylaura: It’s okay to be down, it’s a chance to step back and say,
“maybe we did it wrong.” Rothman

@storylaura: By sequencing Neaderthal DNA we learned ~200 places different
btwn human and Neanderthal and chimpanzee. Cool! Rothman
@ConnectMinds: Did you know that the current spacesuit weighs 140 kilos?
MIT’s Dava Newman is out to make things slimmer and more mobile
@audreylinnloves: imagine using a space suit to help kids with cerebral palsy partake in day to day activities
@colincolin: “Maybe the future of science will be in creating puzzles,
then handling them to the world to solve.” – Adrien Treuille
@jdsutterFoldIt game creator: “We’ve in fact crowdsourced the entire scientific method, from hypothesis to experiment to results”

@brainpicker: Wow. 6 months into Eterna experiment
the worst player design was better than
the best computer design. http://j.mp/pBSq2g

@patrickmeier: The next frontier: time-critical #crowdsolving

@brainpicker: “Crowdsourcing has the potential to democratize the economics
and the joys of basic science.”
@ChristieNic: CDC is going to do real time #crowdsourcing to find solutions during next disease outbreak. (wow!) —Rothberg

@rperezzz: PopTech2011 Social Innovation Fellow Michael Murphy from @MASSDesignLab talks about buildings that heal. twitpic.com/739o8n

 

Crisis Mapping Analysis of London Riots 2011

My colleague Adam White from GroupShot just shared an interesting location analysis study of the recent London riots. The study was carried out by the group Space Syntax and is available here (PDF). The purpose of the study was to test whether the overly complex spatial layout of large post-war housing estates has “an effect on social patterns, often leading to social malaise and anti-social behavior.” While the study’s methods are interesting, I’m concerned about some of the underlying socio-economic assumptions that buttress the analysis.

According to the study, 84% of verified incidents in north London and 96% in south London took place within a five minute walk—400 meters—of both: 1) An established town centre, and 2) a large post-war housing estate. Meanwhile, local centres without large post-war estates nearby were unaffected.

The study makes some interesting assumptions, e.g., “most post-war housing estates have been designed in such a way that they create over-complex, and as a result, under-used spaces. These spaces are populated by large groups of unsupervised children and teenagers, where peer socialisation can occur between them without the influence of adults. This pattern of activity, and the segregation of user groups, is not found in non-estate street networks. Our analysis of court records shows that the majority of convicted rioters in the study areas live on large post-war housing estates.”

The reason I’m uncomfortable with the above has to do with the implied solution, i.e., simplify the complex spaces and bring more social traffic to under-utilized areas. This will ensure that children and teenagers are more supervised and prevent peer socialization from taking place without the influence of adults. In other words, simply replace the “hardware” so the “social software” won’t have any more bugs. Snap, if only Mubarak could have hacked Tahrir Square before the revolution. Sarcasm aside, there were some real and legitimate grievances that motivated some of the protestors in England (and Egypt), which this study doesn’t address.

The next version of the analysis is supposed to include socio-economic data to understand the relationship between deprivation and rioting, which in my mind should have come first. But better late than never. In the meantime, here is a post on the tactical use of technology for nonviolent protests with a reference to London: “Maps, Activism and Technology: Check-In’s with a Purpose.”

The Horn of Africa and the Crisis Mapping Community

“… the Horn of Africa famine and the associated crises gravely affecting millions of people has not animated the crisis-mapping community and its online platforms to the extent of post-Haiti or, more recently, following the 2011 earthquake in Japan.”

I’m somewhat concerned by the phrasing of this statement, which comes from this recent article published by ICT4Peace. Perhaps the author is simply unaware of the repeated offers made by the crisis mapping community to provide crisis mapping solutions, mobile information collection platforms, short codes, call center services, etc., to several humanitarian organizations including UN OCHA, UNDP and WFP over the past three months.

In the case of OCHA, the team in Somalia replied that they had everything under control. In terms of UNDP, the colleagues we spoke with simply did/do not have the capacity, time or skill-set to leverage new crisis mapping solutions to improve their situational awareness or better communicate with disaster affected comm-unities. And WFP explained that lack of access rather than information was the most pressing challenge they were facing (at least two months ago), an issue echoed by two other humanitarian organizations.

This excellent report by Internews details the complete humanitarian tech-nology failure in Dadaab refugee camp and underscores how limited and behind some humanitarian organizations still are vis-a-vis the prioritization of “new” in-formation and communication technologies (ICTs) to improve humanitarian response and the lives of refugees in crisis situations. These organizations require support and core funding to “upgrade”. Throwing crisis mapping technologies at the problem is not going to solve many problems if the under-lying humanitarian mechanisms are not in place to leverage these solutions.

This is not a criticism of humanitarian organizations but rather hard reality. I’ve had numerous conversations with both technology and humanitarian colleagues over the past three months about how to reach for low hanging fruits and catalyze quick-wins with even the most minimal ICT interventions. But as is often the case, the humanitarian community is understandably overwhelmed and genu-inely trying to do the best they can given the very difficult circumstances. Indeed, Somalia presents a host of obvious challenges and risks that were not present in either Haiti or Japan. (Incidentally, only a fraction of the crisis mapping commu-nity was involved in Japan compared to overall efforts in Somalia).

Perhaps ICT4Peace is also unaware that some colleagues and I spent many long days and nights in August and September preparing the launch of a live crisis map for Somalia, which ESRI, Google, Nethope and several other groups provided critical input on. See my blog post on this initiative here. But the project was torpedoed by a humanitarian organization that was worried about the conse-quences of empowering the Somali Diaspora, i.e., that they would become more critical of the US government’s perceived inaction as a result of the information they collected—a consequence I personally would have championed as an indica-tor of success.

Maybe ICT4Peace is also unaware that no humanitarian organization formally requested the activation of the Standby Volunteer Task Force (SBTF) in August. That said, the SBTF did engage in this pilot project to crowdsource the geo-tagging of shelters in Somalia in September as a simple trial run. Since then, the SBTF has officially partnered with UNHCR and the Joint Research Center (JRC) to geo-tag IDP camps in specific regions in Somalia next month. Digital Globe is a formal partner in this project, as is Tomnod. Incidentally, JRC is co-hosting this year’s International Conference of Crisis Mappers (ICCM 2011).

ICT4Peace is perhaps also not aware of a joint project between Ushahidi and UN OCHA Kenya to provide crisis mapping support, or of recent conversations with Al Jazeera, Souktel, the Virgin Group, K’naan, PopTech, CeaseFire, PeaceTXT, GSMA, DevSeed and others on implementing crisis mapping and SMS solutions for Somalia. In addition, the Humanitarian Open Street Map Team (HOT) has been busy improving the data for Somalia and the only reason they haven’t been able to go full throttles forward is because of data licensing issues beyond their control. Colleagues from the Harvard Humanitarian Initiative (HHI) have also been offering their help where and when they can.

In sum, to say that the crisis mapping community has not been as “animated” in response to the crisis in the Horn is misleading and rather unfortunate given that ICT4Peace is co-hosting this year’s International Conference of Crisis Mappers (ICCM 2011). All ICT4Peace had to do was to send one simple email to the CrisisMappers.net membership to get all the above information (and likely more). Just because these efforts are not captured on CNN or on the front pages of the UN Chronicle does not mean that there haven’t been numerous ongoing efforts behind the scenes by dozens of different partners and members of the crisis mapping community.

I would therefore not be so quick to dismiss the perceived inaction of this comm-unity. I would also not make an automatic assumption that crisis mapping platforms and mobile technology solutions will always be “easy” or feasible to deploy in every context, especially if this is attempted reactively in the middle of a complex humanitarian crisis. Both Haiti and Japan provided permissive envi-ronments, unlike recent crisis mapping projects in Libya, Egypt and the Sudan which present serious security challenges. Finally, if direct offers of support by the crisis mapping community are not leveraged by field-based humanitarian organizations, then how exactly is said crisis mapping community supposed to be more animated?

Tracking Population Movements using Mobile Phones and Crisis Mapping: A Post-Earthquake Geospatial Study in Haiti

I’ve been meaning to blog about this project since it was featured on BBC last month: “Mobile Phones Help to Target Disaster Aid, says Study.” I’ve since had the good fortune of meeting Linus Bengtsson and Xin Lu, the two lead authors of this study (PDF), at a recent strategy meeting organized by GSMA. The authors are now launching “Flowminder” in affiliation with the Karolinska Institutet in Stockholm to replicate their excellent work beyond Haiti. If “Flowminder” sounds familiar, you may be thinking of Hans Rosling’s “Gapminder” which also came out of the Karolinska Institutet. Flowminder’s mission: “Providing priceless information for free for the benefit of those who need it the most.”

As the authors note, “population movements following disasters can cause important increases in morbidity and mortality.” That is why the UN sought to develop early warning systems for refugee flows during the 1980’s and 1990’s. These largely didn’t pan out; forecasting is not a trivial challenge. Nowcasting, however, may be easier. That said, “no rapid and accurate method exists to track population movements after disasters.” So the authors used “position data of SIM cards from the largest mobile phone company in Haiti (Digicel) to estimate the magnitude and trends of population movements following the Haiti 2010 earthquake and cholera outbreak.”

The geographic locations of SIM cards were determined by the location of the mobile phone towers that SIM cards were connecting to when calling. The authors followed the daily positions of 1.9 million SIM cards for 42 days prior to the earthquake and 158 days following the quake. The results of the analysis reveal that an estimated 20% of the population in Port-au-Prince left the city within three weeks of the earthquake. These findings corresponded well with of a large, retrospective population based survey carried out by the UN.

“To demonstrate feasibility of rapid estimates and to identify areas at potentially increased risk of outbreaks,” the authors “produced reports on SIM card move-ments from a cholera outbreak area at its immediate onset and within 12 hours of receiving data.” This latter analysis tracked close to 140,000 SIM cards over an 8 day period. In sum, the “results suggest that estimates of population movements during disasters and outbreaks can be delivered rapidly and with potentially high validity in areas with high mobile phone use.”

I’m really keen to see the Flowminder team continue their important work in and beyond Haiti. I’ve invited them to present at the International Conference of Crisis Mappers (ICCM 2011) in Geneva next month and hope they’ll be able to join us. I’m interested to explore the possibilities of combining this type of data and analysis with crowdsourced crisis information and satellite imagery analysis. In addition, mobile phone data can also be used to estimate the hardest hit areas after a disaster. For more on this, please see my previous blog post entitled “Analyzing Call Dynamics to Assess the Impact of Earthquakes” and this post on using mobile phone data to assess the impact of building damage in Haiti.

Crowdsourcing Will Solve All Humanitarian Problems

Here’s one of my favorite false arguments: “There are some people who believe that crowdsourcing will solve all humanitarian challenges….” So said a good colleague of mine vis-a-vis crisis response at a recent strategy meeting. Of course, when I pressed him for names, he didn’t have a reply. I don’t know anyone who subscribes to the above-mentioned point of view. While I understand that he made the statement in jest and primarily to position himself, I’m concerned that some in the humanitarian community actually believe this comment to be true.

First of all, suggesting that some individuals subscribe to an extreme point of view is a cheap debating tactic and a real pet peeve of mine. Simply label your “opponent” as holding a fundamentalist view of the world and everything you say following that statement holds true, easily discrediting your competition in the eyes of the jury. Surely we’ve moved beyond these types of false arguments in the crisis mapping community.

Secondly, crowdsourcing  is simply one among several methodologies that can, in some cases, be useful to collect information following a crisis. And as mentioned in this previous blog post entitled, “Demystifying Crowdsourcing: An Intro-duction to Non-Random Sampling,” the use of crowdsourcing, like any metho-dology, comes with advantages and disadvantages that depend both on goals and context. Surely, this is now common knowledge.

My point here is neither defend nor dismiss the use of crowdsourcing. My hope is that we move away from such false, dichotomous debates to conversations that recognize the complexities of an evolving situation; dialogues that value having more methodologies in the toolbox rather than fewer—and corresponding manuals that give us clarification on trade-offs and appropriate guidance on when to use which methods, why and how. Crowdsourcing crisis information has never been an either-or argument, so lets not turn it into one. Polarizing the con-versation with fictitious claims will only get in the way of learning and innovation.

Detecting Emerging Conflicts with Web Mining and Crisis Mapping

My colleague Christopher Ahlberg, CEO of Recorded Future, recently got in touch to share some exciting news. We had discussed our shared interests a while back at Harvard University. It was clear then that his ideas and existing technologies were very closely aligned to those we were pursuing with Ushahidi’s Swift River platform. I’m thrilled that he has been able to accomplish a lot since we last spoke. His exciting update is captured in this excellent co-authored study entitled “Detecting Emergent Conflicts Through Web Mining and Visualization” which is available here as a PDF.

The study combines almost all of my core interests: crisis mapping, conflict early warning, conflict analysis, digital activism, pattern recognition, natural language processing, machine learning, data visualization, etc. The study describes a semi-automatic system which automatically collects information from pre-specified sources and then applies linguistic analysis to user-specified extract events and entities, i.e., structured data for quantitative analysis.

Natural Language Processing (NLP) and event-data extraction applied to crisis monitoring and analysis is of course nothing new. Back in 2004-2005, I worked for a company that was at the cutting edge of this field vis-a-vis conflict early warning. (The company subsequently joined the Integrated Conflict Early Warning System (ICEWS) consortium supported by DARPA). Just a year later, Larry Brilliant told TED 2006 how the Global Public Health Information Net-work (GPHIN) had leveraged NLP and machine learning to detect an outbreak of SARS 3 months before the WHO. I blogged about this, Global Incident Map, European Media Monitor (EMM), HavariaHealthMap and Crimson Hexagon back in 2008. Most recently, my colleague Kalev Leetaru showed how applying NLP to historical data could have predicted the Arab Spring. Each of these initiatives represents an important effort in leveraging NLP and machine learning for early detection of events of interest.

The RecordedFuture system works as follows. A user first selects a set of data sources (websites, RSS feeds, etc) and determines the rate at which to update the data. Next, the user chooses one or several existing “extractors” to find specific entities and events (or constructs a new type). Finally, a taxonomy is selected to specify exactly how the data is to be grouped. The data is then automatically harvested and passed through a linguistics analyzer which extracts useful information such as event types, names, dates, and places. Finally, the reports are clustered and visualized on a crisis map, in this case using an Ushahidi platform. This allows for all kinds of other datasets to be imported, compared and analyzed, such as high resolution satellite imagery and crowdsourced data.

A key feature of the RecordedFuture system is that extracts and estimates the time for the event described rather than the publication time of the newspaper article parsed, for example. As such, the harvested data can include both historic and future events.

In sum, the RecordedFuture system is composed of the following five features as described in the study:

1. Harvesting: a process in which text documents are retrieved from various sources and stored in the database. The documents are stored for long-term if permitted by terms of use and IPR legislation, otherwise they are only stored temporarily for the needed analysis.

2. Linguistic analysis: the process in which the retrieved texts are analyzed in order to extract entities, events, time and location, etc. In contrast to other components, the linguistic analysis is language dependent.

3. Refinement: additional information can be obtained in this process by synonym detection, ontology analysis, and sentiment analysis.

4. Data analysis: application of statistical and AI-based models such as Hidden Markov Models (HMMs) and Artificial Neural Networks (ANNs) to generate predictions about the future and detect anomalies in the data.

5. User experience: a web interface for ordinary users to interact with, and an API for interfacing to other systems.

The authors ran a pilot that “manually” integrated the RecordedFuture system with the Ushahidi platform. The result is depicted in the figure below. In the future, the authors plan to automate the creation of reports on the Ushahidi platform via the RecordedFuture system. Intriguingly, the authors chose to focus on protest events to demo their Ushahidi-coupled system. Why is this intriguing? Because my dissertation analyzed whether access to new information and communication technologies (ICTs) are statistically significant predictors of protest events in repressive states. Moreover, the protest data I used in my econometric analysis came from an automated NLP algorithm that parsed Reuters Newswires.

Using RecordedFuture, the authors extracted some 6,000 protest event-data for Quarter 1 of 2011. These events were identified and harvested using a “trained protest extractor” constructed using the system’s event extractor frame-work. Note that many of the 6,000 events are duplicates because they are the same events but reported by different forces. Not surprisingly, Christopher and team plan to develop a duplicate detection algorithm that will also double as a triangulation & veracity scoring feature. I would be particularly interested to see them do this kind of triangulation and validation of crowdsourced data on the fly.

Below are the protest events picked up by RecordedFuture for both Tunisia and Egypt. From these two figures, it is possible to see how the Tunisian protests preceded those in Egypt.

The authors argue that if the platform had been set up earlier this year, a user would have seen the sudden rise in the number of protests in Egypt. However, the authors acknowledge that their data is a function of media interest and attention—the same issue I had with my dissertation. One way to overcome this challenge might be by complementing the harvested reports with crowdsourced data from social media and Crowdmap.

In the future, the authors plan to have the system auto-detect major changes in trends and to add support for the analysis of media in languages beyond English. They also plan to test the reliability and accuracy of their conflict early warning algorithm by comparing their forecasts of historical data with existing conflict data sets. I have several ideas of my own about next steps and look forward to speaking with Christopher’s team about ways to collaborate.