Monthly Archives: June 2010

The Crowdsourcing Detective: Crisis, Deception and Intrigue in the Twittersphere

My colleague Anahi Ayala recently started a blog called “Diary of a Crisis Mapper.” I highly recommend it. Her latest blog post on Ushahidi-Chile relates some intriguing detective work that all started with the following tweet:

The Tweet was mapped on Ushahidi-Chile. You’ll note in the discussion forum part of this report that a volunteer (Annette) updated the alert by adding that the person had been rescued. She did this after seeing a Tweet from @biodome10 saying that he had been rescued. This is when I emailed Anahi (on February 27th) to ask whether she could try and confirm this case.

It turns out the person behind the Twitter handle @biodome10 was pretending to be Gerry Fraley, a journalist with the Dallas Morning News. Biodome10 even used Gerry Fraley’s own profile picture on their Twitter profile. It also turns out that @biodome10 has a history of disseminating false information. But this first Tweet set in motion an intriguing series of events.

Someone in Chile saw the Tweet and actually called the police to report that a person was trapped under a building at that address. Anahi did some online detective work and found out that “the police decided to send 3 trucks from the fire dept, 30 cops from the rescue department, and the chief of Security in person to the address to save the person in question (see James’s blog on this, and also reported from @criverap from Twitter).”

Anahi followed this lead further:

It was Saturday night in Santiago and even if there had been one of the worst earthquake of the last 25 years, life was still going on. So it was for Dinamarca Pedro and Vargas Elba, a couple that was celebrating that night its 39 wedding anniversary. Of course, there was not much to celebrate, so at 11pm Pedro and Elba were preparing to go to bed. They lived in Lautaro 1712, Santiago, Chile.

When the door was open by force by police, carabineros and detectives, with the chief of Security in person leading the operation, the couple almost had a heart attack. No person to rescue, only an old couple which is going to remember its 39 anniversary for the rest of its life! (reported on Las Ultima Noticias Newspaper on the 1st of March 2010).

That’s not all, @biodome10 sent out a second false Tweet the following day, which was also mapped on Ushahidi-Chile:

Again the police mobilized to the location after seeing the Tweet on Twitter. When they arrived at the scene, there were no collapsed buildings in that part of the city, so they promptly left. This time though, the Tweet actually ended up on hundreds of t-shirts as part of a fund raising campaign for the Red Cross.

As Anahi writes:

Both those tweets were false. Now we know that because we know that Biodome10 was posting false information. But the Chilean police didn’t have the time and the resources to verify this information. They had priorities: go and save people as quickly as possible. They wasted time two times following both those twitter messages, while they could have been use this time to save other people in danger. Biodome10 was not only playing with social networks and Twitter, he was playing with people’s life.

At the same time, though, Anahi notes that she was able to do all her detective work right from her laptop by just accessing the web:

[...] crowd sourcing verification of information works. I have been able to find all those information just by sitting on my desk in NYC/Boston. Others have done investigations for me. I have been just collecting people’s tweets, read their blogs, and put together all the information. The fact that different people, from different parts of the world have been investigating by themselves this issue, has given me the possibility to find out that Biodome10 was a liar, independently from his identity and to be sure that the two reports we posted on our Ushahidi map were actually false.

To this end, Anahi advises groups that crowdsource crisis information to set up a verification team that can use crowdsourcing to verify information. This is precisely the principle behind Swift River, using crowdsourcing and natural language processing to crowdsource the filter. As Anahi remarks, “Crowd sourcing verification of information in this case worked. True, it was too late, especially for the police dept in Santiago, but next time we all will be ready.”

Hopefully they’ll be ready and using Swift River since the point of Swift River is to help groups validate information in near real-time. In fact, had Swift River been used during Chile, those two Tweets by @biodome10 would have received the lowest scores because there was only one “witness” reporting the incidents. Of course, Swift River is not a silver bullet as I have already elaborated on here. But we must be careful not to despair because of two false Tweets. They represent 0.0016% of all the reports mapped on Ushahidi-Chile, and they were subsequently verified.

As Dave Warner noted at a panel on USIP Mobile Phones and Afghanistan last week, “Snipers in Afghanistan use roads. Does that mean we should go and tear up all the roads?” Of course not.

I have 3 take-aways from these anecdotes:

  1. I wouldn’t be surprised if there were grounds to take legal action against @biodome10. This should send a signal to others who want to play with people’s lives during disasters.
  2. The Tweets were verifiable by anyone with an Internet connection, the main challenge was time not verifiability.
  3. Both Tweets could have been verified more quickly by others on site had they gotten wind of the information. They could easily have gone down the street and taken a picture showing that there was a wedding anniversary at the first location and intact buildings at the second. They could prove the date by including a picture of the day’s newspaper in the photographs.

In sum, I’m still of the opinion that greater connectivity can lead to greater self-correction. The detective work can be crowdsourced, my dear Watson.

Patrick Philippe Meier

On Technology and Learning, Or Why the Wright Brothers Did Not Create the 747

I am continuously amazed by critics who outlaw learning curves, especially when it comes to new and innovative projects. These critics expect instantaneous perfection from the outset even when new technologies are involved. So when a new project doesn’t meet their satisfaction (based on their own, often arbitrary measurement of success), they publicly castigate the groups behind the projects for their “failures”.

What would the world look like if these critics were in charge? There would be little to no innovation, progress or breakthrough’s. Isaac Newton would never have spoken the words “If I have seen so far it is because I have stood on the shoulders of giants.” There would be no giants, no perspective, and no accumulation of knowledge.

Take the Wright brothers, Orville and Wilbur, who invented and built the world’s first successful airplane by developing controls that made fixed-wing powered flight possible. They too had their critics. Some of them in “the European aviation community had converted the press to an anti-Wright brothers stance. European newspapers, especially in France, were openly derisive, calling them bluffers.” The Wright brothers certainly failed on numerous occasions. But they kept tinkering and experimenting. It was their failures that eventually made them successful. As the Chinese proverb goes, “Failure is not falling down but refusing to get up.”

The Wright brothers did not create the 747 because they had to start from scratch. They first had to figure out the basic principles of flight. Today’s critics of new technology-for-social impact projects are basically blaming NGOs in the developing world for not creating 747s.

Finally, it is problematic that many critics of technology-for-social impact projects have little to no scholarly or professional background in monitoring and evaluation (M&E). These critics think that because they are technology or development experts they know how to evaluate projects—one of the most common mistakes made by self-styled evaluators as noted at the start of The Fletcher School’s introductory course on M&E.

The field of M&E is a science, not an amateur sport. M&E is an independent, specialized field of expertise in it’s own right, one that requires months (if not several semesters) of dedicated study and training.

Patrick Philippe Meier

Demystifying Crowdsourcing: An Introduction to Non-Probability Sampling

The use of crowdsourcing may be relatively new to the technology, business and humanitarian sectors but when it comes to statistics, crowdsourcing is a well known and established sampling method. Crowdsourcing is just non-probability sampling. The crowdsourcing of crisis information is simply an application of non-probability sampling.

Lets first review probability sampling in which every unit in the population being sampled has a known probability (greater than zero) of being selected. This approach makes it possible to “produce unbiased estimates of population totals, by weighting sampled units according to their probability selection.”

Non-probability sampling, on the other hand, describes an approach in which some units of the population have no chance of being selected or where the probability of selection cannot be accurately determined. An example is convenience sampling. The main drawback of non-probability sampling techniques is that “information about the relationship between sample and population is limited, making it difficult to extrapolate from the sample to the population.”

There are several advantages, however. First, non-probability sampling is a quick way to collect way to collect and analyze data in range of settings with diverse populations. The approach is also a “cost-efficient means of greatly increasing the sample, thus enabling more frequent measurement.” In some cases, the non-probability sampling may actually be the only approach available—a common constrain in a lot of research, including many medical studies, not to mention Ushahidi Haiti. The method is also used in exploratory research, e.g., for hypothesis generation, especially when attempting to determine whether a problem exists or not.

The point is that non-probability sampling can save lives, many lives. Much of the data used for medical research is the product of convenience sampling. When you see your doctor, or you’re hospitalized, that is not a representative sample. Should the medical field throw away all this data based on the fact that it constitutes non-probability sampling. Of course not, that would be ludicrous.

The notion of bounded crowdsourcing, which I blogged about here, is also a known sampling technique called purposive sampling. This approach involves targeting experts or key informants. Snowball sampling is another type of non-probability sampling, which may also be applied to crowdsource of crisis information.

In snowball sampling, you begin by identifying someone who meets the criteria for inclusion in your study. You then ask them to recommend others who they may know who also meet the criteria. Although this method would hardly lead to representative samples, there are times when it may be the best method available. Snowball sampling is especially useful when you are trying to reach populations that are inaccessible or hard to find.

A project like Mission 4636 and Ushahidi-Haiti could take advantage of this approach by using two-way SMS communication to ask respondents to spread the word. Individuals who sent in text messages about persons trapped under the rubble could (later) be sent an SMS asking them to share the 4636 short code with people who may know of other trapped individuals. When the humanitarian response began to scale during the search and rescue operations, purposive sampling using UN personnel could also have been implemented.

In contrast to non-probability sampling techniques, probability sampling often requires considerable time and extensive resources. Furthermore, non-response effects can easily turn any probability design into non-probability sampling if the “characteristics of non-response are not well understood” since these modify each unit’s probability of being sampled.

This is not to suggest that one approach is better than the other since this depends entirely on the context and research question.

Patrick Philippe Meier

Think You Know What Ushahidi Is? Think Again

Ushahidi is the name of both the organization (Ushahidi Inc) and the platform. This understandably leads to some confusion. So let me elaborate on both.

Ushahidi the platform is a piece of software, not a methodology. The Ushahidi platform allows users to map information of interest to them. I like to think of it as democratizing map making in the style of neogeography. How users choose to collect the information they map is where methodology comes in. Users themselves select which methodology they want to use, such as representative sampling, crowdsourcing, etc. In other words, Ushahidi is not exclusively a platform for crowdsourcing. Nor is Ushahidi restricted to mapping crisis information. A wide range of events can be mapped using the platform. Non-events can also be mapped, such as football stadiums, etc.

The platform versus methodology distinction is significant. Why? Because new users often don’t realize that they themselves need to think through which methodology they should use to collect information. Furthermore, once they’ve chosen the methodology, they need to set up the appropriate tools to collect information using that methodology, and then collect.

For example, if a user wants to collect election data using representative sampling, they will need to ensure that they select a sample of polling stations that are likely to be representative of the overall population in terms of voting behavior. They will then need to decide whether they want to use SMS, email, phone calls, etc., to relay that information. Next, they’ll want to hire trusted monitors and train them on what and how to report. But none of this has anything to do with Ushahidi the platform.

Here’s an analogy: Microsoft Word won’t tell me what methodology to use if I want to write a paper on the future of technology. That is up to me, the author, to decide. If I don’t have any training in research methods and design, then I need to get up to speed independently. MS Word won’t provide me with insights on research methods. MS Word is just the platform. Coming back to Ushahidi, if an organization does not have adequate expertise, staff, capacity, time and resources to deploy Ushahidi, that is not the fault of the platform.

In many ways, the use of Ushahidi will only be as good as the organization or persons using the tool.

Ushahidi is only 10% of the solution (graphic by Chris Blow)

As my colleague Ory aptly cautioned: “Don’t get too jazzed up about Ushahidi. It is only 10% of the solution.” The other 90% is up to the organization using the platform. If they don’t have their act together, the Ushahidi platform won’t change that. If they do and successfully deploy the Ushahidi platform, then at least 90% of the credit goes to them.

Ushahidi the organization is a non-profit tech company. The group is not a humanitarian organization. We do not take the lead in deployments. In the case of Haiti, I launched the Ushahidi platform at The Fletcher School (where I am a PhD student) and where graduate students (not Ushahidi employees) created a “live” map of the disaster for several weeks. The Ushahidi tech team provided invaluable technical support around the clock during those weeks. It was thus a partnership led by The Fletcher Team.

We do not have a comparative advantage in deploying platforms and our core mission is to continue developing the Ushahidi platform. On occasion, we partner on select projects but do not take the lead on these projects. Why do we partner at all? Because we are required to diversify our business model as part of the grant we received from the Omidyar Network. And I think that’s a good idea.

Patrick Philippe Meier

Information Sharing During Crisis Management in Hierarchical vs. Network Teams

The month of May turned out to be ridiculously busy, so much so that I haven’t been able to blog. And when that happens, I know I’m doing too much. So my plan for June is to slow down, prioritize and do more of what I enjoy, e.g., blog.

In the meantime, the Journal of Contingencies and Crisis Management just published an interesting piece on “Information Sharing During Crisis Management in Hierarchical vs. Network Teams.” The topic and findings have implications for digital activism as well as crisis management.

Here’s the abstract:

This study examines the differences between hierarchical and network teams in emergency management. A controlled experimental environment was created in which we could study teams that differed in decision rights, availability of information, information sharing, and task division. Thirty-two teams of either two (network) or three (hierarchy) participants (N=80 in total) received messages about an incident in a tunnel with high-ranking politicians possibly being present. Based on experimentally induced knowledge, teams had to decide as quickly and as accurately as possible what the likely cause of the incident was: an attack by Al Qaeda, by anti-globalists, or an accident. The results showed that network teams were overall faster and more accurate in difficult scenarios than hierarchical teams. Network teams also shared more knowledge in the difficult scenarios, compared with the easier scenarios. The advantage of being able to share information that is inherent in network teams is thus contingent upon the type of situation encountered.

The authors define a hierarchical team as one in which members pass on information to a leader, but not to each other. In a network team, members can freely exchange information with each other. Here’s more on the conclusions derived by the study:

Our goal with the present study was to focus on a relatively simple comparison between a classic hierarchical structure and a network structure. The structures differed in terms of decision rights, availability of information, information sharing, and task division. Although previous research has not found unequivocal support in terms of speed or accuracy for one structure or the other, we expected our network structure to perform better and faster on the decision problems. We also expected the network teams to learn faster and exchange more specialist knowledge than the hierarchical teams.

Our hypotheses are partially supported. Network teams are indeed faster than hierarchical teams. Further analyses showed that network teams were, on average, as fast as the slowest working individual in the hierarchical teams. Analyses also showed that network teams very early on converged on a rapid mode of arriving at a decision, whereas hierarchical teams took more time. The extra time needed by hierarchical teams is therefore due to the time needed by the team leader to arrive at his or her decision.

We did not find an overall effect of team structure on the quality of team decision, contrary to our prediction. Interestingly, we did find that network teams were significantly better than hierarchical teams on the Al Qaeda scenarios (as compared with the anti-globalist scenarios). The Al Qaeda scenarios were the most difficult scenarios. Furthermore, scores on the Post-test showed that there was a larger transfer of knowledge on Al Qaeda from the specialist to the nonspecialist in the network condition as compared with the hierarchical condition. These results indicate that a high level of team member interaction leads to shared specialist knowledge, particularly in difficult scenarios. This in turn leads to more accurate decisions.

This study focused on the information assessment part of crisis management, not on the operative part. However, there may not be that much of a difference in terms of the actual teamwork involved. When team members have to carry out particular tasks, they may frequently also have to share specialist knowledge. Wilson, Salas, Priest, and Andrews (2007) have studied how teamwork breakdowns in the military may contribute to fratricide, the accidental shooting of one’s own troops rather than the enemy. This is obviously a very operative part of the military task. Teamwork breakdowns are subdivided into communication, coordination and cooperation, with information exchange mutual performance monitoring, and mutual trust as representative teamwork behaviours for each category (Wilson et al., 2007).

We believe that it is precisely these behaviours that are fostered by network structures rather than hierarchical structures. Network structures allow teams to exchange information quickly, monitor each other’s performance, and build up mutual trust. This is just as important in the operative part of crisis management work as it is in the information assessment part.

In conclusion, then, network teams are faster than hierarchical teams, while at the same time maintaining the same level of accuracy in relatively simple environments. In relatively complex environments, on the other hand, network teams arrive at correct decisions more frequently than hierarchical teams. This may very likely be due to a better exchange of knowledge in network teams.

Patrick Philippe Meier