Tag Archives: Egypt

Egypt Twitter Map of iPhone, Android and Blackberry Users

Colleagues at GNIP and MapBox recently published this high-resolution map of iPhone, Android and Blackberry users in the US (click to enlarge). “More than 280 million Tweets posted from mobile phones reveal geographic usage patterns in unprecedented detail.” These patterns are often insightful. Some argue that “cell phone brands say something about socio-economics – it takes a lot of money to buy a new iPhone 5,” for example (1). So a map of iPhone users based on where these users tweet reveals where relatively wealthy people live.

Phones USA

As announced in this blog post, colleagues and I at QCRI, Harvard, MIT and UNDP are working on an experimental R&D project to determine whether Big Data can inform poverty reduction strategies in Egypt. More specifically, we are looking to test whether tweets provide a “good enough” signal of changes in unemployment and poverty levels. To do this, we need ground truth data. So my MIT colleague Todd Mostak put together the following maps of cell phone brand ownerships in Egypt using ~3.5 million geolocated tweets from October 2012 to June 2013. Red dots represent the location of tweets posted by Android users; Green dots – iPhone; Purple – Blackberry. Click figures below to enlarge.

Egypt Mobile Phones

Below is a heatmap of the % of Android users. As Todd pointed out in our email exchanges, “Note the lower intensity around Cairo.”

Egypt Android

This heatmap depicts the density of tweeting iPhone users:

Egypt iPhone users

Lastly, the heatmap below depicts geo-tagged tweets posted by Blackberry users.

BB Egypt

As Todd notes, “We can obviously break these down by shyiyakha and regress against census data to get a better idea of how usage of these different devices correlate with proxy for income, but at least from these maps it seems clear that iPhone and Blackberry are used more in urban, higher-income areas.” Since this data is time-stamped, we may be able to show whether/how these patterns changed during last week’s widespread protests and political upheaval.

bio

Using Twitter to Analyze Secular vs. Islamist Polarization in Egypt (Updated)

Large-scale events leave an unquestionable mark on social media. This was true of Hurricane Sandy, for example, and is also true of the widespread protests in Egypt this week. On Wednesday, the Egyptian Military responded to the large-scale demonstrations against President Morsi by removing him from power. Can Twitter provide early warning signals of growing political tension in Egypt and elsewhere? My QCRI colleagues Ingmar Weber & Kiran Garimella and Al-Jazeera colleague Alaa Batayneh have been closely monitoring (PDF) these upheavals via Twitter since January 2013. Specifically, they developed a Political Polarization Index that provides early warning signals for increased social tensions and violence. I will keep updating this post with new data, analysis and graphs over the next 24 hours.

morsi_protests

The QCRI team analyzed some 17 million Egyptian tweets posted by two types of Twitter users—Secularists and Islamists. These user lists were largely drawn from this previous research and only include users that provide geographical information in their Twitter profiles. For each of these 7,000+ “seed users”, QCRI researchers downloaded their most recent 3,200 tweets along with a set of 200 users who retweet their posts. Note that both figures are limits imposed by the Twitter API. Ingmar, Kiran and Alaa have also analyzed users with no location information, corresponding to 65 million tweets and 20,000+ unique users. Below are word clouds of terms used in Twitter profiles created by Islamists (left) and secularists (right).

Screen Shot 2013-07-06 at 2.58.25 PM

QCRI compared the hashtags used by Egyptian Islamists and secularists over a year to create an insightful Political Polarization Index. The methodology used to create this index is described in more detail in this post’s epilogue. The graph below displays the overall hashtag polarity over time along with the number of distinct hashtags used per time interval. As you’ll note, the graph includes the very latest data published today. Click on the graph to enlarge.

hashtag_polarity_over_time_egypt_7_july

The spike in political polarization towards the end of 2011 appears to coincide with “the political struggle over the constitution and a planned referendum on the topic.” The annotations in the graph refer to the following violent events:

A – Assailants with rocks and firebombs gather outside Ministry of Defense to call for an end to military rule.

B – Demonstrations break out after President Morsi grants himself increased power to protect the nation. Clashes take place between protestors and Muslim Brotherhood supporters.

C, D - Continuing protests after the November 22nd declaration.

E – Demonstrations in Tahrir square, Port Said and all across the country.

F,G – Demonstrations in Tahrir square.

H,I – Massive demonstrations in Tahrir and removal of President Morsi.

In sum, the graph confirms that the political polarization hashtag can serve as a barometer for social tensions and perhaps even early warnings of violence. “Quite strikingly, all outbreaks of violence happened during periods where the hashtag polarity was comparatively high.” This also true for the events of the past week, as evidenced by QCRI’s political polarization dashboard below. Click on the figure to enlarge. Note that I used Chrome’s translate feature to convert hashtags from Arabic to English. The original screenshot in Arabic is available here (PNG).

Hashtag Analysis

Each bar above corresponds to a week of Twitter data analysis. When bars were initially green and yellow during the beginnings of Morsi’s Presidency (scroll left on the dashboard for the earlier dates). The change to red (heightened political polarization) coincides with increased tensions around the constitutional crisis in late November, early December. See this timeline for more information. The “Tending Score” in the table above combines volume with recency. A high trending score means the hashtag is more relevant to the current week. 

The two graphs below display political polarization over time. The first starts from January 1, 2013 while the second from June 1, 2013. Interestingly, February 14th sees a dramatic drop in polarization. We’re not sure if this is a bug in the analysis or whether a significant event (Valentine’s?) can explain this very low level of political polarization on February 14th. We see another major drop on May 10th. Any Egypt experts know why that might be?

graph1

The political polarization graph below reveals a steady increase from June 1st through to last week’s massive protests and removal of President Morsi.

graph2

To conclude, large-scale political events such as widespread political protests and a subsequent regime change in Egypt continue to leave a clear mark on social media activity. This pulse can be captured using a Political Polarization Index based on the hashtags used by Islamists and secularists on Twitter. Furthermore, this index appears to provide early warning signals of increasing tension. As my QCRI colleagues note, “there might be forecast potential and we plan to explore this further in the future.”

Bio

Acknowledgements: Many thanks to Ingmar and Kiran for their valuable input and feedback in the drafting of this blog post.

Methods: (written by Ingmar): The political polarization index was computed as follows. The analysis starts by identifying a set of Twitter users who are likely to support either Islamists or secularists in Egypt. This is done by monitoring retweets posted by a set of seed users. For example, users who frequently retweet Muhammad Morsi  and never retweeting El Baradei would be considered Islamist supporters. (This same approach was used by Michael Conover and colleagues to study US politics).

Once politically engaged and polarized users are identified, their use of hashtags is monitored over time. A “neutral” hashtags such as #fb or #ff is typically used by both camps in Egypt in roughly equal proportions and would hence be assigned a 50-50 Islamist-secular leaning. But certain hashtags reveal much more pronounced polarization. For example, the hashtag #tamarrod is assigned a 0-100 Islamist-secular score. Tamarrod refers to the “Rebel” movement, the leading grassroots movement behind the protests that led to Morsi’s ousting.

Similarly the hashtag #muslimsformorsi is assigned a 90-10 Islamist-secular score, which makes sense as it is clearly in support of Morsi. This kind of numerical analysis is done on a weekly basis. Hashtags with a 50-50 score in a given week have zero “tension” whereas hashtags with either 100-0 or 0-100 have maximal tension. The average tension value across all hashtags used in a given week is then plotted over time. Interestingly, this value, derived from hashtag usage in a language-agnostic manner, seems to coincide with outbreaks of violence on the ground as shown in bar chart above.

Using Big Data to Inform Poverty Reduction Strategies

My colleagues and I at QCRI are spearheading a new experimental Research and Development (R&D) project with the United Nations Development Program (UNDP) team in Cairo, Egypt. Colleagues at Harvard University, MIT and UC Berkeley have also joined the R&D efforts as full-fledged partners. The research question: can an analysis of Twitter traffic in Egypt tell us anything about changes in unemployment and poverty levels? This question was formulated with UNDP’s Cairo-based Team during several conversations I had with them in early 2013.

Egyptian Tweets

As is well known, a major challenge in the development space is the lack of access to timely socio-economic data. So the question here is whether alternative, non-traditional sources of information (such as social media) can provide a timely and “good enough” indication of changing trends. Thanks to our academic partners, we have access to hundreds of millions of Egyptian tweets (both historical and current) along with census and demographic data for ground-truth purposes. If the research yields robust results, then our UNDP colleagues could draw on more real-time data to complement their existing datasets, which may better inform some of their local poverty reduction and development strategies. This more rapid feedback loop could lead to faster economic empowerment for local communities in Egypt. Of course, there are many challenges to working with social data vis-a-vis representation and sample bias. But that is precisely why this kind of experimental research is important—to determine whether any of our results are robust to biases in phone ownership, twitter-use, etc.

bio

Social Media as Passive Polling: Prospects for Development & Disaster Response

My Harvard/MIT colleague Todd Mostak wrote his award-winning Master’s Thesis on “Social Media as Passive Polling: Using Twitter and Online Forums to Map Islamism in Egypt.” For this research, Todd evaluated the “potential of Twitter as a source of time-stamped, geocoded public opinion data in the context of the recent popular uprisings in the Middle East.” More specifically, “he explored three ways of measuring a Twitter user’s degree of political Islamism.” Why? Because he wanted to test the long-standing debate on whether Islamism is associated with poverty.

Screen Shot 2013-02-18 at 11.17.09 AM

So Todd collected millions of geo-tagged tweets from Egypt over a six month period, which he then aggregated by census district in order to regress proxies for poverty against measures of Islamism drived from the tweets and the users’ social graphs. His findings reveal that “Islamist sentiment seems to be positively correlated with male unemployment, illiteracy, and percentage of land used in agriculture and negatively correlated with percentage of men in their youth aged 15-25. Note that female variables for unemployment and age were statistically insignificant.” As with all research, there are caveats such as the weighting scale used for the variables and questions over the reliability of census variables.

Screen Shot 2013-02-18 at 11.15.59 AM

To carry out his graduate research, Todd built a web-enabled database (MapD) powered by a Graphics Processing Units (GPU) to perform real-time querying and visualization of big datasets. He is now working with Harvard’s Center for Geographic Analysis (CGA) to put make this available via a public web interface called Tweetmap. This Big Data streaming and exploration tool presen-tly displays 119 million tweets from 12/10/2012 to 12/31/2012. He is adding 6-7 million new georeferenced tweets per day (but these are not yet publicly available on Tweetmap). According to Todd, the time delay from live tweet to display on the map is about 1 second. Thanks to this GPU-powered approach, he expects that billions of tweets could be displayed in real-time.

Screen Shot 2013-02-18 at 11.14.02 AM

As always with impressive projects, no one single person was behind the entire effort. Ben Lewis, who heads the WorldMap initiative at CGA deserves a lot of credit for making Tweetmap a reality. Indeed, Todd collaborated directly with CGA’s Ben Lewis throughout this project and benefited extensively from his expertise. Matt Bertrand (lead developer for CGA) did the WorldMap-side integration of MapD to create the TweetMap interface.

Todd and I recently spoke about integrating his outstanding work on automated live mapping to QCRI’s Twitter Dashboard for Disaster Response. Exciting times. In the meantime, Todd has kindly shared his dataset of 700+ million geotagged tweets for my team and I to analyze. The reason I’m excited about this approach is best explained with this heatmap of the recent snow-storm in the northeastern US. Todd is already using Tweetmap for live crisis mapping. While this system filters by keyword, our Dashboard will use machine learning to provide more specific streams of relevant tweets, some of which could be automatically mapped on Tweetmap. See Todd’s Flickr page for more Tweetmap visuals.

Screen Shot 2013-02-18 at 11.30.54 AM

I’m also excited by Todd’s GPU-powered approach for a project I’m exploring with UN and World Bank colleagues. The purpose of that research project is to determine whether socio-economic trends such as poverty and unemployment can be captured via Twitter. Our first case study is Egypt. Depending on the results, we may be able to take it one step further by applying sentiment analysis to real-time, georeferenced tweets to visualize Twitter users’ per-ception vis-a-vis government services—a point of interest for my UN colleagues in Cairo.

bio

Traditional vs. Crowdsourced Election Monitoring: Which Has More Impact?

Max Grömping makes a significant contribution to the theory and discourse of crowdsourced election monitoring in his excellent study: “Many Eyes of Any Kind? Comparing Traditional and Crowdsourced Monitoring and their Contribu-tion to Democracy” (PDF). This 25-page study is definitely a must-read for anyone interested in this topic. That said, Max paints a false argument when he writes: “It is believed that this new methodology almost magically improves the quality of elections [...].” Perhaps tellingly, he does not reveal who exactly believes in this false magic. Nor does he cite who subscribes to the view that  “[...] crowdsourced citizen reporting is expected to have significant added value for election observation—and by extension for democracy.”

My doctoral dissertation focused on the topic of crowdsourced election observa-tion in countries under repressive rule. At no point in my research or during interviews with activists did I come across this kind of superficial mindset or opinion. In fact, my comparative analysis of crowdsourced election observation showed that the impact of these initiatives was at best minimal vis-a-vis electoral accountability—particularly in the Sudan. That said, my conclusions do align with Max’s principle findings: “the added value of crowdsourcing lies mainly in the strengthening of civil society via a widened public sphere and the accumulation of social capital with less clear effects on vertical and horizontal accountability.”

This is huge! Traditional monitoring campaigns don’t strengthen civil society or the public sphere. Traditional monitoring teams are typically composed of inter-national observers and thus do not build social capital domestically. At times, traditional election monitoring programs may even lead to more violence, as this recent study revealed. But the point is not to polarize the debate. This is not an either/or argument but rather a both/and issue. Traditional and crowdsourced election observation efforts can absolutely complement each other precisely because they each have a different comparative advantage. Max concurs: “If the crowdsourced project is integrated with traditional monitoring from the very beginning and thus serves as an additional component within the established methodology of an Election Monitoring Organization, the effect on incentive structures of political parties and governments should be amplified. It would then include the best of both worlds: timeliness, visualization and wisdom of the crowd as well as a vetted methodology and legitimacy.”

Recall Jürgen Habermas and his treatise that “those who take on the tools of open expression become a public, and the presence of a synchronized public increasingly constrains un-democratic rulers while expanding the right of that public.” Why is this important? Because crowdsourced election observation projects can potentially bolster this public sphere and create local ownership. Furthermore, these efforts can help synchronize shared awareness, an important catalyzing factor of social movements, according to Habermas. Furthermore, my colleague Phil Howard has convincingly demonstrated that a large active online civil society is a key causal factor vis-a-vis political transitions towards more democratic rule. This is key because the use of crowdsourcing and crowd-mapping technologies often requires some technical training, which can expand the online civil society that Phil describes and render that society more active (as occurred in Egypt during the 2010 Parliamentary Elections—see  dissertation).

The problem? There is very little empirical research on crowdsourced election observation projects let alone assessments of their impact. Then again, these efforts at crowdsourcing are only a few years old and many do’ers in this space are still learning how to be more effective through trial and error. Incidentally, it is worth noting that there has also been very little empirical analysis on the impact of traditional monitoring efforts: “Further quantitative testing of the outlined mechanisms is definitely necessary to establish a convincing argument that election monitoring has positive effects on democracy.”

In the second half of his important study, Max does an excellent job articulating the advantages and disadvantages of crowdsourced election observation. For example, he observes that many crowdsourced initiatives appear to be spon-taneous rather than planned. Therein lies part of the problem. As demonstrated in my dissertation, spontaneous crowdsourced election observation projects are highly unlikely to strengthen civil society let alone build any kind of social capital. Furthermore, in order to solicit a maximum number of citizen-generated election reports, a considerable amount of upfront effort on election awareness raising and education needs to take place in addition to partnership outreach not to mention a highly effective media strategy.

All of this requires deliberate, calculated planning and preparation (key to an effective civil society), which explains why Egyptian activists were relatively more successful in their crowdsourced election observation efforts compared to their counterparts in the Sudan (see dissertation). This is why I’m particularly skeptical of Max’s language on the “spontaneous mechanism of protection against electoral fraud or other abuses.” That said, he does emphasize that “all this is of course contingent on citizens being informed about the project and also the project’s relevance in the eyes of the media.”

I don’t think that being informed is enough, however. An effective campaign not only seeks to inform but to catalyze behavior change, no small task. Still Max is right to point out that a crowdsourced election observation project can “encou-rage citizens to actively engage with this information, to either dispute it, confirm it, or at least register its existence.” To this end, recall that political change is a two-step process, with the second—social step—being where political opinions are formed (Katz and Lazarsfeld 1955). “This is the step in which the Internet in general, and social media in particular, can make a difference” (Shirky 2010). In sum, Max argues that “the public sphere widens because this engagement, which takes place in the context of the local all over the country, is now taken to a wider audience by the means of mapping and real-time reporting.” And so, “even if crowdsourced reports are not acted upon, the very engagement of citizens in the endeavor to directly make their voices heard and hold their leaders accountable widens the public sphere considerably.”

Crowdsourcing efforts are fraught with important and very real challenges, as is already well known. Reliability of crowdsourced information, risk of hate speech spread via uncontrolled reports, limited evidence of impact, concerns over security and privacy of citizen reporters, etc. That said, it is important to note that this “field” is evolving and many in this space are actively looking for solutions to these challenges. During the 2010 Parliamentary Elections in Egypt, the U-Shahid project was able to verify over 90% of the crowdsourced reports. The “field” of information forensics is becoming more sophisticated and variants to crowdsourcing such as bounded crowdsourcing and crowdseeding are not only being proposed but actually implemented.

The concern over unconfirmed reports going viral has little to do with crowd-sourcing. Moreover, the vast majority of crowdsourced election observation initiatives I have studied moderate all content before publication. Concerns over security and privacy are issues not limited to crowdsourced election observation and speak to a broader challenge. There are already several key initiatives underway in the humanitarian and crisis mapping community to address these important challenges. And lest we forget, there are few empirical studies that demonstrate the impact of traditional monitoring efforts in the first place.

In conclusion, traditional monitors are sometimes barred from observing an election. In the past, there have been few to no alternatives to this predicament. Today, crowdsourced efforts are sure to swell up. Furthermore, in the event that traditional monitors conclude that an election was stolen, there’s little they can do to catalyze a local social movement to place pressure on the thieves. This is where crowdsourced election observation efforts could have an important contribution. To quote Max: “instead of being fearful of the ‘uncontrollable crowd’ and criticizing the drawbacks of crowdsourcing, [...] governments would be well-advised to embrace new social media. Citizens [...] will use new techno-logies and new channels for information-sharing anyway, whether endorsed by their governments or not. So, governments might as well engage with ICTs and crowdsourcing proactively.”

Big thanks to Max for this very valuable contribution to the discourse and to my colleague Tiago Peixoto for flagging this important study.

Evolution in Live Mapping: The 2012 Egyptian Presidential Elections

My doctoral dissertation compared the use of live mapping technology in Egypt and the Sudan during 2010. That year was the first time that Ushahidi was deployed in those two countries. So it is particularly interesting to see the technology used again in both countries in 2012. Sudanese activists are currently using the platform to map #SudanRevolts while Egyptian colleagues have just used the tool to monitor the recent elections in their country.

Analyzing the evolution of live mapping technology use in non-permissive environments ought to make for a very interesting piece of research (any takers?). In the case of Egypt, one could compare the use of the same technology and methods before and after the fall of Mubarak. In 2010, the project was called U-Shahid. This year, the initiative was branded as the “Egypt Elections Project.”

According to my colleagues in Cairo who managed the interactive map, “more than 15 trainers and 75 coordinators were trained to work in the ‘operation room’ supporting 2200 trained observers scattered all over Egypt. More than 17,000 reports, up to 25000 short messages were sent by the observers and shown on Ushahid’s interactive map. Although most reports received shown a minimum amount of serious violations, and most of them were indicating the success of the electoral process, our biggest joy was being able to monitor freely and to report the whole process with full transparency.”

Contrast this situation with how Egyptian activists struggled to keep their Ushahidi project alive under Mubarak in 2010. Last week, the team behind the current live map was actually interviewed by state television (picture above), which was formerly controlled by the old regime. Interestingly, the actual map is no longer the centerpiece of the project when compared to the U-Shahid deploy-ment. The team has included and integrated a lot more rich multimedia content in addition to data, statistics and trends analysis. Moreover, there appears to be a shift towards bounded crowdsourcing rather than open crowd-sourcing as far as election mapping projects go.

These two live mapping projects in Egypt and the Sudan are also getting relatively more traction than those in 2010. Some 17,000 reports were mapped in this year’s election project compared to 2,700 two years ago. Apparently, “millions of users logged into the [Egypt Project Elections] site to check the outcome of the electoral process,” compared to some 40,000 two years ago. Sudanese activists in Khartoum also appear to be far better organized and more agile at leverage social media channels to garner support for their movement than in 2010. Perhaps some of the hard lessons from those resistance efforts were learned.

This learning factor is key and relates to an earlier blog post I wrote on “Technology and Learning, Or Why the Wright Brothers Did Not Create the 747.” Question is: do repressive regimes learn faster or do social movements operate with more agile feedback loops? Indeed, perhaps the technology variable doesn’t matter the most. As I explained to Newsweek a while back, “It is the organiza-tional structure that will matter the most. Rigid structures are unable to adapt as quickly to a rapidly changing environment as a decentralized system. Ultimately, it is a battle of organizational theory.” In the case of Egypt and Sudan today, there’s no doubt that activists in both countries are better organized while the technologies themselves haven’t actually changed much since 2010. But better organization is a necessary, not sufficient, condition to catalyze positive social change and indirect forms of democracy.

Pierre Rosanvallon (2008) indentifies three channels whereby civil society can hold the state accountable during (and in between) elections, and independent of their results.

“The first refers to the various means whereby citizens (or, more accurately, organizations of citizens) are able to monitor and publicize the behavior of elected and appointed rulers; the second to their capacity to mobilize resistance to specific policies, either before or after they have been selected; the third to the trend toward ‘juridification’ of politics when individuals or social groups use the courts and, especially, jury trials to bring delinquent politicians to judgment.”

Live maps and crowdsourcing can be used to monitor and publicize the behavior of politicians. The capacity to mobilize resistance and bring officials to judgment may require a different set of strategies and technologies, however. Those who don’t realize this often leave behind a cemetery of dead maps.

Using Rayesna to Track the 2012 Egyptian Presidential Candidates on Twitter

My (future) colleague at the Qatar Foundation’s Computing Research Institute (QCRI) have just launched a new platform that Al Jazeera is using to track the 2012 Egyptian Presidential Candidates on Twitter. Called Rayesna, which  means “our president” in colloquial Egyptian Arabic, this fully automated platform uses cutting-edge Arabic computational linguistics processing developed by the Arabic Language Technology (ALT) group at QCRI.

“Through Rayesna, you can find out how many times a candidate is mentioned, which other candidate he is likely to appear with, and the most popular tweets for a candidate, with a special category for the most retweeted jokes about the candidates. The site also has a time-series to explore and compares the mentions of the candidate day-by-day. Caveats: 1. The site reflects only the people who choose to tweet, and this group may not be representative of general society; 2. Tweets often contain foul language and we do not perform any filtering.”

I look forward to collaborating with the ALT group and exploring how their platform might also be used in the context of humanitarian response in the Arab World and beyond. There may also be important synergies with the work of the UN Global Pulse, particularly vis-a-vis their use of Twitter for real-time analysis of vulnerable communities.