Tag Archives: Evaluation

Results: Evaluation of UAVs for Humanitarian Use

UAViators Logo

My team & I at the Humanitarian UAV Network (UAViators) have just completed the first phase of our evaluation and welcome feedback on the results. We have reviewed over 150 UAV models along with camera technologies, payload units as well as image processing and analysis software. Each of these items have been reviewed within the context of humanitarian applications and with humanitarian practitioners in mind as end-users.

The results of the evaluation are available here in the form of an open and editable Google spreadsheet. We are actively looking for feedback and very much welcome additional entries. So feel free to review & add more UAVs and related technologies directly to the spreadsheet. Our second phase will involve the scoring/weighing of the results to identity the UAVs, cameras and software that may be the best fit for humanitarian organizations.

In the meantime, big thanks to my research assistants who carried out all the research for this review.

bio

See Also:

  • Humanitarian UAV Network: Strategy for 2014-2015 [link]
  • Humanitarians in the Sky: Using UAVs for Disaster Response [link]
  • Humanitarian UAV Missions During Balkan Floods [link]
  • UAVs, Community Mapping & Disaster Risk Reduction in Haiti [link]
  • Crisis Map of UAV Videos for Disaster Response [link]
  • Check-List for Flying UAVs in Humanitarian Settings [link]

 

Comprehensive List of UAVs for Humanitarian Response

The Humanitarian UAV Network (UAViators) is carrying out a comprehensive evaluation of UAVs and related technologies for use in humanitarian settings. We’ve developed an evaluation framework for this assessment and have now drafted this list of UAV models to determine which are worth evaluating. As you’ll note, the link also points to three other related lists: cameras, sensors and software for image processing and analysis. We’ll be evaluating these as well to identify which are the best fit for use by humanitarians in the field.

xUAV3

We are actively seeking feedback on these preliminary lists in order to identify which items we should prioritize for evaluation. The lists are available in this open and editable Google Spreadsheet. Each list include a column for you to add any comments you might have about any entries. We’re particularly interested in getting feedback on which items are not worth evaluating and which should be added at to the list. Thank you!

Bio

See Also:

  • Crisis Map of UAV/Aerial Videos for Disaster Response [link]
  • How UAVs are Making a Difference in Disaster Response [link]
  • Humanitarians Using UAVs for Post Disaster Recovery [link]
  • Grassroots UAVs for Disaster Response [link]
  • Using UAVs for Search & Rescue [link]
  • Debrief: UAV/Drone Search & Rescue Challenge [link]
  • Crowdsourcing Analysis of UAV Imagery for Search/Rescue [link]
  • Check-List for Flying UAVs in Humanitarian Settings [link]

Evaluating UAVs for Humanitarian Response

The Humanitarian UAV Network is carrying out an evaluation of UAVs and related technologies for use in humanitarian settings. The related technologies being evaluated include cameras, payload units, image processing & analysis software. As a first step, we have created an evaluation framework based on parameters relevant for field-based deployments of UAVs by humanitarian organizations. Before moving to the next step in the evaluation—identifying which UAVs and related technologies to evaluate—we want to make sure that we’re on the right track as far as our evaluation framework goes. So the purpose of this blog post is to seek feedback on said framework.

UAViators Logo

To recap, we are evaluating four distinct technologies: 1) UAVs; 2) Cameras; 3) Payload units; and 4) Image Processing & Analysis Software specifically for humanitarian use. So below are the evaluation criteria we have identified for each technology.

UAVs: Type, Cost, Size, Weight, Appearance, Noise Factor, Durability, Ease of Use, Ease of Repair, Payload Capacity, Flight Time, Transmitter Range,  Autonomy, Camera/Gimbal Compatibility and Legality/Customs.

Most of the parameters are self-explanatory but a few require some elaboration. Type refers to whether the UAV is a fixed-wing, rotary-wing or a hybrid. Appearance seeks to evaluate whether the UAV airframe looks threatening or more like a toy, for example. Autonomy refers to whether the UAV can be flown autonomously. Legality/Customs seeks to assess whether the transportation of the UAV across borders is likely to be easy or challenging.

Cameras: Cost, Size, Weight, Megapixels, Memory, Control, Durability, Easy of Use, Ease of Repair, Lens Type and Gimbal Compatibility.

Payload Units: Cost, Size, Weight, Type of Release Mechanism, Release Mechanism, Ease of Use and Ease of Repair.

Image Software: Cost, Image Processing, Image Analysis, Ease of Use, System Requirements, Compatibility and Type of License.

We need to make sure that we fill any gaps in our evaluation criteria before proceeding with the assessment. So what parameters are we missing?

bio

See Also:

  • Welcome to the Humanitarian UAV Network [link]
  • How UAVs are Making a Difference in Disaster Response [link]
  • Humanitarians Using UAVs for Post Disaster Recovery [link]
  • Grassroots UAVs for Disaster Response [link]
  • Using UAVs for Search & Rescue [link]
  • Debrief: UAV/Drone Search & Rescue Challenge [link]
  • Crowdsourcing Analysis of UAV Imagery for Search/Rescue [link]
  • Check-List for Flying UAVs in Humanitarian Settings [link]

 

Humanitarian UAV Network: Strategy for 2014-2015

The purpose of the Humanitarian UAV Network (UAViators) is to guide the responsible and safe use of small UAVs in humanitarian settings while promoting information sharing and enlightened policymaking. As I’ve noted in the past, UAVs are already being used to support a range of humanitarian efforts. So the question is not if, but rather how to facilitate the inevitable expanded use of UAVs in a responsible and safe manner. This is just one of many challenging questions that UAViators was created to manage.

UAViators Logo

UAViators has already drafted a number of documents, including a Code of Conduct and an Operational Check-List for the use of UAVs in humanitarian settings. These documents will continue to be improved throughout the year, so don’t expect a final and perfect version tomorrow. This space is still too new to have all the answers in a first draft. So our community will aim to improve these documents over time. By the end of 2014, we hope to have a solid version of the of Code of Conduct for organizations and companies to publicly endorse.

OCHA UAV

In the meantime, my three Research Assistants (RA’s) and I are working on (the first ever?) comprehensive evaluation of 1) Small UAVs; 2) Cameras; 3) Payload Units; and 4) Imagery Software specifically for humanitarian field-workers. The purpose of this evaluation is to rate which technologies are best suited to the needs of humanitarians in the field. We will carry out this research through interviews with seasoned UAV experts coupled with secondary, online research. Our aim is to recommend 2-3 small UAVs, cameras, payload units and software solutions for imagery processing and analysis that make the most sense for humanitarians as end users. These suggestions will then be “peer reviewed” by members of the Humanitarian UAV Network.

Following this evaluation, my three RA’s and I will create a detailed end-to-end operational model for the use of UAVs in humanitarian settings. The model will include pre-flight guidance on several key issues including legislation, insurance, safety and coordination. The pre-flight section will also include guidance on how to program the flight-path of the UAVs recommended in the evaluation. But the model does not end with the safe landing of a UAV. The operational model will include post-flight guidance on imagery processing and analysis for decision support as well as guidelines on information sharing with local communities. Once completed, this operational model will also be “peer reviewed” by members of the UAViators.

Credit Drone Adventures

Both deliverables—the evaluation and model—will be further reviewed by the Advisory Board of UAViators and by field-based humanitarians. We hope to have this review completed during the Humanitarian UAV Experts Meeting, which I am co-organizing with OCHA in New York this November. Feedback from this session will be integrated into both deliverables.

Our plan is to subsequently convert these documents into training materials for both online and onsite training. We have thus far identified two sites for this training, one in Southeast Asia and the other in southern Africa. We’re considering a potential third site in South America depending on the availability of funding. These trainings will enable us to further improve our materials and to provide minimum level certification to humanitarians participating in said trainings. To this end, our long-term strategy for the Humanitarian UAV Network is not only to facilitate the coordination of small UAVs in humanitarian settings but also to provide both training and certification in collaboration with multiple humanitarian organizations.

I recognize that the above is highly ambitious. But all the signals I’m getting from humanitarian organizations clearly demonstrate that the above is needed. So if you have some expertise in this space and wish to join my Research Assistants and I in this applied and policy-focused research, then please do get in touch. In addition, if your organization or company is interested in funding any of the above, then do get in touch as well. We have the initial funding for the first phase of the 2014-2015 strategy and are in the process of applying for funding to complete the second phase.

One final but important point: while the use of many small and large UAVs in complex airspaces in which piloted (manned) aircraft are also flying poses a major challenge in terms of safety, collision avoidance and coordination, this obviously doesn’t mean that small UAVs should be grounded in humanitarian contexts with far simpler airspaces. Indeed, to argue that small UAVs cannot be responsibly and safely operated in simpler airspaces ignores the obvious fact that they already have—and continue to be responsibly & safely used. Moreover, I for one don’t see the point of flying small UAVs in areas already covered by larger UAVs and piloted aircraft. I’m far more interested in the rapid and local deployment of small UAVs to cover areas that are overlooked or have not yet been reached by mainstream response efforts. In sum, while it will take years to develop effective solutions for large UAV-use in dense and complex airspaces, small UAVs are already being used responsibly and safely by a number of humanitarian organizations and their partners.

Bio

See Also:

  • Welcome to the Humanitarian UAV Network [link]
  • How UAVs are Making a Difference in Disaster Response [link]
  • Humanitarians Using UAVs for Post Disaster Recovery [link]
  • Grassroots UAVs for Disaster Response [link]
  • Using UAVs for Search & Rescue [link]
  • Debrief: UAV/Drone Search & Rescue Challenge [link]
  • Crowdsourcing Analysis of UAV Imagery for Search/Rescue [link]
  • Check-List for Flying UAVs in Humanitarian Settings [link]

Could Lonely Planet Render World Bank Projects More Transparent?

That was the unexpected question that my World Bank colleague Johannes Kiess asked me the other day. I was immediately intrigued. So I did some preliminary research and offered to write up a blog post on the idea to solicit some early feedback. According to recent statistics, international tourist arrivals numbered over 1 billion in 2012 alone. Of this population, the demographic that Johannes is interested in comprises those intrepid and socially-conscious backpackers who travel beyond the capitals of developing countries. Perhaps the time is ripe for a new form of tourism: Tourism for Social Good.

tourism_socialmedia

There may be a real opportunity to engage a large crowd because travelers—and in particular the backpacker type—are smartphone savvy, have time on their hands, want to do something meaningful, are eager to get off the beaten track and explore new spaces where others do not typically trek. Johannes believes this approach could be used to map critical social infrastructure and/or to monitor development projects. Consider a simple smartphone app, perhaps integrated with existing travel guide apps or Tripadvisor. The app would ask travelers to record the quality of the roads they take (with the GPS of their smartphone) and provide feedback on the condition, e.g.,  bumpy, even, etc., every 50 miles or so.

They could be asked to find the nearest hospital and take a geotagged picture—a scavenger hunt for development (as Johannes calls it); Geocaching for Good? Note that governments often do not know exactly where schools, hospitals and roads are located. The app could automatically alert travelers of a nearby development project or road financed by the World Bank or other international donor. Travelers could be prompted to take (automatically geo-tagged) pictures that would then be forwarded to development organizations for subsequent visual analysis (which could easily be carried out using microtasking). Perhaps a very simple, 30-second, multiple-choice survey could even be presented to travelers who pass by certain donor-funded development projects. For quality control purposes, these pictures and surveys could easily be triangulated. Simple gamification features could also be added to the app; travelers could gain points for social good tourism—collect 100 points and get your next Lonely Planet guide for free? Perhaps if you’re the first person to record a road within the app, then it could be named after you (of course with a notation of the official name). Even Photosynth could be used to create panoramas of visual evidence.

The obvious advantage of using travelers against the now en vogue stakeholder monitoring approach is that they said bagpackers are already traveling there anyway and have their phones on them to begin with. Plus, they’d be independent third parties and would not need to be trained. This obviously doesn’t mean that the stakeholder approach is not useful. The travelers strategy would simply be complementary. Furthermore, this tourism strategy comes with several key challenges, such as the safety of backpackers who choose to take on this task, for example. But appropriate legal disclaimers could be put in place, so this challenge seems surmountable. In any event, Johannes, together with his colleagues at the World Bank (and I), hope to explore this idea of Tourism for Social Good further in the coming months.

In the meantime, we would be very grateful for feedback. What might we be overlooking? Would you use such an app if it were available? Where can we find reliable statistics on top backpacker destinations and flows?

Bio

See also: 

  • What United Airlines can Teach the World Bank about Mobile Accountability [Link]

Why Ushahidi Should Embrace Open Data

“This is the report that Ushahidi did not want you to see.” Or so the rumors in certain circles would have it. Some go as far as suggesting that Ushahidi tried to burry or delay the publication. On the other hand, some rumors claim that the report was a conspiracy to malign and discredit Ushahidi. Either way, what is clear is this: Ushahidi is an NGO that prides itself in promoting transparency & accountability; an organization prepared to take risks—and yes fail—in the pursuit of this  mission.

The report in question is CrowdGlobe: Mapping the Maps. A Meta-level Analysis of Ushahidi & Crowdmap. Astute observers will discover that I am indeed one of the co-authors. Published by Internews in collaboration with George Washington University, the report (PDF) reveals that 93% of 12,000+ Crowdmaps analyzed had fewer than 10 reports while a full 61% of Crowdmaps had no reports at all. The rest of the findings are depicted in the infographic below (click to enlarge) and eloquently summarized in the above 5-minute presentation delivered at the 2012 Crisis Mappers Conference (ICCM 2012).

Infographic_2_final (2)

Back in 2011, when my colleague Rob Baker (now with Ushahidi) generated the preliminary results of the quantitative analysis that underpins much of the report, we were thrilled to finally have a baseline against which to measure and guide the future progress of Ushahidi & Crowdmap. But when these findings were first publicly shared (August 2012), they were dismissed by critics who argued that the underlying data was obsolete. Indeed, much of the data we used in the analysis dates back to 2010 and 2011. Far from being obsolete, however, this data provides a baseline from which the use of the platform can be measured over time. We are now in 2013 and there are apparently 36,000+ Crowdmaps today rather than just 12,000+.

To this end, and as a member of Ushahidi’s Advisory Board, I have recommended that my Ushahidi colleagues run the same analysis on the most recent Crowdmap data in order to demonstrate the progress made vis-a-vis the now-outdated public baseline. (This analysis takes no more than an hour a few days to carry out). I also strongly recommend that all this anonymized meta-data be made public on a live dashboard in the spirit of open data and transparency. Ushahidi, after all, is a public NGO funded by some of the biggest proponents of open data and transparency in the world.

Embracing open data is one of the best ways for Ushahidi to dispel the harmful rumors and conspiracy theories that continue to swirl as a result of the Crowd-Globe report. So I hope that my friends at Ushahidi will share their updated analysis and live dashboard in the coming weeks. If they do, then their bold support of this report and commitment to open data will serve as a model for other organizations to emulate. If they’ve just recently resolved to make this a priority, then even better.

In the meantime, I look forward to collaborating with the entire Ushahidi team on making the upcoming Kenyan elections the most transparent to date. As referenced in this blog post, the Standby Volunteer Task Force (SBTF) is partnering with the good people at PyBossa to customize an awesome micro-tasking platform that will significantly facilitate and accelerate the categorization and geo-location of reports submitted to the Ushahidi platform. So I’m working hard with both of these outstanding teams to make this the most successful, large-scale microtasking effort for election monitoring yet. Now lets hope for everyone’s sake that the elections remain peaceful. Onwards!

Innovation and the State of the Humanitarian System

Published by ALNAP, the 2012 State of the Humanitarian System report is an important evaluation of the humanitarian community’s efforts over the past two years. “I commend this report to all those responsible for planning and delivering life saving aid around the world,” writes UN Under-Secretary General Valerie Amos in the Preface. “If we are going to improve international humanitarian response we all need to pay attention to the areas of action highlighted in the report.” Below are some of the highlighted areas from the 100+ page evaluation that are ripe for innovative interventions.

Accessing Those in Need

Operational access to populations in need has not improved. Access problems continue and are primarily political or security-related rather than logistical. Indeed, “UN security restrictions often place sever limits on the range of UN-led assessments,” which means that “coverage often can be compromised.” This means that “access constraints in some contexts continue to inhibit an accurate assessment of need. Up to 60% of South Sudan is inaccessible for parts of the year. As a result, critical data, including mortality and morbidity, remain unavailable. Data on nutrition, for example, exist in only 25 of 79 countries where humanitarian partners have conducted surveys.”

Could satellite and/or areal imagery be used to measure indirect proxies? This would certainly be rather imperfect but perhaps better than nothing? Could crowdseeding be used?

Information and Communication Technologies

“The use of mobile devices and networks is becoming increasingly important, both to deliver cash and for communication with aid recipients.” Some humanitarian organizations are also “experimenting with different types of communication tools, for different uses and in different contexts. Examples include: offering emergency information, collecting information for needs assessments or for monitoring and evaluation, surveying individuals, or obtaining information on remote populations from an appointed individual at the community level.”

“Across a variety of interventions, mobile phone technology is seen as having great potential to increase efficiency. For example, […] the governments of Japan and Thailand used SMS and Twitter to spread messages about the disaster response.” Naturally, in some contexts, “traditional means like radios and call centers are most appropriate.”

In any case, “thanks to new technologies and initiatives to advance commu-nications with affected populations, the voices of aid recipients began, in a small way, to be heard.” Obviously, heard and understood are not the same thing–not to mention heard, understood and responded to. Moreover, as disaster affected communities become increasingly “digital” thanks to the spread of mobile phones, the number of voices will increase significantly. The humanitarian system is largely (if not completely) unprepared to handle this increase in volume (Big Data).

Consulting Local Recipients

Humanitarian organizations have “failed to consult with recipients […] or to use their input in programming.” Indeed, disaster-affected communities are “rarely given opportunities to assess the impact of interventions and to comment on performance.” In fact, “they are rarely treated as end-users of the service.” Aid recipients also report that “the aid they received did not address their ‘most important needs at the time.'” While some field-level accountability mechanisms do exist, they were typically duplicative and very project oriented. To this end, “it might be more efficient and effective to have more coordination between agencies regarding accountability approaches.”

While the ALNAP report suggests that these shortcomings could “be addressed in the near future by technical advances in methods of needs assessment,” the challenge here is not simply a technical one. Still, there are important efforts underway to address these issues.

Improving Needs Assessments

The Inter-Agency Standing Committee’s (IASC) Needs Assessment Task Force (NAFT) and the International NGO-led Assessment Capacities Project (ACAPS) are two such exempts of progress. OCHA serves as the secretariat for the NAFT through its Assessment and Classification of Emergencies (ACE) Team. ACAPS, which is a consortium of three international NGOs (X, Y and Z) and a member of NATF, aims to “strengthen the capacity of the humanitarian sector in multi-sectoral needs assessment.” ACAPS is considered to have “brought sound technical processes and practical guidelines to common needs assessment.” Note that both ACAPS and ACE have recently reached out to the Digital Humanitarian Network (DHNetwork) to partner on needs-assessment projects in South Sudan and the DRC.

Another promising project is the Humanitarian Emergency Settings Perceived Needs Scale (HESPER). This join initiative between WHO and King’s College London is designed to rapidly assess the “perceived needs of affected populations and allow their views to be taken into consideration. The project specifically aims to fill the gap between population-based ‘objective’ indicators […] and/or qualitative data based on convenience samples such as focus groups or key informant interviews.” On this note, some NGOs argue that “overall assessment methodologies should focus far more at the community (not individual) level, including an assessment of local capacities […],” since “far too often international aid actors assume there is no local capacity.”

Early Warning and Response

An evaluation of the response in the Horn of Africa found “significant disconnects between early warning systems and response, and between technical assessments and decision-makers.” According to ALNAP, “most commentators agree that the early warning worked, but there was a failure to act on it.” This disconnect is a concern I voiced back in 2009 when UN Global Pulse was first launched. To be sure, real-time information does not turn an organization into a real-time organization. Not surprisingly, most of the aid recipients surveyed for the ALNAP report felt that “the foremost way in which humanitarian organizations could improve would be to: ‘be faster to start delivering aid.'” Interestingly, “this stands in contrast to the survey responses of international aid practitioners who gave fairly high marks to themselves for timeliness […].”

Rapid and Skilled Humanitarians

While the humanitarian system’s surge capacity for the deployment of humanitarian personnel has improved, “findings also suggest that the adequate scale-up of appropriately skilled […] staff is still perceived as problematic for both operations and coordination.” Other evaluations “consistently show that staff in NGOs, UN agencies and clusters were perceived to be ill prepared in terms of basic language and context training in a significant number of contexts.” In addition, failures in knowledge and understanding of humanitarian principles were also raised. Furthermore, evaluations of mega-disasters “predictably note influxes or relatively new staff with limited experience.” Several evaluations noted that the lack of “contextual knowledge caused a net decrease in impact.” This lend one senior manager noted:

“If you don’t understand the political, ethnic, tribal contexts it is difficult to be effective… If I had my way I’d first recruit 20 anthropologists and political scientists to help us work out what’s going on in these settings.”

Monitoring and Evaluation

ALNAP found that monitoring and evaluation continues to be a significant shortcoming in the humanitarian system. “Evaluations have made mixed progress, but affected states are still notably absent from evaluating their own response or participating in joint evaluations with counterparts.” Moreover, while there have been important efforts by CDAC and others to “improve accountability to, and communication with, aid recipients,” there is “less evidence to suggest that this new resource of ground-level information is being used strategically to improve humanitarian interventions.” To this end, “relatively few evaluations focus on the views of aid recipients […].” In one case, “although a system was in place with results-based indicators, there was neither the time nor resources to analyze or use the data.”

The most common reasons cited for failing to meet community expectations include the “inability to meet the full spectrum of need, weak understanding of local context, inability to understand the changing nature of need, inadequate information-gathering techniques or an inflexible response approach.” In addition, preconceived notions of vulnerability have “led to inappropriate interventions.” A major study carried out by Tufts University and cited in the ALNAP report concludes that “humanitarian assistance remains driven by ‘anecdote rather than evidence’ […].” One important exception to this is the Danish Refugee Council’s work in Somalia.

Leadership, Risk and Principles

ALNAP identifies an “alarming evidence of a growing tendency towards risk aversion” and a “stifling culture of compliance.” In addition, adherence to humanitarian principles were found to have weakened as “many humanitarian organizations have willingly compromised a principled approach in their own conduct through close alignment with political and military activities and actors.” Moreover, “responses in highly politicized contexts are viewed as particularly problematic for the retention of humanitarian principles.” Humanitarian professionals who were interviewed by ALNAP for this report “highlighted multiple occasions when agencies failed to maintain an impartial response when under pressure from strong states, such as Pakistan and Sri Lanka.”