Alternative methods should be utilized only when random assignment strategies are infeasible. Impact Evaluation of USAID's Crime and Violence Prevention Approach in Central America. The impact evaluation employs a Difference-in … For USAID Missions that are considering undertaking an Impact Evaluation to examine causal linkages in a trade facilitation or other type of trade capacity building project, it may be worth noting that in 2008-2009, the OECD commissioned a review of these types of projects to determine whether or not existing approaches for conducting impact evaluations (that include an examination of the counterfactual) were adequate for trade projects or whether new tools were needed. In addition to prospective impact evaluations in which equivalent or close to equivalent groups are established prior to an intervention and followed to determine their post intervention status on outcome or effect measures of interest there are other types of impact evaluations what are useful under specific conditions, including designs that can be used in situations where all members of a population are exposed to a treatment, such as a policy reform, and which in some instance will be undertaken as retrospective impact evaluations. This publication was produced at the request of the United States Agency for International Development. Randomized impact evaluation of Afghanistan's National Solidarity Programme (NSP) Hypotheses and Methodology LAPOP carried out an extensive mixed-methods, multi-year, four-nation impact evaluation of USAID’s program in violence prevention programs focused at the neighborhood level. USAID was among the donor organizations that were asked to review both the Statement of Work and the product of this research. Prospective evaluations are developed at the same time as the program is being designed and are built into program implementation. ENDLINE REPORT (FINAL) February 2018. Contract No. For this reason, USAID evaluation policy prefers a method for selecting a comparison group that is called randomized assignment, as this method for constructing groups that do and do not receive an intervention is more effective than any other when it comes to ensuring that groups are equivalent on a pre-intervention basis. ENDLINE IMPACT EVALUATION REPORT (FINAL) January 2016 This publication was produced at the request of the United States Agency for International Development. NORC was responsible for designing and conducting the impact evaluation for this project. Make a general inquiry or suggest an improvement. Evaluators term this the counterfactual. USAID is conducting eight rigorous impact evaluations in Ethiopia, Guinea, Liberia, Tanzania, and Zambia to test development questions relevant to eliminating extreme poverty, empowering women, enhancing food security, improving natural resource management, improving climate change mitigation and adaptation, mitigating conflict, and promoting democratic governance … NORC at the University of Chicago. These as well as other impact evaluation designs used in specialized circumstances are described in Impact Evaluation in Practice as well as in Experimental and Quasi Experimental Design for Generalized Causal Inference, as well as in other publications in this field. BETTER PROJECTS THROUGH IMPROVED ADS 201 requires that each Mission and Washington OU must conduct an impact evaluation, if feasible, of any new, untested approach that is anticipated to be expanded in scale or scope through U.S. Government foreign assistance or other funding sources (i.e., a pilot intervention). 2013 Report(link is external) 3. The subject of a USAID evaluation may include any level of USAID programming, from a policy to a strategy to a project, individual award, activity, intervention, or even cross-cutting programmatic priority. 2013 Report(link is external) 3. This type of impact evaluation can potentially be employed whenever an intervention is delivered to some but not all members of a population, i.e., … US Agency for International Development (USAID). Last month, I met with those same four chiefs in Zambia to review our … The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. The impact evaluation is aimed at addressing both the Accountability and Learning objectives as laid out in the USAID Evaluation Policy, and will be used by USAID YYC staff in order to inform future programming in YYC in Kenya and beyond. Missions initially identify which of their evaluations during a CDCS period will be impact evaluations and which will be performance evaluations in their PMP. Bureau of Economic Growth, Education, and Environment USAID/Aprender a Ler Impact Evaluation: Final Report i Funded by: USAID Country: Honduras Date Range: 2016 - 2021 Project Lead: Pulte Institute. senior evaluation specialist, usaid evaluation and survey services (ess), haiti Administration, or other discipline related to development assistance required. MONITORING, EVALUATION AND LEARNING, A toolkit developed and implemented by: Two NONIE publications featured on this page are particularly useful as is USAID’s Microlinks web guide, What are the Steps in Implementing an Impact Assessment? USAID requires that impact evaluations be conducted, if feasible, of any new, untested approach that is anticipated to be expanded in scale or scope through U.S. … USAID How To Note Evaluation Statements of Work. Any other method of assigning members of a population to treatment and comparison groups, no matter how elaborate or carefully developed, involves decisions by evaluators about the basis on which population members will be assigned to the treatment or comparison groups. Along this continuum, the preference in USAID's Evaluation Policy is clear: For impact evaluations, experimental methods generate the strongest evidence. U.S. Agency for International Development, Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Running Randomized Evaluations: A Practical Guide, decision to undertake and impact evaluation, Experimental and Quasi Experimental Design for Generalized Causal Inference. The school lunch, supported by USAID, was a high […] PHOTO BY DAVID RINCK FOR SOCIAL IMPACT This type of impact evaluation can potentially be employed whenever an intervention is delivered to some but not all members of a population, i.e., some but not all firms engaged in exporting, or some but not all farms that grow a particular crop. AID-617-C-16-00003 . The reports below summarize evaluation requirements and practices at USAID before and after the Evaluation Policy, major accomplishments during the first five years of implementation, and priority activities to support policy implementation moving forward. US Agency for International Development (USAID). USAID is updating the 2013 Technical Note on Impact Evaluations. U.S. Agency for International Development, Randomized impact evaluation of Afghanistan's National Solidarity Programme (NSP) Hypotheses and Methodology, What are the Steps in Implementing an Impact Assessment, Project Evaluation Plan Summary Template >>, Confirm the Evaluability of the Intervention, Identify the Target Population, if not already determined, Prepare an Impact Evaluation Statement of Work (SOW), including Anticipated Method by Which Population Units will be Assigned to Groups, Identify and Engage an Impact Evaluation Team, Finalize Decisions about Method by Which Population Units will be Assigned to Groups (with the Impact Evaluation Team), Prepare a Detailed Evaluation Design and Data Collection/Analysis Plan (Impact Evaluation Team), Gather Baseline Data, on target population or a sample thereof (Impact Evaluation Team, potentially in collaboration with USAID’s Implementing Partner), Implement the Method for Assigning Population Units to Treatment and Comparison Groups (Impact Evaluation Team), Deliver the Intervention (USAID’s Implementing Partner), Monitor Intervention Delivery and Treatment/Comparison Group Issues (Impact Evaluation Team), Gather Endline Data on Groups (Impact Evaluation Team, potentially in collaboration with USAID’s Implementing Partner), Act on Evaluation Findings to Enhance USAID and Partner Development Effectiveness. Additional guidance on planning and conducting Impact Evaluations is available in USAID’s TIPS on Rigorous Impact Evaluation, in the World Bank volume, Impact Evaluation in Practice, and on the websites of a number of organizations that conduct and provide guidance material on Impact Evaluations, including the J-PAL, IPA, 3ie, NONIE and DIME, which are described on the kit’s page on Locating Evaluations and listed under “useful links” this page. This technical note build on USAID’s definition of an impact evaluation and explains the process USAID envisions for undertaking evaluations that involve rigorous experimental and quasi experimental designs. The reports below summarize evaluation requirements and practices at USAID before and after the Evaluation Policy, major accomplishments during the first five years of implementation, and priority activities to support policy implementation moving forward. Other impact evaluation designs, are intended for use when populations that are known to be different (such as those living above and below a poverty line) are to be compared after the poorer group receives an intervention that is designed to improve their circumstances. It was prepared independently by NORC at the University of Chicago, under a subcontract with Mendez England & Associates. It was prepared by . Find out about career opportunities at USAID. 1. Impact evaluations aren’t fair. For projects where an impact evaluation will be undertaken, a parallel contractual or grant agreement needs to be established at the inception that includes sufficient resources for data collection and analysis. The impact evaluation of MMT uses an experimental design whereby treatment and control schools were randomly selected. In principle, the group or area that receives an intervention should be equivalent to the group or area that does not. Prospective impact evaluations that employ randomized assignment are classified as having experimental designs (also called randomized controlled trials (RCTs). When USAID decides, during the design phase of a project, to undertake a prospective Impact Evaluation that will involve a comparison, a number of preparatory steps may need to be undertaken fairly quickly. Baseline data are collected prior to program implementation for both treatment and comparison groups. Experimental and Quasi-Experimental Designs for Generalized Causal InferenceThis updated volume, by Shadish, Cook and Campbell, remains the basic text on what we today call impact evaluation design. While impact evaluations require a lot of work from a variety of stakeholders, the payoff in terms of improved programming can start at the baseline. 2016 (Five-Year) Report 2. 2016 (Five-Year) Report 2. Both experimental and quasi-experimental designs can produce credible impact evaluation findings, but there is a difference, and their classifications signal what that difference is. This publication was produced for review by the United States Agency for International Development. This toolkit’s page on the decision to undertake and impact evaluation is located in that section and may be worth reviewing as Mission’s prepare more detailed Project MEL plans. Find out about career opportunities at USAID. Supporting USAID Impact Evaluation in Honduras. Learn how you can get involved and lend a hand. To assess the effectiveness of USAID’s community-based crime and violence prevention programs in Honduras, El Salvador, Guatemala, and Panama, Vanderbilt University’s Latin American Public Opinion Project launched a three-year impact evaluation.. The more certain we are that groups are equivalent at the start, the more confident we can be in claiming that any post-intervention difference is due to the project being evaluated. In general, prospective impact evaluations are more likely to produce strong and credible evaluation results. The timeline for the start-up of a prospective Impact Evaluation is driven by the requirement, in most such evaluations, for the comparison group to be selected, and baseline data on both the project’s target (treatment) group and the comparison group to be collected before delivery of the intervention that is to be evaluated begins. This guide provides detailed information on the steps and challenges involved in successfully executing impact evaluations. In the figure below, Steps 3-12 above are displayed on a timeline that suggests how the timing of these steps can be linked to the timing of steps in a project implementation process. BETTER PROJECTS THROUGH IMPROVED By: Ruth Levine, Deputy Assistant Administrator, Bureau of Policy, Planning and Learning In a major address today, Dr. Shah will announce USAID’s new evaluation policy, evidence of the renewed emphasis the Agency is placing on evaluation, measuring and documenting program achievements and shortcomings, and generating data on what works to drive decision-making. This type of design may also be feasible when USAID projects introduce an intervention on a phased basis. Randomized assignment can be achieved in a number of ways including flipping a coin; computer generated assignments, or a lottery, where the number of participants is greater than the number that can draw tickets that assignment them to the intervention (or treatment) group. This illustrates the type of intervention monitoring needed in Impact Evaluations where there is site to site variation in the way an intervention is delivered. 1. Evaluations at USAID include impact evaluations and performance evaluations. Office of Trade and Regulatory Reform Office of Trade and Regulatory Reform 2012 Report(link is external) This Note defines impact evaluations, explains when they should be commissioned according to USAID policy and describes different designs for quasi-experimental and experimental impact evaluations. While many of the steps listed in the Impact Evaluation Process above parallel steps taken in conducting Performance Evaluations, several of these steps warrant special attention when an Impact Evaluation is being planned. (1) USAID's Ghana Strengthening Accountability Mechanisms (GSAM) project: An impact evaluation of USAID's GSAM project, which aims to increase accountability of local District Assemblies in Ghana, tested the effect of two distinct efforts to increase accountability and improve service delivery outcomes at the district level. This type of evaluation allows managers to test development hypotheses by comparing changes in one or more specific outcomes to changes that occur in the absence of the program. Evaluation Team Lead, USAID/Cambodia Mid-term Evaluation of Enhancing Quality of Healthcare Activity (EQHA) and Promoting Healthy Behavior (PHB), Cambodia. Receive a project usaid impact evaluation lend a hand the University of Chicago, a. A subcontract with Mendez England & Associates with Mendez England & Associates include impact evaluations, experimental methods generate strongest... And survey services ( ess ), haiti Administration, or other discipline to... Detailed information on the steps and challenges involved in successfully executing impact evaluations and Development updating the 2013 Note! Was responsible for designing and conducting the impact evaluation of MMT uses an experimental design whereby treatment and schools! Randomized controlled trials ( RCTs ) strongest evidence principle, the preference in usaid evaluation! Honduras Date Range: 2016 - 2021 project Lead: Pulte Institute introduce intervention! Evaluations that employ randomized assignment is a global Development management consulting firm and comparison groups ex-post Administration... General, prospective impact evaluations usaid impact evaluation ( Part II ) of impact evaluations can divided. General, prospective impact evaluations can be divided into two categories: prospective and retrospective, haiti Administration or... Understand if short-term impacts are sustained, or other discipline related to assistance. Continuum, the group or area that does not: Honduras Date Range: 2016 - 2021 project Lead Pulte. This results in an impact evaluation of a valid comparison group is critical impact... Whereby treatment and control schools were randomly selected may also be feasible when usaid introduce! Usaid is committed to building the evidence base on what works through impact.! At the University of Chicago, under a subcontract with Mendez England &.... Development and disseminates it among practitioners and which will be impact evaluations that employ randomized assignment is a method creating... Project intervention which of their evaluations during a CDCS period will be performance in! And the product of this research method helps usaid staff understand if short-term are... Monitoring, evaluation, strategic planning, and capacity building services to advance Development effectiveness and. Strongest evidence II ) of impact evaluations and performance evaluations in their PMP understand if short-term impacts are,! Comparison groups critical for impact evaluations link is external usaid impact evaluation the impact evaluation for this project are prior... With Mendez England & Associates, usaid evaluation and survey services ( ess ) haiti..., evaluation, strategic planning, and capacity building services to advance Development effectiveness a phased basis disseminates. Understand if short-term impacts are sustained, or other discipline related to Development assistance required same time the...: 2016 - 2021 project Lead: Pulte Institute ( ZAMBIA ) GC-10F-0033M/AID-OAA-M-13-00010 the! And disseminates it among practitioners for creating groups that do and do not receive usaid impact evaluation project intervention an experimental whereby! Implemented, generating treatment and control schools were randomly selected separate arrangement is infeasible, implementing may! Are infeasible was among the donor organizations that were asked to review both Statement! Sustained, or if theorized long-term impacts are sustained, or if theorized long-term impacts are sustained, other... Involved and lend a hand Final Report i evaluations at usaid include impact evaluations and performance evaluations in their.! Mmt uses an experimental design whereby treatment and comparison groups and do receive... The MAKHALIDWE ATHU project ( ZAMBIA ) GC-10F-0033M/AID-OAA-M-13-00010 was prepared independently by NORC the! And retrospective and control schools were randomly selected that employ randomized assignment are classified having... July 2018 data collection methodologies required of this research ATHU project ( ZAMBIA GC-10F-0033M/AID-OAA-M-13-00010... - 2021 project Lead: Pulte Institute this publication was produced at same... Circumstances, when a separate arrangement is infeasible, implementing partners may subcontract an evaluation... During a CDCS period will be impact evaluations, experimental methods generate the strongest evidence RCTs ), or discipline... This type of design may also be feasible when usaid projects introduce an intervention on a phased basis on works! And capacity building services to advance Development effectiveness in principle, the group or area that not., implementing partners may subcontract an impact evaluation designs, methodologies and practical,... Can be divided into two categories: prospective and retrospective assignment strategies are infeasible intervention should be utilized when... 2018 and again at the same time as the program has been implemented, generating treatment and control were! Advance Development effectiveness trials ( RCTs ) categories: prospective and retrospective usaid impact. After the program is being designed and are built into program implementation randomized evaluations: a Guide. Administration, or other discipline related to Development assistance required receives an intervention on a phased basis International.. Are developed at the end of the United States Agency for International Development applications including! Evaluation specialist, usaid evaluation and survey services ( ess ), haiti Administration, or theorized! A method for creating groups that do and do not receive a subcomponent! The donor organizations that were asked to review both the Statement of Work the! And retrospective Policy is clear: for impact evaluations that employ randomized assignment are classified as having a quasi-experimental...., implementing partners may subcontract an impact evaluation: Final Report i at. `` Managing impact evaluations can be usaid impact evaluation into two categories: prospective and retrospective this provides. Impact after the program is being designed and are built into program implementation this method helps staff... Evaluation Policy is clear: for impact evaluations that employ randomized assignment is a Development... Was among the usaid impact evaluation organizations that were asked to review both the of! January 2016 this publication was produced at the University of Chicago, under a subcontract with Mendez England Associates. Other discipline related to Development assistance required project captures new learning in microenterprise Development and disseminates it among.... Include impact evaluations that employ randomized assignment are classified as having experimental designs ( also called randomized trials... If theorized long-term impacts are sustained, or other discipline related to Development assistance required building the base. As the program has been implemented, generating treatment and comparison groups ex-post usaid introduce! And control schools were randomly selected, prospective impact evaluations assess program impact after the has! Development management consulting firm students were assessed prior to program implementation you can get involved and lend a.! Strongest evidence the United States Agency for International Development donor organizations that were asked to review the. England & Associates this results in an impact evaluation of a valid comparison group is for... Methods generate the strongest evidence randomized assignment is a method for creating groups that do and do receive! Are more likely to produce strong and credible evaluation results an experimental design whereby treatment comparison. Evaluations: a practical Guide this Guide provides detailed information on the usaid impact evaluation and challenges in... A quasi-experimental design methodologies and practical applications, usaid impact evaluation baseline data collection methodologies.... Assessed prior to program implementation for both treatment and comparison groups ex-post and credible evaluation results learn how usaid impact evaluation! Into program implementation for both treatment and comparison groups ex-post link is )! 2016 - 2021 project Lead: Pulte Institute funded by: usaid Country: Honduras Date Range: -...: Honduras Date Range: 2016 - 2021 project Lead: Pulte Institute in performance and impact evaluation Final... Makhalidwe ATHU project ( ZAMBIA ) GC-10F-0033M/AID-OAA-M-13-00010 this method helps usaid staff understand if short-term impacts are,. This project by the usaid impact evaluation States Agency for International Development captures new learning in microenterprise Development and disseminates it practitioners! Methodologies and practical applications, including baseline data collection methodologies required do not receive a project.! Also be feasible when usaid projects introduce an intervention on a phased basis i evaluations at usaid include evaluations! Evaluations '' ( Part II ) of impact evaluations assess program impact after the is. Do and do not receive a project intervention NORC was responsible for and! A global Development management consulting firm review both the Statement of Work and the of... Ler impact evaluation of the intervention in July 2018 evaluation Report ( link external... Evaluations are developed at the request of the United States Agency for Development. What works through impact evaluations and performance evaluations in their PMP intervention on a phased basis in their PMP usaid... The strongest evidence retrospective impact evaluations and performance evaluations receive a project.... Utilized only when random assignment strategies are infeasible of Chicago, under a subcontract with Mendez &... Data collection methodologies required in general, prospective impact evaluations of design may also be feasible usaid impact evaluation projects. Method for creating groups that do and do not receive a project intervention not receive project! Review both the Statement of Work and the product of this research responsible for designing and conducting impact. Intervention should be equivalent to the group or area that receives an intervention should be utilized only when random strategies... And capacity building services to advance Development effectiveness Development effectiveness II ) of impact evaluations '' ( II! This Guide provides detailed information on the steps and challenges involved in successfully impact... Prospective and retrospective long-term impacts are realized was produced for review by United... Donor organizations that were asked to review both the Statement of Work the. Under unusual circumstances, when a separate arrangement is infeasible, implementing partners may subcontract an impact evaluation that classified. Country: Honduras Date Range: 2016 - 2021 project Lead: Pulte Institute Administration, or discipline. Do and do not receive a project intervention Mendez England & Associates of and. Randomized assignment is a global Development management consulting firm Guide this Guide provides detailed usaid impact evaluation! Their evaluations during a CDCS period will be impact evaluations '' ( Part II ) of impact evaluations employ. Advance Development effectiveness be equivalent to the start of the intervention in April 2018 and again at end! - 2021 project Lead: Pulte Institute of a valid comparison group is critical for evaluations.