Full project name: Impact Assessment of Research in the CPWF: A Basin Focal Project
The project carried out ex-ante impact assessment of 29 of the 32 CPWF First Call Projects through constructing impact pathways for each project. We developed Participatory Impact Pathways Analysis (PIPA) which will be used as the basis for program evaluation in Phase II. Research carried out with the University of Oxford based on network data gathered in impact pathways workshops shows that the CPWF has been successful in its founding goal of better linking the food and water sectors. Extrapolation domain analysis has shown demonstrated the potential reach of several CPWF project technologies, including aerobic rice and the Quesungual alternative to slash-and-burn.
- Participatory Impact Pathways Analysis
The project developed the Participatory Impact Pathways Analysis (PIPA) methodology, which is a practical planning, and monitoring and evaluation approach developed for use with complex projects in the water and food sectors. PIPA begins with a participatory workshop where stakeholders make explicit their assumptions about how their project will make an impact. Participants construct problem trees, carry out a visioning exercise and draw network maps to help them clarify their ‘impact pathways’. These are then articulated in two logic models. The outcomes logic model describes the project’s medium term objectives in the form of hypotheses: which actors need to change, what are those changes and which strategies are needed to realise these changes. The impact logic model describes how, by helping to achieve the expected outcomes, the project will impact on people’s livelihoods. Participants derive outcome targets and milestones which are regularly revisited and revised as part of project monitoring and evaluation (M&E). PIPA goes beyond the traditional use of logic models and log-frames by engaging stakeholders in a structured participatory process, promoting learning and providing a framework for ‘action research’ on processes of change. The two logic models provide predictions of future impact which can be used in priority setting. They also provide impact hypotheses required for ex-post impact assessment (Douthwaite et al. 2007). The specification of impact pathways, using PIPA or Outcome Mapping, is now recommended good practice in the CGIAR for monitoring and evaluation and as precursor to ex-post impact assessment (Walker et al., 2008).
- Impact Assessment of CPWF networking
Increasingly, development projects list social capital development and network brokerage among their objectives. How do we quantitatively evaluate such initiatives? Best practice, difference-in-difference methods, which require a control group may be impossible or too costly. Network data gathered in CPWF impact pathways workshops has been analyzed in collaboration with the University of Oxford. We developed and applied a methodology for evaluating CPWF’s success in brokering and strengthening linkages between different types of organization.
We used data that are byproducts of the network broking process to evaluate the Challenge Program for Water and Food along this dimension. We find that, in accordance with its objectives, the program is associated with bridging between organizations in the water and food sectors and between CGIAR members and their counterparts in government and that, in the case of the former, the association may be causal (Barr et al., forthcoming).
- Extrapolation Domain Analysis
One of the CPWF’s goals is to extend and accelerate impact. A key research question is therefore: how is it possible to accelerate impact?
The question of ‘how’ is not easy to define. Simply taking localized success stories and scaling up and out is a misguided approach, because not all places have the same characteristics at the start. To identify accurately other areas in the world where an activity or a project can be reproduced requires analysis of huge amounts of data to determine whether the conditions are favorable to the introduction of the new approach. It also requires understanding of how lessons and knowledge acquired previously can be modified to fit the new geographical context to which it will be introduced. The CPWF sought to use geographical information approaches, which have been well established in the literature, and developed the concept of “Extrapolation Domain Analysis” (EDA). EDA is a means to identify areas where new methods of ecosystem management might be introduced with a high probability of success.
EDA combines a number of techniques of spatial analysis. It was first investigated in 2006, when it was applied to assess how similarity analysis could be used to scale out research findings within seven Andean basins. The method was developed further by incorporating socio-economic variables into the Homologue analysis used to identify sites elsewhere in the tropics that are similar to a site with known characteristics. It has since been used to evaluate impact pathways and in global impact analysis.
The project developed a step-by-step guide to EDA to help decision makers and project implementers identify potential areas in which new methods and technologies might be applied with confidence, so that investments may be more accurately targeted, and thereby ensure better success rates.
In an effort to improve the sensitivity of EDA to social as well as agro-ecological determinants of adoption, a method was developed to provide better targeting of interventions that require participation and strong institutional settings. We developed a calculation of individuals’ willingness to participate (IWP) and an institutional environment index (IEI). The method was developed using secondary data from Bolivian municipalities. The method can be easily applied to countries with relatively good socio-economic secondary data at municipal level to produce continental or sub-continental pictures of these two key factors. (Rubiano and Garcia, 2009).
Rubiano, J., and J. Garcia. 2009. Improving knowledge for targeting interventions: willingness of individuals to participate and calculation of institutional environment indices. CPWF Working Paper No. 5. Colombo, Sri Lanka: CGIAR Challenge Program on Water and Food.
Rubiano, J., and V. Soto. 2009. Geographical extrapolation domain analysis: scaling up watershed management research projects, a toolkit to guide implementation. CPWF Working Paper No. 4. Colombo, Sri Lanka: CGIAR Challenge Program on Water and Food.
- Project and basin impact narratives (continuation of Phase I activity)
- Journal article on the methodology for carrying out extrapolation domain analysis and scenario analysis
- Two journal articles based on research results (theories of action and network models)
- How-to guide to carrying out extrapolation domain analysis
- How-to guide on the impact pathways integrated project management approach
- Journal article on effective pathways and networks for impact (product from Objective 4)
- Book chapter on the use of IPs for priority setting
- Book chapter on evaluation IPs as a practical method for monitoring and evaluation (the re-thinking impact compilation)
For more information on Phase 1 outputs please contact Udana Ariyawansa.