Ex post evaluations are a key element of the AF-TERG FY21-FY23 strategy and work programme, and beyond, originating from the request of the Adaptation Fund Board to develop post-implementation learning for Fund projects and programmes and provide accountability of results financed by the Fund.
They intend to evaluate aspects of both sustainability of outcomes and climate resilience, and over time feed into ex-post-evaluation-informed adjustments within the Fund’s Monitoring Evaluation and Learning (MEL) processes.
The framework developed by the AF-TERG is an innovative and Fund-specific framework to conduct ex post evaluations. Given the relative novelty of climate change adaptation portfolios and the limited body of work on ex post evaluation for adaptation, it presents possible methods that have been piloted and field-tested in ex post evaluations in Samoa and Ecuador during fiscal year 2022 (FY22).
Ex post evaluation summary – Ecuador (2022)
National evaluator Monica Ribadeneira Sarmiento began the Ecuador ex post evaluation in November 2021. Over ten months, it was carried out in different stages: review of project documentation; capacity-building training; selection of outcomes to evaluate ex post; field visit and data collection; data analysis; and report write-up.
The fieldwork showed that sustainability of outcomes after project completion was only moderate. This result was driven by partial ownership of assets depending on the site, few resources for partnerships, and a limited impact on “decreased vulnerability.” Results differed significantly between sites. Assets in Cochapata were relevant and well maintained by the community, but those in Nabón were partially unusable or abandoned.
The resilience analysis tool indicates weaker prospects for climate resilience. The resilience characteristics exhibited by the assets did not seem to influence food security in both sites or were not enough to prevent outmigration from one site. The overall strategy of the outcomes was to maintain systems (and their structures and functions) for water management and food security. Thus, the resilience of investments can be classified as active and passive resistance.
Ex post evaluation summary Samoa (2022)
National evaluator Karen Komiti began the ex post evaluation of the Samoa project “Enhancing Resilience of Samoa’s Coastal Communities to Climate Change”, at the end of October 2021. It covered different stages over five months: review of project documentation; selection of outcomes to evaluate ex post; field visit and data collection; data analysis; and report write-up.
Five years after construction, the structures across four sites/six villages remain physically intact but some sections of Manase and Salimu / Musumusu rockwalls appear to be deteriorating. In general, the structures are adequately and routinely maintained by stakeholders at household, village, and government levels. These activities have not diminished in the years since project closure, despite the absence of secure funding in government operational budget, an infrastructure-specific risk management plan, and co-financing to enable maintenance beyond closure.
As far as the resistance-resilience-transformation typology, all seven structures exhibit various levels of resilience (mainly passive resistance and some resilience) . All have remained standing in the face of several climate disturbance impacts in the intervening years.
Information update on phase 2 – October 2022
An information update on phase 2 of the ex post project sustainability evaluations (AFB/EFC.30/Inf.3) was presented to the thirtieth Ethics and Finance Committee (EFC) meeting in October 2022 and outlines the findings of the first two ex post evaluation pilots, after completion of fieldwork, and highlights lessons for the Fund and its implementers. The document also describes the next steps for AF-TERG ex post evaluations based on lessons learned from the first two pilots.
The AF-TERG will take steps to move from the pilot phase of ex post evaluation to a standard set of procedures for selecting and implementing ex post evaluations, including protocols for engaging with in-country partners and for communicating results. The AFTERG will also continue to refine the methodology for ex post evaluations. Furthermore, the AF-TERG will discuss potential approaches for evaluations of fragile states, while ensuring the safety of all participants. Fragile states comprise a significant share of the current eligible pool of completed projects.
Progress Update to the Board – March 2022
A progress update on ex post evaluations and emerging lessons from Phase 2 (AFB/EFC.29/Inf.4) were presented to the twenty-ninth EFC meeting in March 2022, to update the Board on the second phase of ex post evaluations. The ex post evaluation work has progressed to its second phase and is piloting ex post methods in selected projects. At the point of reporting, fieldwork was completed in Samoa and and about to start in Ecuador.
The pilot of the ex post methodological approach in Samoa has provided a rich set of lessons that will inform adjustment of the approach for future ex post work. A few of these lessons are:
- The selected projects and IEs were receptive to the pilot evaluation
- Co-creation significantly facilitated stakeholder participation
- Mapping exercises were useful in identifying key stakeholders
- M&E data availability was a major constraint and needed adapted methodologies
- The absence of a Theory of Change at project design was a limitation to assessing sustainability.
Progress Update to the Board – October 2021
A progress update on ex post evaluations (AFB/EFC.28/Inf.4) was presented to the twenty-eighth Ethics and Finance Committee (EFC) meeting in October 2021, to update the Board on progress in the AF-TERG ex post evaluations and the future piloting of such evaluations.
The document concludes that following the shortlisting of five completed projects as potential candidates for the pilots, two projects were selected for ex post evaluations:
– Enhancing Resilience of Samoa’s Coastal Communities to Climate Change (Samoa, UNDP)
– Enhancing resilience of communities to the adverse effects of climate change on food security, in Pichincha Province and the Jubones River basin (Ecuador, WFP)
The next step will be to train evaluators for the piloting of ex post evaluation methods. This phase will enable the AF-TERG to revisit and shape methods with national partners, in line with its co-creation principle. This allows evaluators to adapt methods to assess sustainability and resilience of project outcomes in the field.
Phase one report
The Phase one report on ex post project sustainability evaluation provides an overview of the first stage of ex post evaluations. Building on preliminary studies, the report presents the methods that will be piloted in two ex post evaluations in FY22. It also identifies a list of potential projects for ex post evaluation pilots from the Fund’s 17 completed and evaluated projects.
The framework presented in the report introduces possible methods to evaluate the sustainability of project outcomes, considering the characteristics, strengths and weaknesses of the Fund portfolio. It presents an analysis tool to assess climate resilience, bearing in mind that this area is pivotal to climate change adaptation yet has rarely been measured.
The resilience analysis tool covers five components:
- The climate disturbances
- The human and natural systems (and their nexus) affected by and affecting project outcomes
- The characteristics of resilience in the outcomes
- means and actions supporting outcomes (exemplifying characteristics of resilience), and
- A typology of resistance-resilience-transformation (RRT) into which the overall project can be mapped based on how actions are designed to maintain or change existing structures and functions.
Training material for ex post pilots
The following videos and supporting material for conducting ex-post evaluations are used to train evaluators, relevant implementing entity counterparts, and national partners prior to field work. The approach focuses on a participatory evaluation of AF project results 3 to 5 years ex post by analyzing emerging sustainability, adaptation, and resilience through the proposed framework. The data collection and evaluation methods are tailored to specific country and on-the-ground realities.
The training is divided into two sessions (see videos and corresponding pdfs):
Part 1: Preparation for the Ex Post Evaluation (The “What”) introduces a framework for conducting ex post evaluations focused on sustainability, adaptation, and climate resilience through engaging in the ‘co-creation’ process. It covers how AF projects are chosen for ex post evaluation; sets a basis of understanding ex post evaluations of adaptation projects; introduces a framework of analysis for sustainability and for adaption and resilience; and, finally, highlights initial considerations for fieldwork preparation.
Part 2: Design of the Ex Post Evaluation (The “How”) discusses the project and its data more in-depth to understand how the selected project outcome(s) will be evaluated in ex post. This training covers understanding intervention context (stakeholder mapping, site selection) and logistical considerations; outlines in detail how to utilize both the sustainability and resilience frameworks and accompanying tools for data collection and analysis; and provides an intervention-specific recommendation for data collection methods. A third training deck/pdf provides an overview of a menu of several evaluation methods from which the evaluation team might choose for their particular intervention, as well as a worksheet on fieldwork troubleshooting.