Ex post evaluations are a key element of the AF-TERG FY21-FY23 strategy and work programme, and beyond, originating from the request of the Adaptation Fund Board to develop post-implementation learning for Fund projects and programmes and provide accountability of results financed by the Fund.
They intend to evaluate aspects of both sustainability of outcomes and climate resilience, and over time feed into ex-post-evaluation-informed adjustments within the Fund’s Monitoring Evaluation and Learning (MEL) processes.
The framework developed by the AF-TERG is an innovative and Fund-specific framework to conduct ex post evaluations. Given the relative novelty of climate change adaptation portfolios and the limited body of work on ex post evaluation for adaptation, it presents possible methods that have been piloted and field-tested in ex post evaluations in Samoa and Ecuador during fiscal year 2022 (FY22).
Ex post evaluation summary – Ecuador (2022)
National evaluator Monica Ribadeneira Sarmiento began the Ecuador ex post evaluation in November 2021. Over ten months, it was carried out in different stages: review of project documentation; capacity-building training; selection of outcomes to evaluate ex post; field visit and data collection; data analysis; and report write-up.
The fieldwork showed that sustainability of outcomes after project completion was only moderate. This result was driven by partial ownership of assets depending on the site, few resources for partnerships, and a limited impact on “decreased vulnerability.” Results differed significantly between sites. Assets in Cochapata were relevant and well maintained by the community, but those in Nabón were partially unusable or abandoned.
The resilience analysis tool indicates weaker prospects for climate resilience. The resilience characteristics exhibited by the assets did not seem to influence food security in both sites or were not enough to prevent outmigration from one site. The overall strategy of the outcomes was to maintain systems (and their structures and functions) for water management and food security. Thus, the resilience of investments can be classified as active and passive resistance.
Ex post evaluation summary Samoa (2022)
National evaluator Karen Komiti began the ex post evaluation of the Samoa project “Enhancing Resilience of Samoa’s Coastal Communities to Climate Change”, at the end of October 2021. It covered different stages over five months: review of project documentation; selection of outcomes to evaluate ex post; field visit and data collection; data analysis; and report write-up.
Five years after construction, the structures across four sites/six villages remain physically intact but some sections of Manase and Salimu / Musumusu rockwalls appear to be deteriorating. In general, the structures are adequately and routinely maintained by stakeholders at household, village, and government levels. These activities have not diminished in the years since project closure, despite the absence of secure funding in government operational budget, an infrastructure-specific risk management plan, and co-financing to enable maintenance beyond closure.
As far as the resistance-resilience-transformation typology, all seven structures exhibit various levels of resilience (mainly passive resistance and some resilience) . All have remained standing in the face of several climate disturbance impacts in the intervening years.
Information update on phase 2 – October 2022
An information update on phase 2 of the ex post project sustainability evaluations (AFB/EFC.30/Inf.3) was presented to the thirtieth Ethics and Finance Committee (EFC) meeting in October 2022 and outlines the findings of the first two ex post evaluation pilots, after completion of fieldwork, and highlights lessons for the Fund and its implementers. The document also describes the next steps for AF-TERG ex post evaluations based on lessons learned from the first two pilots.
The AF-TERG will take steps to move from the pilot phase of ex post evaluation to a standard set of procedures for selecting and implementing ex post evaluations, including protocols for engaging with in-country partners and for communicating results. The AFTERG will also continue to refine the methodology for ex post evaluations. Furthermore, the AF-TERG will discuss potential approaches for evaluations of fragile states, while ensuring the safety of all participants. Fragile states comprise a significant share of the current eligible pool of completed projects.
Progress Update to the Board – March 2022
A progress update on ex post evaluations and emerging lessons from Phase 2 (AFB/EFC.29/Inf.4) were presented to the twenty-ninth EFC meeting in March 2022, to update the Board on the second phase of ex post evaluations. The ex post evaluation work has progressed to its second phase and is piloting ex post methods in selected projects. At the point of reporting, fieldwork was completed in Samoa and and about to start in Ecuador.
The pilot of the ex post methodological approach in Samoa has provided a rich set of lessons that will inform adjustment of the approach for future ex post work. A few of these lessons are:
- The selected projects and IEs were receptive to the pilot evaluation
- Co-creation significantly facilitated stakeholder participation
- Mapping exercises were useful in identifying key stakeholders
- M&E data availability was a major constraint and needed adapted methodologies
- The absence of a Theory of Change at project design was a limitation to assessing sustainability.
Progress Update to the Board – October 2021
A progress update on ex post evaluations (AFB/EFC.28/Inf.4) was presented to the twenty-eighth Ethics and Finance Committee (EFC) meeting in October 2021, to update the Board on progress in the AF-TERG ex post evaluations and the future piloting of such evaluations.
The document concludes that following the shortlisting of five completed projects as potential candidates for the pilots, two projects were selected for ex post evaluations:
– Enhancing Resilience of Samoa’s Coastal Communities to Climate Change (Samoa, UNDP)
– Enhancing resilience of communities to the adverse effects of climate change on food security, in Pichincha Province and the Jubones River basin (Ecuador, WFP)
The next step will be to train evaluators for the piloting of ex post evaluation methods. This phase will enable the AF-TERG to revisit and shape methods with national partners, in line with its co-creation principle. This allows evaluators to adapt methods to assess sustainability and resilience of project outcomes in the field.
Phase one report
The Phase one report on ex post project sustainability evaluation provides an overview of the first stage of ex post evaluations. Building on preliminary studies, the report presents the methods that will be piloted in two ex post evaluations in FY22. It also identifies a list of potential projects for ex post evaluation pilots from the Fund’s 17 completed and evaluated projects.
The framework presented in the report introduces possible methods to evaluate the sustainability of project outcomes, considering the characteristics, strengths and weaknesses of the Fund portfolio. It presents an analysis tool to assess climate resilience, bearing in mind that this area is pivotal to climate change adaptation yet has rarely been measured.
The resilience analysis tool covers five components:
- The climate disturbances
- The human and natural systems (and their nexus) affected by and affecting project outcomes
- The characteristics of resilience in the outcomes
- means and actions supporting outcomes (exemplifying characteristics of resilience), and
- A typology of resistance-resilience-transformation (RRT) into which the overall project can be mapped based on how actions are designed to maintain or change existing structures and functions.
Training material for ex post pilots
The training material for ex-post pilots are used for the training of evaluators and national partners before the field work. The training focuses on evaluating projects at ex post and emerging sustainability and resilience, as well as presenting and adapting methods to country and project realities.
The training is divided into three sessions:
Part A: Understanding ex post and resilience evaluations. Introduce and understand ex post evaluations of sustainability and resilience, especially in the field of climate change adaptation
Part B: Discussing country-specific outcome priorities and co-creating learning with stakeholders. Discuss the project and its data more in-depth to understand and select what outcome(s) will be evaluated at ex post
Part C: Developing country-specific methods and approaches. Discuss range of methods with the national evaluator and M&E experts to best evaluate the selected outcome(s) and impact(s)