Ex post evaluations are a key element of the AF-TERG FY21-FY23 strategy and work programme, and beyond, originating from the request of the Adaptation Fund Board to develop post-implementation learning for Fund projects and programmes and provide accountability of results financed by the Fund.

They intend to evaluate aspects of both sustainability of outcomes and climate resilience, and over time feed into ex-post-evaluation-informed adjustments within the Fund’s Monitoring Evaluation and Learning (MEL) processes.

The framework developed by the AF-TERG is an innovative and Fund-specific framework to conduct ex post evaluations. Given the relative novelty of climate change adaptation portfolios and the limited body of work on ex post evaluation for adaptation, it presents possible methods that have been piloted and field-tested in ex post evaluations in Samoa and Ecuador during fiscal year 2022 (FY22).

Information update on phase 2 – October 2022

An information update on phase 2 of the ex post project sustainability evaluations (AFB/EFC.30/Inf.3) was presented to the thirtieth Ethics and Finance Committee (EFC) meeting in October 2022 and outlines the findings of the first two ex post evaluation pilots, after completion of fieldwork, and highlights lessons for the Fund and its implementers. The document also describes the next steps for AF-TERG ex post evaluations based on lessons learned from the first two pilots.

The AF-TERG will take steps to move from the pilot phase of ex post evaluation to a standard set of procedures for selecting and implementing ex post evaluations, including protocols for engaging with in-country partners and for communicating results. The AFTERG will also continue to refine the methodology for ex post evaluations. Furthermore, the AF-TERG will discuss potential approaches for evaluations of fragile states, while ensuring the safety of all participants. Fragile states comprise a significant share of the current eligible pool of completed projects.

 

Progress Update to the Board – March 2022

A progress update on ex post evaluations and emerging lessons from Phase 2 (AFB/EFC.29/Inf.4) were presented to the twenty-ninth EFC meeting in March 2022, to update the Board on the second phase of ex post evaluations. The ex post evaluation work has progressed to its second phase and is piloting ex post methods in selected projects. At the point of reporting, fieldwork was completed in Samoa and and about to start in Ecuador.

The pilot of the ex post methodological approach in Samoa has provided a rich set of lessons that will inform adjustment of the approach for future ex post work. A few of these lessons are:

  • The selected projects and IEs were receptive to the pilot evaluation
  • Co-creation significantly facilitated stakeholder participation
  • Mapping exercises were useful in identifying key stakeholders
  • M&E data availability was a major constraint and needed adapted methodologies
  • The absence of a Theory of Change at project design was a limitation to assessing sustainability.

 

Progress Update to the Board – October 2021

A progress update on ex post evaluations (AFB/EFC.28/Inf.4) was presented to the twenty-eighth Ethics and Finance Committee (EFC) meeting in October 2021, to update the Board on progress in the AF-TERG ex post evaluations and the future piloting of such evaluations.

The document concludes that following the shortlisting of five completed projects as potential candidates for the pilots, two projects were selected for ex post evaluations:

– Enhancing Resilience of Samoa’s Coastal Communities to Climate Change (Samoa, UNDP)

– Enhancing resilience of communities to the adverse effects of climate change on food security, in Pichincha Province and the Jubones River basin (Ecuador, WFP)

The next step will be to train evaluators for the piloting of ex post evaluation methods. This phase will enable the AF-TERG to revisit and shape methods with national partners, in line with its co-creation principle. This allows evaluators to adapt methods to assess sustainability and resilience of project outcomes in the field.

 

Phase one report

The Phase one report on ex post project sustainability evaluation provides an overview of the first stage of ex post evaluations. Building on preliminary studies, the report presents the methods that will be piloted in two ex post evaluations in FY22. It also identifies a list of potential projects for ex post evaluation pilots from the Fund’s 17 completed and evaluated projects.

The framework presented in the report introduces possible methods to evaluate the sustainability of project outcomes, considering the characteristics, strengths and weaknesses of the Fund portfolio. It presents an analysis tool to assess climate resilience, bearing in mind that this area is pivotal to climate change adaptation yet has rarely been measured.

The resilience analysis tool covers five components:

  1. The climate disturbances
  2. The human and natural systems (and their nexus) affected by and affecting project outcomes
  3. The characteristics of resilience in the outcomes
  4. means and actions supporting outcomes (exemplifying characteristics of resilience), and
  5. A typology of resistance-resilience-transformation (RRT) into which the overall project can be mapped based on how actions are designed to maintain or change existing structures and functions.

 

Training material for ex post pilots

The training material for ex-post pilots are used for the training of evaluators and national partners before the field work. The training focuses on evaluating projects at ex post and emerging sustainability and resilience, as well as presenting and adapting methods to country and project realities.

The training is divided into three sessions:

Part A: Understanding ex post and resilience evaluations. Introduce and understand ex post evaluations of sustainability and resilience, especially in the field of climate change adaptation

Part B: Discussing country-specific outcome priorities and co-creating learning with stakeholders. Discuss the project and its data more in-depth to understand and select what outcome(s) will be evaluated at ex post

Part C: Developing country-specific methods and approaches. Discuss range of methods with the national evaluator and M&E experts to best evaluate the selected outcome(s) and impact(s)