Evaluation Guidelines for IOM Development Fund Projects 2020

IOM DEVELOPMENT FUND PROJECT EVALUATIONS

Background

  1. Since 2017, the IOM Development Fund requires all projects to include an evaluation preferably 6 to 12 months after the project is completed. The specific objective of the evaluations is to assess the relevance of project design, coherence of the interventions vis a vis other interventions, the effectiveness and performance of the project, the efficiency of project management and implementation, the impact and sustainability of the project.
  1. Evaluations promote transparency and accountability, which will assist the IOM Development Fund in its decision-making on the use of the Fund as seed funding, on project management and to fine-tune interpretation and categorisation of the funding criteria and overall regional disbursement strategies.

 

Planning and Administration of the Evaluation

  1. During project development, the Project Developer needs to include in the Operational part of the Budget the cost of this evaluation of at least five per cent (5%) of the total budget. The allocated budget is strictly for project evaluation use and should remain unspent until the project has been completed or until the project evaluation is to be conducted.
  1. RO Monitoring and Evaluation Officers can contribute to carry out these evaluations so that only flight and DSA costs need to be budgeted. However, this has to be coordinated at the project development stage with the RO M&E Officers in order for them to include the exercises in their evaluation plans. Otherwise, the mission will have to rely on other IOM staff or on external consultants. The Evaluation Unit of the Office of the Inspector General, as well as the RO M&E Officers can provide support to identify internal options or to contract an external evaluator. Depending on the option chosen, evaluation estimated costs should be within five per cent (5%) of the total budget.
  1. During the final stage of project implementation and in particular if the Project Manager is no longer available after project end to supervise an ex-post evaluation, the conduct of the project evaluation is coordinated by the IOM managing Mission with the IOM Development Fund.
  1. For ex-post project evaluations, the budget for the evaluation will be transferred to the IOM Development Fund evaluation code as the main project code will be closed during Regional Accounting Support (RAS) endorsement of the final financial report.
  1. A WBS code from the IOM Development Fund evaluation code will be provided to the IOM Mission upon confirmation of the evaluation period. Only costs related to the ex-post project evaluation should be charged to the relevant WBS code provided and this will be closed once the evaluation is completed.
  1. The IOM Development Fund will request confirmation that the evaluation is on-going.
  1. Any remaining balance on the evaluation budget will remain with the IOM Development Fund and be carried forward for future IOM Development Fund regional and thematic project evaluation initiatives.

 

Conducting Evaluation

  1. Terms of Reference (ToRs) need to be developed by the relevant Mission and shared beforehand for review with the IOM Development Fund and the RO M&E Officers, as well as with other relevant stakeholders, in particular other Missions involved in the implementation of the project. The ToRs should include a brief project background, define a clear objective for the evaluation, explaining why the evaluation is undertaken and what the evaluation intends to accomplish, the scope of the evaluation or plan for conducting the evaluation, the issues to be addressed through the listing of the evaluation criteria that will be used, as well as related questions, the products expected from the evaluation, which may include in addition to the report itself particular strategies or recommendations on some specific areas, the methodology used, the party responsible for conducting the evaluation and finally the implementation arrangement, including a realistic timeframe and logistical arrangements. Gender related analysis and questions also need to be included in the ToRs and be properly covered by the evaluation. An example of TOR can be found in Annex 6.1 of the IOM Project Handbook (second edition) and the UNEG Quality Checklist for Evaluation ToRs is available under the IOM Evaluation Webpage https://www.iom.int/technical-references .
  1. Refer to ‘IOM Evaluation Guidelines’ of January 2006 and Module 6 “Evaluation” of the IOM Project Handbook (second edition) to assist in the preparation of the Evaluation.
  1. The following OECD/DAC evaluation criteria should be included in the ToRs:

a. Relevance of the project - the extent to which the project/programme’s objectives and intended results remain valid and pertinent either as originally planned or as subsequently modified (it can include an analysis of the validity of design).

b. Effectiveness of the project considers the extent to which a project or programme achieves its objectives/intended results, taking into account their relative importance. It is equally important to examine if changes would have occurred regardless, without the implementation of the project or programme. Effectiveness is also used as an aggregate measure of (or judgment about) the merit or worth of an activity. Cost-effectiveness examines whether the objectives were achieved at minimal cost (or at the lowest possible cost), or whether the results or benefits justify the cost.

c. Efficiency of project management and implementation - analyses how well resources in general (funds, expertise, time, etc.), or inputs, are used to undertake activities, and are converted into outputs. Sometimes, the definition also takes into account the analysis of alternatives for the use of resources.

d. Coherence of the interventions vis a vis other interventions The extent to which other interventions (particularly policies) support or undermine the intervention, and vice versa. Includes internal coherence and external coherence: Internal coherence addresses the synergies and interlinkages between the intervention and other interventions carried out by the same institution/government, as well as the consistency of the intervention with the relevant international norms and standards to which that institution/government adheres. External coherence considers the consistency of the intervention with other actors’ interventions in the same context. This includes complementarity, harmonisation and co-ordination with others, and the extent to which the intervention is adding value while avoiding duplication of effort.

e. Impact - how activities of the project/programme contributed to a change in a situation, intended or unintended, directly or indirectly, positive or negative, and what the project/programme was expected to bring. This criterion is raising major debates, especially due to the methodological constraints imposed by its measurement. What is important to remember when discussing the impact is first to properly define what is intended by ‘impact’, or which impact, in the framework of the evaluation and then examine how it can be observed, measured and described. Due to the complexity of measuring it rigorously, the ‘judgment of the wise’ based on qualitative and quantitative elements available and observation can sometimes be used as a solution for discussing globally the impact.

f. Sustainability of the project to date - is the durability of the project’s results, or the continuation of the project’s benefits once external support ceases. The notion of ‘seed money’ as an IOM Development Fund criteria should be examined here.

  1. Examples of questions related to the Evaluation criteria can be found in annex 3 of the ‘IOM Evaluation Guidelines’ of January 2006.
  1. An Evaluation Matrix is required as part of the evaluation process. The evaluation matrix is a tool for guiding the evaluation by specifying: (a) the criteria being assessed by the evaluation; (b) the questions and sub-questions that will be answered in order to assess each criteria; (c) the indicators to be used to guide the assessment; (d) the sources of data; and (e) the data collection tools.
  1. For standardisation, IOM Development Fund requires the use of Annex 6.8 of the IOM Project Handbook (Second Edition): Template for a Final Evaluation Report, p.476.
  1. A draft copy of the evaluation report, evaluation brief and evaluation management response needs to be submitted to the IOM Development Fund within two months of the evaluation start date. The final copy of the evaluation report, evaluation brief and evaluation management response needs to be submitted to the IOM Development Fund and to OIG Evaluation Unit within two months of completion of the draft report.
  1. The evaluation report will then be shared with the relevant IOM mission(s) and governments and will be published on the OIG webpage and evaluation repository.