Defining monitoring and evaluation

How do we Evaluate Programmes?

Definitions

Attention: do not confuse project monitoring with project evaluation. 

Project monitoring is identical with project performance measurement and belongs to the project execution and control  phase of project management. Monitoring and reporting the project performance is to review the project progress against expected milestones, timelines and costs. The purpouse of reporting is to share the information required to manage  Manage CSSQ (Cost, Scope, Schedule, and Quality). . With reference to tasks, actions and projects, monitoring is looking at what is currently happening. It is functional to the objective of delivering the project outputs with the expected quality and within the time and cost constrains defined in the project plan document. The deliverable of project appraisal is the Project Status (or progress) report (see project execution templates).

Project evaluation is not a part of project execution but is a different phase of the programme cycle.  Project evaluation is not done by the project manager but by the project sponsor (e.g. the Programme Manager, etc.) or by someone supporting her. It requires Project Status reports as a fundamental input. Its outputs are projects evaluation reports  and program lessons learned. Evaluation is based on monitoring but adds a "judgment" on the correlation between activities performed, outputs delivered, changes induced objectives achieved and impacts obtained upon the factors generating the problems and opportunities that motivated the project. While the delivery of project outputs is the responsibility of the project team, the achievements of project outcomes and objectives depends also by the way stakeholders utilize the project outputs to interact with the rest of the community and contribute to achieve the project objectives.

Monitoring is looking at what is going on. With reference to tasks, actions and projects, monitoring is looking at what is currently happening. for example: Are the people working at their tasks? Are these tasks being executed? Are the expected deliverables being delivered? Is the money being spent?  Are the procurements being consigned to the team? Are the outputs consigned to the beneficiaries?.  Through monitoring we arrive to an "assessment of facts".

As we have mentioned elsewhere, in a projectised organisation, programmes are implemented through projects. And in order to implement projects, teams carry out activities and tasks. Before beginning a project, the project manager, along with his team, makes a work plan that details the timelines of activities, individual responsibilities in the team and budget allocations for each activity. The work plan helps the team in having clarity on what they are doing, on their timelines to complete activities and tasks and on what their budgets allocations are. If you are a project manager, you need the workplan to monitor whether your team is implementing the project as planned and is spending within stipulated budgets. Monitoring is thus an integral activity of project management and implementation. The activity of establishing a monitoring and evaluation system is an activity in itself and must be planned well.

In order to make corrective measures, it is important that the information emerging from monitoring is analysed at the end of each project year. This exercise of analysing monitoring information is called a monitoring review. The objective of monitoring review is to empower staff in delivering programmes effectively. The learning that emerges from a monitoring review must be integrated into current programme implementation. In order that the learning become integrated into new programming, the information must be shared, among the wider programme team, during country learning reviews usually held every alternate year.

Evaluation is based on monitoring but adds a "judgment".  Is what happening supposed to happen? What we are doing is it the right thing? What we have done, was it done right?  Do the actions follow the plan?  Are the employees doing their tasks as they were expected to do? Are the tasks managers working efficiently?  Is the team working efficaciously?  Was the plan well conceived?  Are there new factors taking place that require modifications in the plan? Are the project deliverable leading to the expected project outputs? Are the project outputs leading to the expected project objectives? What are the outcomes and the impacts of the project? Is the project contributing as intended to the wider programme to which it was a component?  Are we learning from the experiences?

These judgment are possible as far as there is the possibility of comparing what is happening to what was expected to happen: therefore in order to carry out an evaluation we need at least two commonly visualized scenarios: the one that was expected (the program/project plan) and the one that is factual (the facts resulting from monitoring).   When comparison highlights difference between expectations and results, evaluation will attempt to find the causes of such discrepancies, attributing them either to lack of capacity of the implementing team, or lack of vision of planner, or to unexpected external factors, or other factors as it may be the case.  If the project is still running evaluations report will possibly indicate how to redesign the remaining actions in order to conciliate expectations and results. If the project is already completed, evaluations will indicate the  lessons learned from the project to be considered in the successive planning and implementation of similar projects. Evaluations are therefore an indispensable means for .programme cycle management and constitute an essential element of its actions.

While implementing a project the project team always need to carry on a certain level of monitoring and evaluation upon its own activities in order to constantly re-orient the work towards the expected results (see the section on project execution and control).  But self evaluation of the implementing team is not sufficient because development projects are part of a wider programme and therefore it is necessary that programme stakeholders understand how efficiently and efficaciously the resources devoted to the project have been used for the achievement of the expected project outcomes and impacts.  Evaluation therefore becomes a shared effort between various project stakeholders who want to understand how successful was a project and learn what are the factors that have to be considered in order to manage successfully follow up activities or similar projects in other contexts. While designing a new project and new programmes the lessons learnt from the evaluation of past actions is fundamental.  (See also  clarifying the relationship: Responsibilities of Project Managers and programme Managers).

Indicators

The purpose of evaluations is to empowering the different stakeholders to understand and learn all the project implications. Since evaluations are judgments, it is important that these judgments are based on documented facts and are interpreted at the light of values amongst project major stakeholders. If judgments happen to be perceived as arbitrary and subjective they cannot lead to meaningful evaluations.  While preparing a project it is therefore important to arrive at a consensus upon what will be measured during monitoring and evaluation phases, how it will be measured and against what parameters these measures will be interpreted.  What is going to be measured is called an "indicator".  A constellation of indicators quantifies qualitative processes and facilitate the description of the complex balance of social systems.  Development projects aim at improving the quality of life of the beneficiaries. And so the big question is "how to measure objectively and transparently the "quality of life"?  Only quantities can be measured objectively, judgment about qualities tent to be subjective to the value system of the judge. So in order to be "as objective as possible" we have to identify a constellation of quantitative parameters that can be actually measured and that have a logical relationship with the quality we intend to evaluate. Like in the case of the "health of a person" (that is a quality) we can use a number of objectively verifiable parameters that can help understanding the health balance (various physical analysis, psychological responses, etc.). In the same manner we can identify a series of economic, social and cultural parameters that can through light on the quality of life of the beneficiary population.

There are different kind of evaluations according if it is with reference to the effectiveness or the efficiency of the action, i.e. with reference of what qualitative difference has it made to the life of the beneficiaries and according to what qualitative standard has the action been carried out. (see later sections of this chapter). Accordingly in the logical framework we describe different indicators (and different sources of information) as they are chosen in order to evaluate how far the project was able to achieve the objectives (specific and overall), to produce the outputs (and their quality standard) and carry out the intended activities (with what level of efficiency).

In order to through light on project effectiveness and efficiency indicators should respond to certain "validity" criteria, i.e. they should be:

• measurable,

• understandable,

• acceptable (by all stakeholders),

• pertinent.

oxfam definition of indicators

----  Other resources from the web:

• Evaluation (1) assesses the effectiveness of an ongoing program in achieving its objectives, (2) relies on the standards of project design to distinguish a program's effects from those of other forces, and (3) aims at program improvement through a modification of current operations.

• Using collected information (assessments) to make informed decisions about continued instruction, programs, and activities.

• The process of determining whether an item or activity meets specified criteria.

• Assessment against a standard. Evaluations can assess both the process (of establishing a programme to deliver an outcome) and outcomes (ultimate objectives)

• An assessment plan to determine the degree to which the project has successfully met the objectives.

• is the assessment of how well a project/activity achieved its objectives.

• an assessment at a point in time of the value, worth or impact of a project or program.

• An analysis undertaken at a fixed point in time to determine the degree to which stated objectives have been reached. This is generally used as a basis for decision making, including updating plans.
   

Definitions of performance evaluation on the Web:

• An evaluation that compares actual performance with that planned in terms of both resource utilization and production. It is used by management to redirect program efforts and resources and to redesign the program structure.

• The method of evaluating an employee's performance which involves tracking, evaluating and giving feedback of actual performance based on key behaviors/competencies established in the goals that support the achievement of the overall organizational mission.

• A step in the performance management process that summarizes the employee's performance over the performance period that has just ended. The process is used to know how he or she is progressing and in what ways development might be desirable; the process may be used to determine employee salary increases.

   

Definitions of process evaluation on the Web:

• Process evaluation focuses on how a program was implemented and operates. It identifies the procedures undertaken and the decisions made in developing the program. It describes how the program operates, the services it delivers, and the functions it carries out. Like monitoring evaluation, process evaluation addresses whether the program was implemented and is providing services as intended. ...  

• The systematic collection of information to document and assess how a program was implemented and operates.

• A process evaluation assesses the extent to which a program or process is operating as intended and identifies opportunities for streamlining or otherwise improving it. Process evaluations often begin with an analysis of how a program currently operates. Process evaluations may also assess the extent to which program activities conform to statutory and regulatory requirements, Agency policies, program design or customer expectations. 

• A process evaluation examines specific components of a program to describe its operations, intentions and internal relationships. Rather than concentrating on the outcomes or results obtained, process evaluations focus on program monitoring and efficiency of services.  

Definitions of outcome evaluation on the Web:

• the systematic collection of information to assess the impact of a program, present conclusions about the merit or worth of a program, and make recommendations about future program direction or improvement.

• An evaluation used to identify the results of a program's/initiative's effort. It seeks to answer the question, "What difference did the program make?" It provides information about effects of a program after a specified period of operation.

• Outcome evaluations assess the reasons for differences between outcomes and objectives (eg, why the number and quality of permits issued exceeded or fell short of objectives). Outcome evaluations may include an examination of program processes and activities to understand how outcomes are achieved and how quality and productivity could be improved.

Definitions of evaluation criteria on the Web:

•  Predetermined requirements used as a basis for assessing tenders or measuring performance.  

Definitions of program evaluation on the Web:

• The systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development.

• Program evaluation is essentially a set of philosophies and techniques to determine if a program 'works'. It is a practice field that has emerged, particularly in the USA, as a disciplined way of assessing the merit, value, and worth of projects and programs. Evaluation became particularly relevant in the 1960s during the period of the Great Society social programs associated with the Kennedy and Johnson administrations.

Definitions of program evaluation review technique on the Web:

• A technique for management of a program through to completion by constructing a network model of integrated activities and events and periodically evaluating the time/cost implications of progressed.

In the course in Canada, they told us that for each specific objective (the level at which we measure outcomes), we must have a group of 3 - 4 indicators that can together measure the change we intend achieving for  that particular outcome. In order to measure the change (in behaviour, practices, standard of living, health, education, other impacts), data must be collected for each indicator at intervals (1 year, 6 months, etc.) and while evaluating, we assess what, where and how of baseline data and assess whether this change is being made

So, while establishing a monitoring system we must include collection of baseline data as one of the activities and decide who, what and where. During evaluation, we must assess whether this baseline data is really giving the data whereby we can measure the change we intend making. 

So, what do you think- we should have a chapter on monitoring and evaluation systems rather than one on monitoring and another one on evaluation? Even revising logical frameworks is part of the evaluation.

Definition by OECD  :    "Performance monitoring is a continuous process of collecting and analysing data to compare how well a project, programme or policy is being implemented against expected results.    Glossary of Key terms in Evaluation and Results Based Management, OECD, Paris, 2002

See also

Cost and Schedule Control

Motivating the Project Team

further reflections

Job profiles of an evaluation expert in a development organization