How do we Evaluate Programmes?
Co-operation
for development projects cannot be measured with strictly financial parameters, because
the sponsor does not expect financial return for its investment. In this case, the
question of evaluation concerns the benefices acquired by the intended target population
and this requires the capacity of understanding unexpressed requirements. On the other
side, the sponsoring entity (EU) expects some sorts of non-financial benefices in terms of
peace-building, image generation, cultural integration etc. So the task of the evaluator
is to understand how those projects actually contribute to such intangible and
non-measurable objectives. For doing that, it has to fully understand the social and
cultural relevance of promoting co-operation for development.
In the field
of communication, plans cannot be evaluated on the grounds of mere technical parameters,
because communication is an interactive process. Good communication results can be
achieved only by valuing creativity and giving scope to the freedom of expression of the
counterpart. Creativity and freedom, by nature, are not items that can be
“planned” ex ante or “measured” ex post.
Therefore, in co-operation and communication projects the typical
difficulty of evaluation, i.e. the absence of “objective parameters, becomes more
prominent.
One may try
to categorise sectors which are more or less “measurable”. But essentially the
hard fact remains that an evaluator cannot be just “objective”, because
evaluation is a judgement, and judgements are expressions of the responsibility and the
intelligence of those who judge. In other
words, one cannot evaluate by simply applying technical parameters; one has necessarily to
use one’s discriminatory insight into the problems.
Some people are terrified by this necessary “subjectivity” of the evaluation process. They dream of a free-from-opinions methodology. This is wrong. Because it is the very subjective nature of evaluation that constitutes its worth. Evaluation is the means to avail of the intellectual resources of those persons who, in the process of evaluation, contribute to orient technical performances towards non technical goals.
In this way,
we have indicated the first danger of evaluation: technicalism, which is the effort to
avoid judgements by stating mere technical facts; this approach deprives evaluation of its
potency.
The second
danger of evaluation is on the opposite front: aestheticism, i.e. a kind of judgement based on preconceived
aspiration and assumptions. This is the mistake of those who see the value of cultural
principles and ethical norms, but do not clearly see the complexity of concrete social
processes.
The real
contribution of evaluation consists in judging the ideas on the ground of their
practicability. In good evaluations, ideas are judged by considering their capacity to
lead to the intended results. What the evaluator values are not the ideas per se. What
really matters is the possibility of the realisation of these ideas, the coherence between
plans and actions, the sustainability of the project/programs undertaken and their impact upon a
wider context.
Technicalism
and aestheticism have their share of merit. Mistakes arise from carrying too far the
solutions adopted in order to correct previous mistakes. Technicalism and aestheticism
arise from the necessity of correcting each other. These two positions find themselves at
the extreme polarities of the possible approaches to evaluation.
Good
evaluation lies in adopting a “middle path”, i.e. a judgement where the
objective and the subjective criteria are well balanced. Good project are projects where
objectives and means are coherent. Evaluation of a communication process requires the
capacity to see the forms and the contents of communication as an inseparable unity.
The capacity
of positioning herself/himself in the “middle path” is even more important when
the evaluator acts as middle-person between the sponsor and executor of a project, as in the context of EU actions for
development co-operation. In this context, the evaluator does not act only as a judge of
the practicability/implementation of projects. The evaluator also has to understand and
re-express the requirements of the two entities involved. As in the case of communicators,
the evaluator has to fully integrate the aspiration and the requirements of both the
sender and the receiver of the information. Only by understanding the message, can one
translate it. Difficulties in communication often lie in the fact that one side adopts a
more “technical” approach and the other side a more “aesthetic” one.
The evaluator who can maintain the “middle path” has also the capability of
promoting the integration of the respective aspirations of the sponsor and the executor of
projects.
This
“middle path approach” is the key element upon which GICO has constructed its
evaluation methodology.
Criteria 1: evaluation of information
actions will focus on their efficacy in raising interest, evoking response and promoting
participation.
The target
public will never bee seen as a passive receiver of pre-conceived notions. Communication
is not a mechanical process of handing over “something” from a sender to a
receiver, but is a human process aimed at generating reciprocal understanding. We will try
to discourage the adoption of rhetorical forms of communication and promote a sense of
respect for the people to whom communication is intended.
Criteria 2:
judgement will be based upon the capacity of communication to make the communicating ends
closer to each other.
Communication
has achieved its purpose when the persons at the two poles of the communication act will
develop a sense of common identity. The success of communication can be measured by the
fact that the we-you polarity is resolved in a
sense of “we” congruity. We will try
to discourage aggressive and effective forms of communication. We will support those
communication approaches capable of generating a sense of authentic solidarity.
Information
and communication materials are means to a non material scope. Quantitative analysis can
be applied only to means. The aim of a communication process can be understood only by
qualitative evaluation. Our analysis will not focus on the use of a particular media but
on the modalities media are used in order to enter into relationship with the
counterparts. Finally the qualitative results of communication depends on the quality of
the relationship established.
One cannot
evaluate the impact of communication actions aimed at promoting development co-operation if one does not share the aspiration
to achieve such co-operation goals. For this purpose we will consider the importance of
including in the evaluation panels the people whose interest are positively related to the
achievement of the objectives of the projects. Relevant witnesses will not only be “interviewed”, but will
be given an opportunity to take part in the process of evaluation.
We believe in
the value of open debate and open criticism. It is important that evaluators keep on
evaluating their own methods. This can be achieved only by disclosing the evaluation
approaches to all stakeholders.
We will try to
discourage a servile attitude of the communicator to power structures. We will promote initiative and creativity.
Even when the
objective of GICO services is not the formulations of recommendations for improving the
impact, GICO will always add to project evaluation reports a retroaction report aimed at
evaluating EU actions on the basis of the success of the
sponsored projects.
We will try
to shield SCR from the risk to pull communicators towards adopting Brussels-styled bureaucratic rhetoric. We will try
to convey to SCR the richness of creative approaches we will discover in project
evaluation processes.
Criteria 7: Evaluation reports will
be organised as to generate a knowledge management system
The task unit
of GICO will take special care to integrate all evaluation reports in a knowledge databank
which can be used as resource for successive evaluations and as a decision support system.
The confidential part of evaluation reports will be made available only to GICO evaluators
and to SCR. The general reflections on the use of information and communication to promote
development co-operation will be used to promote open debate on these issues. For this
scope, if GICO is selected, we will open a
special seminar room in the electronic forum GICO is conducting on “Transmitting the
Heritage - The role of global media in the transmission of cultural heritage for a
sustainable development of human resources compatible with the different cultural
identities” (http://web.tiscalinet.it/heritageforum).