The ‘Most Significant Change’ (MSC) Technique

How do we Evaluate Programmes?

Data collection methods

Data Analysis methods

 

 

• definition of change

• outcomes as behavioral change

• How change happens
• Organizational and Individual Change

 

The most significant change (MSC) technique is a form of participatory monitoring and evaluation.

Essentially, the process involves the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. The designated staff and stakeholders are initially involved by ‘searching’ for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact.

MSC does not make use of pre-defined indicators, especially ones that have to be counted and measured. The answers to the central question about change are often in the form of stories of who did what, when and why – and the reasons why the event was important. )

 

implementation steps

 

1. How to start and raise interest

2. Defining the domains of change

3. Defining the reporting period

4. Collecting SC stories

5. Selecting the most significant of the stories

6. Feeding back the results of the selection process

7. Verification of stories

8. Quantification

9. Secondary analysis and meta-monitoring

10. Revising the system.

 

SC stories are collected from those most directly involved, such as participants and field staff. The stories are collected by asking a simple question such as: ‘During the last month, in your opinion, what was the most significant change that took place for participants in the program?’ It is initially up to respondents to allocate their stories to a domain category. In addition to this, respondents are encouraged to report why they consider a particular change to be the most significant one.

The stories are then analysed and filtered up through the levels of authority typically found within an organisation or program. Each level of the hierarchy reviews a series of stories sent to them by the level below and selects the single most significant account of change within each of the domains. Each group then sends the selected stories up to the next level of the program hierarchy, and the number of stories is whittled down through a systematic and transparent process. Every time stories are selected, the criteria used to select them are recorded and fed back to all interested stakeholders, so that each subsequent round of story collection and selection is informed by feedback from previous rounds. The organisation is effectively recording and adjusting the direction of its attention – and the criteria it uses for valuing the events it sees there.

After this process has been used for some time, such as a year, a document is produced with all stories selected at the uppermost organisational level over that period in each domain of change. The stories are accompanied by the reasons the stories were selected. The program funders are asked to assess the stories in this document and select those that best represent the sort of outcomes they wish to fund. They are also asked to document the reasons for their choice. This information is fed back to project managers.

The selected stories can then be verified by visiting the sites where the described events took place. The purpose of this is two-fold: to check that stories have been reported accurately and honestly, and to provide an opportunity to gather more detailed information about events seen as especially significant. If conducted some time after the event, a visit also offers a chance to see what has happened since the event was first documented.

The next step is quantification, which can take place at two stages. When an account of change is first described, it is possible to include quantitative information as well as qualitative information. It is also possible to quantify the extent to which the most significant changes identified in one location have taken place in other locations within a specific period. The next step after quantification is monitoring the monitoring system itself, which can include looking at who participated and how they affected the contents, and analysing how often different types of changes are reported. The final step is to revise the design of the MSC process to take into account what has been learned as a direct result of using it and from analysing its use.

The kernel 

The kernel of the MSC process is a question along the lines of:

‘Looking back over the last month, what do you think was the most significant change in [particular domain of change]?’

A similar question is posed when the answers to the first question are examined by another group of participants:

‘From among all these significant changes, what do you think was the most significant change of all?’

This process provides a simple means of making sense of a large amount of complex information collected from many participants across a range of settings.

Telling each level about the choice of significant changes made at the higher levels is an essential component of the whole process. This helps readjust the focus of searches for significant change in each subsequent reporting period.

Figure 1. The MSC selection process (example from ADRA Laos)

 

The purpose

There are several reasons why a wide range of organisations have found MSC monitoring very useful and these include the following.

1. It is a good means of identifying unexpected changes.

2. It is a good way to clearly identify the values that prevail in an organisation and to have a practical discussion about which of those values are the most important. This happens when people think through and discuss which of the SCs is the most significant. This can happen at all levels of the organisation.

3. It is a participatory form of monitoring that requires no special professional skills. Compared to other monitoring approaches, it is easy to communicate across cultures. There is no need to explain what an indicator is. Everyone can tell stories about events they think were important.

4. It encourages analysis as well as data collection because people have to explain why they believe one change is more important than another.

5. It can build staff capacity in analysing data and conceptualising impact.

6. It can deliver a rich picture of what is happening, rather than an overly simplified picture where organisational, social and economic developments are reduced to a single number.

7. It can be used to monitor and evaluate bottom-up initiatives that do not have predefined outcomes against which to evaluate.

 

When and when not to use MSC

MSC is better suited to some program contexts than others. In a simple program with easily defined outcomes (such as vaccination, perhaps), quantitative monitoring may be sufficient and would certainly consume less time than MSC. In other program contexts, however, conventional monitoring and evaluation tools may not provide sufficient data to make sense of program impacts and foster learning. The types of programs that are not adequately catered for by orthodox approaches and can gain considerable value from MSC include programs that are:

• complex and produce diverse and emergent outcomes

• large with numerous organisational layers

• focused on social change

• participatory in ethos

• designed with repeated contact between field staff and participants

• struggling with conventional monitoring systems

• highly customised services to a small number of beneficiaries (such as family counselling).

Monitoring and evaluation in an organisation may serve several purposes. MSC addresses some purposes more than others. In our experience, MSC is suited to monitoring that focuses on learning rather than just accountability. It is also an appropriate tool when you are interested in the effect of the intervention on people’s lives and keen to include the words of non-professionals. In addition, MSC can help staff to improve their capabilities in capturing and analysing the impact of their work.

There are also some instances where the benefits may not justify the cost of MSC. While MSC can be used to address the following, there may be other less time-consuming ways to achieve the same objectives:

• capture expected change

• develop good news stories for public relations (PR)

• conduct retrospective evaluation of a program that is complete

• understand the average experience of participants

• produce an evaluation report for accountability purposes

• complete a quick and cheap evaluation.

Some program contexts are more conducive to the successful implementation of MSC. In our experience, some of the key enablers for MSC are:

• an organisational culture where it is acceptable to discuss things that go wrong as well as success

• champions (i.e. people who can promote the use of MSC) with good facilitation skills

• a willingness to try something different

• time to run several cycles of the approach

• infrastructure to enable regular feedback of the results to stakeholders

• commitment by senior managers

 

USA – using MSC for small, individualised programs “… the services provided through this program are highly individualised. Families come to the program with very different needs and skills. We are charged with documenting the number of families that have made progress, but the definition of progress is different for each family. is makes it very difficult to use any kind of standardised measure of change. For all of these reasons we’ve begun investigating the MSC approach.” (Julie Rainey, 2001, Family Literacy Program)

 

Where to get further information

For continuing access to information about MSC, including new usages and the experiences of existing users, you might like to join the Most Significant Changes mailing list at  http://groups.yahoo.com/group/mostsignificantchanges .

Other resources:

• MSC: A Guide to Its Use by Rick Davies and Jess Dart 

• Measuring change

• Background material 

• ODI: Drivers of change