Monitoring and evaluation of a transition project or programme is not business as usual. This is connected with the ambition of transitions and the special features of transitions, which not only raise questions about what you could or should monitor or evaluate but also affect the objective of the monitoring and evaluation.
The special features of transitions are:
Learning and adjusting
Because of the uncertain, complex and dynamic nature of transitions, it might be necessary to revise your point of departure and approach, or even your actual objective, during the project or programme. The monitoring and evaluation of a transition programme serves that goal. The normative choice then is to use monitoring and evaluation in order to keep the project or programme focused as closely as possible on the ambitions for the system innovation.
Reflexive monitoring is:
Three types of activity
Reflexive monitoring involves a series of activities: observation and analysis, reflection and adjustment and/or future development (see also figure below).
Observation and analysis
This could include observation and analysis of:
For methods of observation and analysis, see also the database ‘Methods'.
Reflection
What follows is a critical reflection on the observations and analyses. Essential questions include:
This process calls for participants to reflect critically on their own (theoretical) principles, deeper convictions, values (the obvious ‘mental models'). This is particularly important when one is encountering barriers to the system innovation. See also ‘About transitions'.
Adaptation
The reflection process may make it necessary to revise the objectives, target groups, strategies or activities of the project or programme. It might also lead to the development of follow-up projects or programmes.
Ways in which reflexive monitoring differs from ‘traditional' monitoring include the following:
Important objectives of reflexive monitoring are to learn from practical experience and to apply the findings in a project or programme for system innovation. But there are also other objectives of monitoring and evaluation that you may have to meet. The following list presents the different objectives and their consequences for the use of the method and the target groups.
Objective: learning
To meet this objective you collect information about the progress of your project or programme, about the intended or unintended effects, about developments in the environment and about obstacles that you encounter. You use this information to reflect on your points of departure and your assumptions about the problem or about the possible solutions or the knowledge needed to find solutions. This might lead to the conclusion that you have to revise your automatic assumptions.
Target group: In terms of learning, an important target group are those who are directly involved in carrying out the project or programme. Other important target groups are people engaged in related experiments, managers and financiers. Policymakers should be involved in the process, particularly when reflecting on institutional barriers.
Objective: accountability
The question of accountability mainly concerns whether you have achieved the goals that have been set and have made effective use of the resources provided.
Target group: In this case, the target group is usually the client or financier of a project or programme. In the case of national, government-financed programmes it is the government and parliament.
Objective: intervention or adaptation
Here the objective of the monitoring and evaluation is to decide whether the transition project or programme needs to be modified, for example in terms of the activities or target groups. This step is an integral part of reflexive monitoring.
Target group: Where adaptation is concerned, the most important target groups in reflexive monitoring are stakeholders and the programme management. But interventions or adaptations can also arise from the conclusions reached by clients/financiers as to whether the money has been properly spent, in which case the client and the programme management are the most important target groups.
Objective: generating enthusiasm and transferring knowledge
Creating enthusiasm and transferring knowledge can contribute to the legitimacy of system innovations and help increase support for anchoring them in an organisation.
Target group: The target group consists of the client/financiers and potential or existing fellow innovators, possible future financiers and insurers, regulators or a wider public.
Not automatically. There is often a conflict between monitoring and evaluation for the purposes of learning and adaptation (reflexive monitoring) and monitoring and evaluation for accountability purposes. Aspects in which this is reflected include:
Flexibility versus fixed targets and results. Accountability brings pressure to adhere to rigid, predefined results and to maintain control over a project or programme. This can limit the scope for making changes in the project. In that sense there is a contradiction between the learning and adaptation objectives of reflexive monitoring and the objective of accountability.
Transparency versus 'only the good news'. The learning objective of reflexive monitoring calls for transparency about ‘failures' and unexpected results. For the purposes of accountability one may focus solely on the successes.
Details versus the big picture. The learning objective also often calls for detailed information about the approach taken and the progress of the project or programme. For accountability purposes, it is usually better to provide general information about how the budget is spent, the planning and the results. With a limited budget, this can also create tension.
Accordingly, and also because of the different questions that arise in relation to the different objectives, it is not always easy to combine the learning objective with accountability.
Need for consultation
The programme owner and the client should therefore consult on the objectives of the monitoring and evaluation and indicators of progress. It is sometimes possible to formulate sets of questions and indicators that can serve both the objective of learning and adaptation and the objective of accountability. If not, a good strategy is to start with monitoring and evaluation that provides a clear impression of the progress and results of the programme, in other words for learning purposes. On the basis of that information you can start additional monitoring and evaluation for accountability purposes.
Stakeholders should preferably participate in the reflexive monitoring. There are two connected reasons for this:
It is important to remember, however, that participation is more than just providing information for a report. It also involves taking part in system or actor analyses or in reflection sessions.
Who do you involve?
First and foremost, involve those who are directly involved in carrying out the experiment. Other important target groups are managers, financiers, policymakers and other stakeholders who will play a role in the innovation. Innovators from related programmes could also be relevant for the monitoring. It might be necessary to perform a systematic analysis in order to identify all the relevant actors. See also actor and network analysis under ‘Methods'.
Ultimately, you will have to think carefully about who you choose to participate, when and for which issues. The choice should be determined in part by the long-term ambitions: which parties ought to be involved in the learning process and what do they need to learn?
In principle, it is the reflexive monitor's task to keep reminding participants of the ambitions for system innovation. It is up to the project or programme manager to maintain the balance between those ambitions and other interests.
As project manager, you should make agreements with the monitor about your respective roles early on. Bear in mind any objectives in terms of accountability that the client has given (see also the question ‘Can learning be combined with accountability?'). Be sure that you agree on the objectives of the monitoring and about the activities themselves.
Interventions
The role of the reflexive monitor means that he or she will make interventions in the project or programme. For example, the monitor might confront participants in an experiment who are still thinking in terms of the obvious or might question the course of a project or programme in light of the conclusions from the monitoring. In principle, the monitor's interventions will be based on the project or programme's ambitions for system innovation.
We distinguish four possible tasks for a monitor:
1. To help in performing actor analyses
2. To help in performing system and trend analyses
3. To make suggestions for specific group interventions
4. To help with registration
This refers, for example, to the contributions to data files on changes in practices and regimes or dynamic learning agendas so that what has been learned can be transferred to third parties.
The four tasks apply to the monitoring and evaluation of both transition projects and programmes, although the scope of the analytical contributions is wider in the case of monitoring and evaluation for a programme than for projects. Furthermore, the various analyses for the monitoring and evaluation of a programme are often less detailed.
Although reflexive monitoring work is very similar to ‘regular' transition work, many project or programme managers use an external monitor who has experience with monitoring and evaluation of transitions. The advantage of using an external monitor is that he or she can bring a fresh perspective and may therefore be in a better position to see established thought patterns that could constitute a barrier to system innovation. A third advantage is that the external monitor and the programme or project leader have clearly distinct responsibilities.
The monitoring of a transition programme has to be tailored to the specific circumstances. The priorities among different goals and target groups determine the precise focus of the monitoring. But what you will monitor also depends on the actual project or programme and its environment and on ideas (hypotheses) about the spill-over effects of the transition. It is therefore impossible to describe the questions you should ask or the indicators you should use in general terms.
An additional factor is that the monitoring and evaluation of system innovation projects or programmes is usually not confined to the instruments of reflexive monitoring. It may be necessary and useful to employ more traditional forms of monitoring and evaluation. On this point, see also: 'What are in general objectives of monitoring and evaluation?'
The following steps can be taken to determine where your focus will lie:
1) Start by clarifying the objectives, the questions to be addressed and the target groups for the monitoring and evaluation. This will give you a first impression of where you should focus.
2) Specify the future vision envisaged by the project or programme.
3) Translate the choices you make into indicators.
Indicators
There are various types of indicators. Below is are examples. The appropriate indicators for you are determined in part by the answers you have given to the above questions.
Process indicators
Early signs of practices representing a system innovation
More traditional indicators of sustainability
Objectives and questions that are difficult to reconcile
During the monitoring and evaluation you may be confronted with objectives and questions that are difficult to reconcile. You might therefore be required to consult the client again about further choices and/or about how different monitoring and evaluation activities can be carried out over time. On this point, see also ' Can learning be reconciled with accountability?'