What is monitoring?
The term monitoring usually encompasses all sorts of activities that have to do with the collection and processing of information about the degree of implementation of policy measures and the achievement of expected results. Evaluation is closely related to and builds on monitoring data. While monitoring captures if we are actually doing what we set out to do according to the planned logic of intervention, evaluation provides an ex-post validation of the intervention logic by assessing how and why policies, programmes and projects have actually had the desired effects. Monitoring does not substitute for evaluation; it rather complements it.
RIS3 monitoring focuses on tracking the developments related to policy interventions within the specific priority areas identified in the strategy. The monitoring mechanism should be able to capture and follow the relevant expected changes that are foreseen in each RIS3 priority by means of an appropriate choice of result indicators; it should also capture and follow the policy output that ought to make expected changes happen. Regions and countries with extensive experience in monitoring innovation strategies and territorial development policies in general should build on existing monitoring structures; those with limited experience in this field should begin to develop their own internal capacities and experience starting with simple indicator system. In a 2015 workshop, JRC together with Emilia-Romagna organised a peer discussion on the different elements of monitoring (PXL Feedback Report on Monitoring).
References C. Gianelle and A. Kleibrink, Monitoring Mechanisms for Smart Specialisation Strategies, S3 Policy Brief Series No. 13/2015. E. Magro and J.R. Wilson (2015), "Evaluating Territorial Strategies", in: J.M. Valdaliso and J.R. Wilson (Eds.), Strategies for Shaping Territorial Competitiveness(pp. 94–110), Oxon; New York: Routledge.
We can identify three main purposes of the RIS3 monitoring system: (i) learning about actual transformation processes and informing policy (re)actions accordingly; (ii) building and reinforcing trust and cooperation with and among stakeholders and citizens; (iii) guarantee accountability of policy making. Figure 1 summarises these elements. The monitoring system serves these purposes by performing three key functions: gathering information and making it available to decision makers; clarifying the purpose and functioning of the strategy and making it comprehensible to the broader public; supporting the constructive involvement and participation of stakeholders through transparent communication.
Monitoring as a system to gather and process information
Monitoring is first of all a process of information gathering and systematisation. The main purpose of monitoring in this respect is to enhance our understanding of the achievements that have been put in place through policy interventions. When we monitor we collect data on policy implementation (output indicators) and changes in the result variables (result indicators) that best capture the expected changes we want to bring about. In this understanding, monitoring is key for ensuring accountability of policy making vis-à-vis stakeholders and citizens. Moreover, monitoring is a pre-condition for conducting any meaningful evaluation. It may be useful to set up a monitoring system in parallel to the evaluation design, so that the right metrics required for evaluation are also collected.
We need the monitoring system in order to learn what is actually happening in the region or country with regard to our policy implementation decisions; even more so, we need the monitoring system in order to realise when actual changes are not going in the expected direction. In its original Latin meaning, the word monitor refers to a supervisor or to something that reminds or warns us. This indeed reveals how monitoring can serve as an early warning system that informs us when things are taking the wrong direction and may allow taking well-timed countermeasures to steer processes towards goals. In this understanding, monitoring allows to learn from failure before processes become irreversible. Furthermore, the wealth of information gathered through monitoring constitutes the basis for periodic refinement or refocusing of RIS3 priorities.
Monitoring as a transparent crystallisation of the logic of intervention
The monitoring system is also a way to spell out concisely the logic of intervention that lies behind the RIS3. The purpose of the monitoring system in this respect is twofold. On the one hand, laying down the monitoring system represents an opportunity for strategy designers to streamline and distil the very essence of the RIS3 logical chain that links means to ends, in this way ensuring consistency of the various elements and their appropriateness to the achievement of ultimate goals. In other words, once it is properly and fully defined, the monitoring system is a way to effectively describe the role of priorities, policy instruments and their relationship with strategic objectives. In this way, monitoring systems can help people in charge of policy implementation, stakeholders and citizens to understand the rationale of policy interventions, enabling them to constructively engage in strategy improvement and to quickly react to early warnings.
In this respect, it is useful to remember that even the most sophisticated monitoring system alone cannot allow for a complete and precise identification of the causal impact of policy interventions on the selected result indicators net of the effects of "other factors" or socio-economic dynamics that are external to the cause-effect chain linking policy measures to results.2 Impact evaluations are needed in order to properly identify the contribution of policy measures to the observed changes in the result variables. Although monitoring and evaluation are often used as one conjoined term, this note treats them separately and exclusively focuses on monitoring. We should be aware of the fact that monitoring systems are only a representation of the logic of intervention of RIS3 and thus cannot be regarded by any means as a validation of such logic.
Monitoring as a communication device
A third important role of monitoring derives directly from the previous points. A transparent monitoring system that concisely communicates the relevant information about RIS3 implementation contributes to the credibility and reputation of the ambitious transformational plan contained in the RIS3. As an ideal type, monitoring activities are organised as a continuation of the dialogue with those stakeholders that were involved during the design of the RIS3. In this function, monitoring contributes to building and maintaining dialogue and consensus. Stakeholders can either be involved in the follow-up of monitoring activities or be empowered by having access to factual information on progress made. In this way trust, ownership and commitment can be built up and maintained.
References A. Hanberger (2011), "The Real Functions of Evaluation and Response Systems", Evaluation 17(4), 327–349. H. Mintzberg (1994), The Rise and Fall of Strategic Planning: Reconceiving Roles for Planning, Plans, Planners, New York; Toronto: Free Press; Maxwell Macmillan Canada. C.F. Sabel (1993), Learning by Monitoring: The Institutions of Economic Development, Working Paper No. 102, New York: Center for Law and Economic Studies, Columbia University School of Law.
The European Commission has put strong emphasis on a more results–oriented approach for EU cohesion funding in its legislative proposals for 2014-2020. Relevant guidance documents.
04/2014 Andrés Rodríguez-Pose, Marco di Cataldo and Alessandro Rainoldi
05/2014 Åge Mariussen, Inger Midtkandal and Ruslan Rakhmatullin
13/2015 Carlo Gianelle and Alexander Kleibrink