Monitoring & Evaluation

Innovation activities are inherently characterised by uncertainty regarding the nature of socio-economic problems they are meant to address, the feasibility and effectiveness of proposed solutions, and the capacity of the actors to find and implement those solutions. In this context, innovation policy, and hence Smart Specialisation, is only sustainable if properly supported by (i) the systematic production of information regarding actual policy developments through monitoring and evaluation, and (i) the revision and adjustment of the policy according to the evidence produced by monitoring and evaluation activities.

Since 2014, Smart Specialisation represents the reference framework for innovation policy in Europe and, therefore, there is a widespread need for monitoring its development and assessing its achievements. Despite the acknowledged role and the rationale of a monitoring and evaluation system for a Smart Specialisation strategy, there is scope for improving those activities, in particular  evaluation. Indeed, evaluation is the “Cinderella” of this policy (Hegyi and Prota, 2021).

This section of the website provides an overview of the work of the S3 Platform as regards to monitoring and evaluation, with special reference to a research project run in 2020 to gain insight on the Smart Specialisation policy experience across the EU in its seventh year of implementation. The S3 Platform has already developed substantial methodological and empirical knowledge regarding monitoring which is presented in a range of supporting materials (policy briefs, reports, articles) and was used to produce training tools. The web page will be updated continuously with novel research and support materials.

A sound Smart Specialisation monitoring system “acts as early warning mechanism signalling critical aspects of policy implementation that provide inputs for S3 evaluation. S3 evaluation is only possible if there exist a clear interventions logic that links ‘ends with means’” (Gianelle et al., 2020). This requires a monitoring system in place that collects and manages accurate, complete and relevant data that drives evidence-based decision making.

While supporting the re-examination or validation of earlier policy decisions and the advancement as regards to strategic objectives, the monitoring framework enables policy makers to take better-informed decisions when determining the impact and effectiveness of a policy program. Hence, in line with the European Commission’s Smart Specialisation Implementation handbook, monitoring the implementation of innovation policies contribute to minimising duplication and fragmentation of efforts, while providing policy evaluators a basis for comparison and benchmarking of policies, and policy makers a basis for preparing for the future policy decisions (Gianelle and Kleibrink, 2015).

The JRC has identified the key aims and characteristics of a Smart Specialisation monitoring system and consolidated such knowledge into a Massive Open Online Course (MOOC) Monitoring Smart Specialisation Strategies: 

The MOOC is open to anyone interested in monitoring Smart Specialisation Strategies and knowledge-based regional development in particular. Specific audiences are:
 

• Civil servants working on regional development at the regional, national and European level
• Students or researchers interested in regional development, or research and innovation policies.
• Consultants in regional development or research and innovation policy.

The course is available for free and is organised around five modules, which cover conceptual aspects and provide examples from EU regions, both through videos and through documents.

According to Gianelle et al. (2019), “The ultimate purpose of any evaluation is to produce answers that will be useful and used for improving policy making. For this to happen, however, it is necessary to develop a culture in which policymakers demand evaluation studies not because they have to comply with some administrative requirements, but because they are genuinely interested in, and committed to, learning.” The specific features of Smart Specialisation strategies accentuate the need for exploring new approaches to evaluation (Hegyi and Prota, 2021).

In addition, the usefulness of single evaluations of the effectiveness of individual instruments is highly questionable in this multi-rationales, multi-level, multi-actor policy context. Recognizing the increasing complexity associated with innovation policies makes apparent the need for system-wide evaluation exercises that measure the effectiveness of the overall strategy. This means taking into account changes in behaviour of actors that have not been supported by policy measures included in the strategy, not only changes of behaviour in the target groups of the various interventions, as well as the presence of multiple operating and interacting interventions which create difficulties identifying the effects of one intervention over another (Hegyi and Prota, 2021).

In addition, conceptually the Entrepreneurial Discovery Process has evolved from being an element of the design phase of a smart strategy into a continuous activity. In terms of evaluation methodology, this entails moving from traditional approaches towards participatory approaches in which the focus is on supporting the learning capacity of the system by strengthening feedback loops and improving access to information. Other challenges for evaluation relate to uncertainty in the nature and timing of impacts arising from interventions and to the necessity to pay particular attention to context since economic and innovation systems are different over time and space (Hegyi and Prota, 2021).