08:45 - 09:00 Welcome Coffee and registration

09:00 – 09:15 – Introduction to the day

Mark Boden (JRC): Opening remarks

Elisabetta Marinelli (JRC): Introduction to the day  

09:15-10:15 Open Data and S3 Monitoring – New cases, new issues

Nicola de Chiara (Agid, Agenzia per l’Italia digitale): The ONCE observatory (20 mins)

Ismael Rafols (Ingenio, Spain): On the potential effects of monitoring systems in STI policy (20 mins)

Q&A (20 mins)

10:15- 11:30 Participatory exercise – part 1

• Country-specific issues in open data for monitoring

Country-Specific table discussions:

With the help of the moderator the working tables will discuss the following aspects, based on the exercises just conducted.

• For each data-source
  • Potential for S3 analysis and monitoring
  • Main obstacles encountered to access / analyse the data source

 Accessing the data

 Managing / analysing the data

• Data granularity requirements
• Data openness

 Timely

 Accessible / Machine processable / Interoperable

 Findable

 Licensed

• Potential improvements and paths forward (e.g. Best practice exchanges between administrations and data platforms // Useful dimensions/data currently not accessible that  could be made public)
• choose the best example to report exercise 2

11:30 - 12:00 Coffee Break

12:00 -13:00 Participatory exercise – part 2

3 working groups will be organised, ensuring a good representation of different countries.

• Peer sharing - short oral presentation of 2-3 best practices (by the moderator/rapporteur)
• Comments on the two EU-level initiatives (relevance for S3 monitoring and data openness)
• (If there is time) Discussion on the potential effects of monitoring systems in STI policy

13:00 - 13:30 Reporting back and discussion

• All the rapporteurs of the different groups report back to open-up a discussion

13:30 - 13:40 Proposal from our Critical Friend Attila Varga: Seeking for regions to pilot S3 evaluation.

13:40 – 14:30 Lunch break

14:30 – 15:15 Real-life experiences with S3 evaluation

• Ken Guy – What do you think would happen with a ToR like the one of Castilla y Leon?
• Gregorio Muñoz Abad, Region of Castilla y Leon – What actually happened with the ToR of Castilla Leon

15:15 – 16:00

• Commenting examples of S3 mid-term evaluation ToRs

ToR elements: Are the following aspects clear? Are they appropriate?

• General information:
  • the background, rationale and objectives of the policy to be evaluated, its target recipients, delivery method and intended outcomes;
  • the audience and intended use of the evaluation;

•  Technical information:
  • the extent of the existing evidence base related to the policy;
  •   the evaluation objectives and research questions;
  • the possible evaluation approach, research design and methods;
  • the required capabilities, skills and experience of the proposed evaluation and team;

• Outputs and deliverables
  • the required evaluation outputs (including datasets) and the milestones to be met;
  • data archiving requirements;

• Organisational and Economic information
  • the indicative budget;
  • the evaluation timetable

Conceptual reflections:

• OP vs S3: To what extent this OP evaluation can inform the RIS3 mid-term review?
• OP vs S3: Are the OP evaluation questions sufficient to evaluate a RIS3? What is missing?
• Discuss the trade-off between overspecification and bidding on the price vs more degrees of freedom in proposing a methodology and bidding on price/performance.
• To conclude, what would you expect from those ToR?

• 16:00 – 16:30  - Reporting back + comments by Ken Guy

16:30  - 16:45 Closing remarks

Mark Boden (JRC): Conclusions from the day

Elisabetta Marinelli (JRC): Next steps