This guide provides useful tips for the different phases of selecting and writing case studies for the SEP (2021-2027), with a specific focus on highlighting societal impact. If you would like support during this process, please make sure to reach out to us on time.
Preparing case studies for impact within SEP (2021-2027)
Written by: Lisa Burghardt & Lotte Houtepen
Date: 15.05.2022
Every six years, on a rolling basis, Dutch universities and research institutes are evaluated according to the Strategy Evaluation Protocol (SEP) developed by KNAW, NWO and UNL. Based on ESI’s experience with SEP at the Erasmus University Rotterdam as well as literature on impact evaluation, we provide guidance for selecting and writing impact case studies that are part of a unit’s self-evaluation narrative for the SEP protocol for 2021 – 2027. We start with some background information on societal relevance in SEP assessments before we share our advice. First general advice followed by recommendations tailored to the different phases within the process of preparing impact case studies for SEP.
Societal relevance in SEP
The SEP assessment focuses on the performance of the research unit in light of its own aims and strategy, including the sufficiency or appropriateness of the aims and strategy. The assessment is grounded in a narrative argument that describes the key research findings and what these imply for the overall problem and those affected by it. One of the three main assessment criteria is societal relevance, indicated with evidence for the societal relevance in terms of impact and engagement of the research unit in the narrative. Besides factual evidence (where appropriate, the unit can use quantitative indicators), the research unit should provide one or more case studies to highlight what it considers to be its most distinctive and relevant accomplishment(s). Case studies take the shape of a coherent narrative argument that contains information about the academic as well as the societal aims and output. They are pre-eminently suited to indicate how academic work is connected to societal relevance. The advice in this guide focuses on developing these case studies.
General recommendations
- Consistent impact definition: Given that societal impact can be many different things, depending e.g., on research field or across epistemic cultures, it can be challenging to clearly define what impact is. Therefore, when writing about impact in a narrative, impact needs to be described in a consistent way – consistent with what is considered impact in a specific research unit.
- Reach out on time to other parties who can help: Identifying, selecting, and analysing case studies that are representative of the unit’s impact ambitions is a time-consuming and potentially challenging exercise. The ESI team as well as stakeholders can help you in this process as long as you reach out on time.
The ESI team can help you with the selection of representative case studies and advice you on how to frame them in an impact narrative context. For us to properly support you we need to be brought in on time - meaning during the phase in which cases are being identified.
Stakeholders, either directly involved in or with a stake in the research, can help formulate impact ambitions, selecting cases and the drafting process of the impact case narratives. Our advice is to involve them wherever possible either through focus groups or by making them part of a review committee (Spaapen & van Drooge, 2011). - In line with your unit’s strategy: As part of the SEP process the research unit defines their strategic choices. In their strategy the re'search unit lays out 1) its aims and ambitions as well as 2) the plan of action to achieve these aims. The aims and plan depend on the disciplinary profile of the unit, its institutional context, the aggregation level, the recommendations of the previous assessment committee, etc. Accordingly, there is no general strategy, only a strategy specific of a research unit. To give a sense of direction for strategy formulation, the SEP protocol provides a (non-exhaustive) list of examples for scientific and/or societal aims that a research unit may pursue and its strategic plan to achieve them (see SEP protocol p. 29).
- Representativeness: Case studies can be carried out at various aggregation levels: project, programme, or unit. Whatever the exact aggregation level, the case study needs to be indicative of ‘normal’ practice of the unit. This is the case both on the level of individual researchers – is this impact the result of a conscious approach, extensive experience, or a coincidental circumstance – and on the level of a research group – are there many more examples like this, or is this the exception? Ideally the selection of case studies should represent the content and variety of the programme as a whole. Avoid cherry picking the most recognisable examples or ‘impact heroes’. This can make impact into an individualised exceptional activity. Instead, pay attention to broader conditions for impact: groups of engaged researchers, support staff and arrangements that provide impact support and recognition.
Phase 1 (~ 10 - 8 months prior to site visit): How to identify relevant impact case studies?
The case studies should exemplify how the unit works on (or aims for) impact in practice. Ideally, a unit identifies several cases that are in line with their strategy, before making a case study selection that together represents their entire strategy. Aspects to consider are:
- Nature of the initiative: are there certain impact activities that are especially relevant for your field and that represent impact for your unit?
- Pathways to impact: are there certain routes the unit follows more frequently? If approached from a Theory of Change (ToC) perspective, this entails describing research inputs and activities, subsequent outputs and outcomes, as well as (achieved or potential) societal impact. Such descriptions of impact pathways allow to illustrate “how and why a desired change is expected to occur in a specific problem context” (Belcher et al., 2020).
- Important stakeholders: are certain (longstanding) collaborations or groups key for your academic work?
- Broader conditions for impact: are there groups of engaged researchers, support staff and arrangements that provide impact support and recognition? What policy and practical measures have been taken to create the right conditions within the research unit? Although researchers’ motivation is of course pivotal to effective impact initiatives, a narrative and/or strategy should not rely merely on examples of outstanding individuals. In such a case it usually remains vague how the case relates to the ambitions, conditions, and capabilities of a research unit, which is a central tenet in the SEP evaluation.
Since representativeness is not self-evident, make sure you are transparent about case selection for impact narratives, and you are able to explain and reflect in the self-evaluation narrative on the criteria that have been used.
Phase 2 (~ 8 to 4 months prior to site visit): How to write an impact case study?
Case studies illustrate how the unit works on (or aims for) impact in practice. Usually, case studies are a short summary (about one page) with a description of a project in a narrative form that can be clearly understood by the general public (as the case studies will be made publicly available as part of the quality assurance cycle). Aspects to consider are:
- Nature of the initiative: Set the context for the case study and - where possible - why the activities executed are a good representation for the impact activities in your unit. This should also give insight to why the specific case is a relevant case study to refer to.
- Pathways to impact: Outlining these pathways can help make assumptions and processes explicit. For instance, a common mistake in impact narratives is that relations with actors are described as ‘just being there’. A ToC approach can provide a useful framework for describing these impact pathways and make these underlying assumptions explicit. A ToC based impact analysis usually includes a description of:
- the overarching research aims
- activities that were executed (e.g., project design, fieldwork, communication or engagement with relevant stakeholders, participatory activities)
- research outputs: products, goods, or services that derived from the research or research process (e.g., knowledge or processes generated by the activities)
- research outcomes: any changes in knowledge, attitudes, skills, and relationships that manifested as changes in behaviour, this can also be described per specific actor group - if applicable
- Stakeholders: Include descriptions of productive interactions with your stakeholders, instances of research activities such as co-creation or stakeholder consultations or use of research and/or societal changes. This concerns both descriptions of direct interactions – meetings, informal contacts, board memberships etc. – and indirect interactions – i.e., financial transactions and resources for research.
- Broader conditions for impact: Make the nature of your initiative clear: was it part of a strategy, an outcome of a long-standing collaboration, an experimental attempt, or a coincidental event? Avoid emphasis on ‘impact heroes’. This can make impact into an individualised and exceptional activity. Instead, pay due to broader conditions for impact: groups of engaged researchers, support staff and arrangements that provide impact support and recognition. Also include concrete references to how an impact environment and capacity for researchers to achieve societal impact is created (e.g., to find and collaborate with societal partners; to disseminate findings to professional audiences).
Structure
- Title: What the research is about, or what the outcome of a certain project or programme was.
- Introduction: Start with the topic you are working on and how the scientific work connects to society. Include the impact area(s) that the research aims to contribute to (e.g., policy impact, environmental impact, economic impact, etc.). This should be connected to describing who cares or benefits from this research; what would go wrong in society without this research; what values or issues it responds to; etc. Then describe how the actors involved came to guide the research, i.e., a concise description of the interactions/relations that boosted the project. It needs to be made clear how relations with external actors/productive interactions come about.
- Main body: Describe whether actual benefits or societal change occurred or, given it might take years to identify change, describe potential change. For experimental or fundamental research, it may be a contribution to a knowledge chain. While doing this, include how the societal and scientific achievements relate to one another. Describe the research outputs and the kind of effect the outputs had. Ideally, the narrative relates to impact pathways (as described earlier) that have been identified/described also in the more general and strategic sections of the report. These can be supported by highlighting indicators of use and recognition of the research by societal actors and researchers from other disciplines. In this section the narrative could layout who heard the research findings and how, as well as what changed and who benefitted from the change. While writing this section make explicit how your work was enabled by broader conditions for impact present in your research unit (e.g. support dissemination, reward certain aspects).
- Conclusion: Sum up the effects on the research group and how it relates to the existing and future strategy: Will you continue in this direction? Will you sustain the relations with actors? Will you develop or update your impact ambition to accomplish changes that had not yet been achieved?
Phase 3 (~ 4 to 2 months prior to site visit): How to reference the impact case studies in the self-evaluation narrative?
The impact case studies exemplify and provide evidence for a unit’s societal relevance, particularly for the impact section within the SEP self-evaluation narrative. Common elements should be emphasised, especially when it comes to underlying evidence for pathways to impact, important stakeholders, and conditions for impact. In the overarching self-evaluation narrative, it is essential to put these elements into the bigger picture of the unit’s impact ambitions. For example, are the research activities in the case study representative of research activities executed by the unit? Are the stakeholders that were involved in the case study or had a stake in it involved more often or are there mechanisms in place to foster long term relationships with these stakeholders? These are elements that can already be part of the impact case study and should be highlighted here again. Ideally, the societal impact indicators of the unit’s societal relevance tie in with the processes and indicators described in case studies.
Keep in mind: the impact case studies are supposed to function as evidence for a unit’s societal relevance. They bring your impact story to life and provide practical examples to illustrate how a research unit’s strategy is turned from words into action.