Impact indicators

Written by: Hedi Westerduin

Date: 02-06-2022

Increasingly academics are asked to indicate how their academic work relates to society.[1] Whether via formal evaluation procedures, such as reports to funders or the Strategy Evaluation Protocol (SEP)[2], or as part of the grant application process. Societal impact evaluation can help demonstrate, describe, and share the learning and value we are creating at Erasmus University Rotterdam (EUR). The Evaluating Societal Impact (ESI) project not only aims to develop methods to evaluate the societal impact of our EUR core activities, but also to foster an understanding of why, when and how to use impact evaluation. This understanding is particularly important when it comes to selecting and applying indicators for impact evaluation, since indicators, and especially quantitative indicators, have been a widely discussed topic (Hicks et al., 2015). Based on literature about using impact indicators and our experiences at the EUR, we provide guidance in selecting impact indicators from a (non-exhaustive) list of impact indicators. First, we share general advice about why, when and how to use indicators, before we provide instructions on how to use the (non-exhaustive) list of impact indicators that we curated and provide as a pdf at the bottom of this page. This is an emergent, ever evolving, collaborative document – the intention is to further refine and develop this in co-creation with academics at the Erasmus University Rotterdam. So, if you develop or discover useful indicators for societal impact evaluation or have questions about the current content do let us know.

Why use impact indicators?

Indicators are intended to help monitor outputs and outcomes in relation to an intended societal impact. Some indicators are quantifiable, such as citations in policy papers, prizes or grants. Impact frameworks are also increasingly acknowledging the importance of qualitative indicators for societal value creation that are often less easy to count (e.g. collaborative endeavours, long-term relationships and productive interactions with non-academic stakeholders as pathways to creating impact) (Ludwig Boltzman Gesellschaft, 2021). It is important to note that indicators are never stand-alone pieces of evidence. Only when selected and used in the context of a specific impact objective, can indicators help collect tangible and measurable evidence. If applied in this manner, they can be a powerful piece of evidence for impact.

When to use impact indicators?

Indicators are context dependent, meaning there is no 'one-size-fits-all' indicator.[3] At the same time they can (and are) used in various settings with different intentions and timings. For instance, ex post to evaluate a portfolio of work (e.g. SEP evaluation) or on a project specific basis (e.g. grants). Or ex ante to set up a project with impact monitoring in place or develop a monitoring system for a project portfolio. The diversity of use cases underlines the importance to consider with what intention impact indicators are used in what context.

How to use impact indicators

Our key piece of advice about indicators is to only use them in the context for which they were selected with a well specified impact ambition and aim in place. When evaluating a portfolio of work (as happens in SEP), ideally the indicators are representative for the unit’s impact ambition and strategy by tying in with their preferred processes, stakeholders and impact activities (more information specifically on SEP in our website piece ‘Preparing case studies for impact within SEP 2021-2027’). When used for an individual project we advise to first formulate a Theory of Change (ToC). In the process of formulating a ToC you gain a better understanding of how the change can take place and a better grasp on planning, monitoring and evaluation. You also critically assess your own assumptions about the change process (more information in our website piece ‘Theory of Change’). Moreover, ToC also comes back in several grant guidelines, for instance in EU’s knowledge valorisation for horizon guide and NWO’s impact plan approach.

How to use the (non-exhaustive) list of impact indicators

Many potential indicators exist and several excellent resources already curated lists of indicators.[4] Our aim is not to supersede these lists, but to add an interpretive layer to aid selection in line with what we recommend as ESI. Our list of impact indicators has the character of a selection menu: a non-exhaustive, broad summary of indicators from which academics or research units choose, depending on their specific profile and in line with their own impact ambition and strategy. It offers indicators covering a broad scope of valorization. The way we formatted the table is in line with the approach we recommend.

First start with considering your overall impact ambition – the societal change you are hoping to achieve. Perhaps relate this to a specific thematic type of impact, this allows you to consider more context specific indicators. To help with this we have included tables specifically suitable for a certain type of impact (see Tables 2-8). As an additional lens to impact indicators, we have presented the first table for general indicators (that can be useful regardless of type of impact) by type of change from the Flows of Knowledge evaluation framework (Edwards & Meagher, 2020). Note that different types of change can also be found more implicitly within the thematic types of impact tables.
Second, in light of your mission formulate what type of outcomes you hope to achieve. Outcomes concern medium-term results and usually have a clear relationship with the purpose of the academic project, they can be understood as intermediate steps towards longer-term impacts. In tables we have provided examples of outcomes.
Third, think about the kind of outputs that are necessary to achieve your mission. Output comprises the most immediate results of an academic project. Therefore, you should aim at formulating countable and tangible outputs which the academic work should produce. In tables we have provided examples of outputs. Again, depending on the kind of change you are aiming to achieve, the required outputs by the project might vary.

We deliberately did not specify indicators for activities, as there could be a wide range of activities contributing to outputs and/or outcomes. Activities are the hands-on practical activities that your project can do in order to reach the desired change. One could think of research, education and engagement activities.

By working your way from an outcome level to an activities level, you always have the bigger picture of your project in mind and can therefore make more strategic decisions about the activities and outputs needed throughout the process.

To get a sense of how different types of impact can be evidenced, have a look at the documents below.

This document contains a combined selection of impact indicators from VSNU Raamwerk Valorisatie Indicatoren (2013), Strategy Evaluation Protocol (2021-2027)[6], KNAW Maatschappelijke Impact in Kaart (2018)[7], ZonMw Impact Assesment Framework (2021)[8], Campus Engage Framework (2018)[9] and Research Excellence Framework (2021)[10]

References

Edwards, D. M., & Meagher, L. R. (2020). A framework to evaluate the impacts of research on policy and practice: A forestry pilot study. Forest Policy and Economics, 114, 101975.

Hicks, D., Wouters, P., Waltman, L. et al. (2015) Bibliometrics: The Leiden Manifesto for research metrics. Nature 520, 429–431. https://doi.org/10.1038/520429a

Ludwig Boltzman Gesellschaft, (2021). “What is societal impact of research?” Literature Review: https://ois.lbg.ac.at/wp-content/uploads/sites/24/2022/01/Literature-Review_Societal-Research-Impact.pdf