This is an introduction to Flows of Knowledge as a method for impact assessment. At the bottom of the page, you can find a case study (including a downloadable report) describing how ESI has made use of this method to conduct an impact assessment and our reflections.
Flows of Knowledge
Written by: Margriet Kim Nguyen & Jorrit Smit
Date: 30.05.2022
What is Flows of Knowledge?
Flows of Knowledge (Flows) is a method to assess the policy and practice impacts of scientific research. The method was developed (and updated) based on extensive professional experience in the impact evaluation field in the UK (Meagher et al). As an approach to impact case studies Flows is relatively versatile: the main point is to map qualitatively and, if relevant, quantitatively the different types of impact achieved, specify who were involved, and what causal factors were facilitators or barriers for this change to take place. These conceptual ‘building blocks’ can be combined in many different ways, which allows rich descriptions of the specificities of an impact case.
In the most recent version, described in Edwards & Meagher (2020), the three core questions are
(1) what changed, (2) why and (3) which lessons can be learned? These questions are answered in a narrative, based on information of impacts, actors and causes. To write the narrative, the information is analysed according to a framework depicted in table 1. Flows distinguishes five impact types, five stakeholder categories and eight causal factors that can result in, or risk, the generation of impact. Flows includes a set of building blocks that can be adapted to various situations.
Parts | Core Evaluation Questions |
1A. IMPACTS | What changed? |
Instrumental: Changes to plans, decisions, behaviour, practice, actions and policy | |
Conceptual: Changes to knowledge, awareness, attitudes, emotions | |
Capacity-building: Changes to skills and expertise | |
Enduring connectivity: Changes to number and quality of relationships and trust | |
Culture/ attitude towards knowledge exchange, and research impact itself | |
1B. ACTORS | Who changed? (Influencers and influenced) |
Policymakers: including regulatory bodies; local, national and international | |
Practitioners: public, private, NGOs | |
Communities: of place or interest, general public | |
Researchers: within and beyond the project and institution | |
Other: | |
2. CAUSAL FACTORS
| Why/ how did changes occur? |
Problem-framing: Level of importance; tractability of the problem; active negotiation of research questions; appropriateness of research design. | |
Research Management: Research culture; integration between disciplines and teams; promotion of research services; planning; strategy. | |
Inputs: Funding; staff capacity and turnover; legacy of previous work; access to equipment and resources. | |
Outputs: Quality and usefulness of content; appropriate format. | |
Dissemination: Targeted and efficient delivery of outputs to users and other audiences. | |
Engagement: Level and quality of interaction with users and other stakeholders; co-production of knowledge; collaboration during design, dissemination and uptakes of outputs. | |
Users: Influence of knowledge intermediaries. | |
Context: Social, political, economic, climate and geographical factors. | |
3. LESSONS LEARNED, FUTURE | Lessons learned for impact identification and generation
|
Table 1 Research impact evaluation framework (Edwards & Maegher, 2020)
Why should it be used (or why not)?
Multidirectional Flows of knowledge across a web of networks and relationships are the processes that make generating societal impact possible. Flows provides the conceptual building blocks to structure (mostly qualitative) data collection and narrative construction. The framework for impact assessment is flexible as there is a lot of freedom when using it (as shown in our case study in which we added proximities for further definition). It is a versatile method in intention, set-up and interpretation. It was explicitly intended to move beyond rigid logic models that suggest unambiguous causality, sequence and even linearity in the process of generating impact. Therefore. this method pays attention to ‘more subtle’ impacts like capacities, culture and connectivity. In this way, the method stimulates users to pay attention not only to outputs and outcomes, but also to processes of relationship-building (which are often the conditions for future impact). A remarkable feature is the focus on the importance of intermediaries in processes that lead to research impact. This relationship-building aspect can be a powerful motivation for using Flows.
In terms of set-up Flows offers different building blocks that can be tailored to local needs and impact expectations. This means it is a suitable method for different types of assessments, for example of a funding program, a specific project, or a department. Overall, the framework helps to reveal underlying reasons for impact within the project's context, informing steps that researchers and stakeholders could take in the future. Therefore, it supports internal learning and external communication. The building block approach of Flows of Knowledge might also allow actors to continue keeping track of their impact and its causes also after a first ex-post assessment.
The method does require some time investment as the identification of the type of impacts, causal factors and actors involved gives a skeleton for highlighting the knowledge Flows. The narrative still needs to be constructed from this skeleton which requires time and further reflection. The fact that this approach can be used on different scopes and levels is, on the one hand, its greatest asset; on the other hand, it also expects a proactive attitude and a high level of impact literacy of those tasked with the assessment. From a methodological point of view, the impact enduring connectivity (changes to number and quality of relationships) can overlap with the causal factor engagement (level and quality of interaction with users and other stakeholders). Therefore, one needs to carefully distinguish the types of interactions within the knowledge Flows. Nevertheless, Edwards & Meagher (2020) mentioned that enduring connectivity does not need to be seen as a full impact itself as it can led (or can have the potential to lead to) other forms of impact.
When should it be used?
When the main goal of assessment is formative, Flows can be a good fit. That is, when assessment is done to enable learning, reflection, improvement, and communication about impact. Although the authors note that the conceptual framework can incidentally also be used for external accountability purposes. In addition, it is geared at assessing impact ex-post: after a research or collaboration project has finished. Nevertheless, it can also be used ex ante to plan for desired outcomes which then requires an evaluation ex post.
How can it be used?
Impact assessment with Flows relies first and foremost on qualitative data, where relevant supplemented by quantitative data. A guiding principle is to gather input from multiple stakeholders and data sources. The conceptual framework can be used as a guide for streamlining data collection and interpretation, as well as narrative construction. The choice for which qualitative, and potentially, quantitative methods to use depends on the scope of the program or project to be evaluated. For larger scale cases: surveys, project documents, self-evaluations & preparatory workshops, bibliometrics. For specific case studies: focus groups and interviews in order to obtain an in-depth rich description to identify the Flows as well as addressing non-conscious changes. This was the case within our case study as the emphasis of Flows lies on 'change' and 'effects' in interview questions/evaluation framework, non-conscious changes may be more difficult to address. These often appear in interviews only later, in passing (and perhaps not at all in self-reporting). In the case of evaluating larger programmes or departments, the authors advise to supplement more general data collection with specific case studies.
The data can be used together with the conceptual framework to build an impact narrative. Importantly, this can be done by the analysts (see our example case study, for which we used content analysis and semi-structured interviews) but also by involved actors themselves (see Edwards & Meagher, 2020, who employed a combination of self-evaluation and content analysis).
As mentioned before, this method pays attention to ‘more subtle’ impacts and stimulates users to pay attention not only to outputs and outcomes, but also to processes of relationship-building (which are often the conditions for future impact). The different building blocks of Flows can be used for different types of assessments and allow actors to continue keeping track of their impact and its causes. A necessary condition for applying the Flow is time investment, a proactive attitude, and a basic level of impact literacy.
What is the outcome?
The main outcome of an assessment with Flows is a narrative. If desired this narrative can be supported with a selection of indicators identified by stakeholders during the data collection process. Ultimately, this narrative should be shaped to enable actors to learn about what changed and why (not). The narrative can be part of a bigger report. In such a case the qualitative findings, in the shape of quotes and tables, can give a sense and overview of the various Flows, impacts and actors. One additional outcome that could be produced is an overview of coding frequencies, based on the content analysis by the analysts, to provide a more quantitative sense of what impacts, actors and causal factors were most important in the discussed cases.
What is a ‘knowledge flow’ and who is involved in an assessment using this method?
Originally, the terminology of ‘Flows of knowledge, expertise and influence’ was introduced to reflect a certain understanding of societal impact of scientific research:
Research impact [is] a function of the interaction between the content of the research, the context for its application, and the processes of user engagement, with those processes including multidirectional Flows of knowledge, expertise, and influence across a web of networks and relationships (Meagher et al, 2008; Meagher & Martin, 2017)
With ‘flows’ the focus is thus primary on processes of interaction between research users and producers, and the factors that influence this. This also means that ideally not only researchers (and the analysts) are involved, but also actors from policy, practice and intermediary bodies that play a role in the cases at hand. Nevertheless, from our case study became apparent that using an external evaluator created a certain distance towards the research project in which one could take a more ‘objective’ stand. Once drafting the flows and narrative, feedback was required from the interviewees to reflect upon the flows which created an iterative learning process.
Take a look at an example where ESI researchers applied Flows of Knowledge in the case study at the bottom of this page.
Literature on Flows of Knowledge
Edwards, D. M., & Meagher, L. R. (2020). A framework to evaluate the impacts of research on policy and practice: A forestry pilot study. Forest Policy and Economics, 114, 101975. https://doi.org/10.1016/j.forpol.2019.101975
Meagher, L.R., & Martin, U. (2017). Slightly dirty maths: The richly textured mechanisms of impact, Research Evaluation, 26(1), 15–27. https://doi.org/10.1093/reseval/rvw024
Meagher, L., Lyall, C., & Nutley, S. (2008). Flows of knowledge, expertise and influence: A method for assessing policy and practice impacts from social science research. Research Evaluation, 17(3), 163–173. https://doi.org/10.3152/095820208X331720
Laura R. Meagher, Ursula Martin, Slightly dirty maths: The richly textured mechanisms of impact, Research Evaluation, Volume 26, Issue 1, January 2017, Pages 15–27,
Improving crew planning at Dutch railways - Flows of Knowledge as an impact evaluation tool
Written by: Margriet Kim Nguyen, Jorrit Smit
Date: 30.05.2022
Duration: Project Impact Assessment (PIA) took place between June 2021 and March 2022; the PhD project was concluded in 2020, the collaboration between NS and EUR continues.
Stakeholders: Econometrics Institute at Erasmus School of Economy of Erasmus University Rotterdam (ESE); innovation department at national railways (P&I, NS); Evaluating Societal Impact
ESI Researchers: Margriet Kim Nguyen & Jorrit Smit
In this case study on using algorithms to improve the fairness, efficiency, and quality of crew planning for the national railways, we have used the Flows of Knowledge (Flows) approach as introduced by Meagher, Lyall & Nutley (2008) and described in Edwards & Meagher (2020).
Flows moves beyond a narrow interpretation of impact in the instrumental sense (direct, tangible effects of research on practice or policy). Rather, this method pays attention to ‘more subtle’ impacts like capacities, culture, and connectivity. In this way, the method stimulates users to pay attention not only to outputs and outcomes, but also to processes of relationship-building (which are often the conditions for future impact). Our experience and impressions are summarized below.
An example of Flows of Knowledge ex post: Improving fairness and quality of crew planning
ESI used Flows to assess how a collaboration between econometricians and innovation experts at the NS contributed to changes in the process of crew planning. There is a long-running collaboration between the NS and ESE; for over twenty years there is intensive contact, and the NS has funded ESE research into rail-related mobility and logistics on a structural basis. This collaboration consists of a special chair in Public Transport Optimalisation, a professor who also works as innovation expert at the NS, and regular funding of ESE PhD research. Both the chair and PhD candidates are physically present at the EUR and NS offices. This case study is about one PhD trajectory that focused on improving crew planning in terms of quality and ‘fairness’ for train drivers and conductors. The PhD researcher developed, from theoretical principles, an algorithm that could produce various ‘good’ and ‘fair’ schedules. Subsequently, the algorithm was further developed into a prototype, at the NS, and tested during a pilot-phase in the practice of actual scheduling committees.
What did the assessment consist of?
Impact assessment with Flows of Knowledge relies first and foremost on qualitative data. ESI studied available documents of the project and conducted five in-depth interviews with stakeholders from the EUR and the NS. The interviews were coded on the basis of the Flows conceptual framework to distinguish different types of impact, involved stakeholders and causal factors that made changes possible. The ESI team also paid attention to the different dimensions of proximity that enabled collaboration and exchange to occur. Ultimately, this allowed us to reconstruct in narrative form a complex whole of ‘knowledge flows’ between the actors from the university, the innovation department and the other NS units around the algorithm.
What was the result of the Flows of Knowledge analysis?
This impact assessment, guided by Flows, resulted in a report. It contains a narrative that summarizes the way knowledge ‘flowed’ between different actors and how this led to changes in the practice of making work schedules.
Besides a narrative, the report contains descriptions of the ways in which the project did or did not yet realize five different types of impacts. For each impact, the involved actors are identified and the causal factors that supported (+) or obstructed (-) the generation of lasting change (see table 7, p.13/14 in report). We added another interpretive layer by associating these causal factors with five different dimensions of proximity (Boschma, 2005) (see table 9 or in report table 9). This concept from innovation literature is not a requirement of a Flows-analysis but was relevant to include as in this case the focus was on a (long-term) collaboration between science, innovation, and practice. The flexibility of the Flows approach made it possible to add more analytic tools that were relevant for our understanding and to promote learning.
Ultimately, the Flows analysis shed light on ‘improved crew planning’ which is an example of bottom-up initiated research and structural science-practice collaboration. The question how to optimize the process of constructing crew schedules was initiated ‘bottom up’ and reached the researcher via the intermediary innovation department of the train company. The physical presence of the PhD researcher at the NS office greatly contributed to understanding of the problems, knowledge of the end-users and the specific needs that the algorithm had to respond to. The intermediary co-workers at the innovation department were able to translate the theoretical work into a practically applicable system through a co-creative process with end-users. This resulted in partial adoption of the applicable system: one user group accepted the solution, the other did not fully accept the solution.
Several implementation barriers to implement the outcome within the railway organisation on a national scale became visible, for various technical but mainly organizational reasons. The outcome of the research however intensified social networks within and outside the railway organisation which resulted in further knowledge dissemination and led to changed attitudes regarding the use of algorithms for staff planning and capacities within the organisation.
Please find the report ‘Eerlijke Personeelplanning op het spoor 2022’ below as a downloadable file.