Table 3 Strategies to action evaluation for learning and sustainability
From: Evaluating for learning and sustainability (ELS) framework: a realist synthesis
Context | Mechanism | Associated Action Strategies |
|---|---|---|
Program Theory Domain: Organizational Preconditions | ||
Tolerance of risk and failure in an organization | Staff’s fear of failure | Embracing Risk & Failure - Unite the leadership team in the goal of learning as the most important outcome of evaluation to avoid surface engagement and performative assessment by those involved in the work of innovation evaluation. This may mean getting comfortable with the fact that evaluation results may publicly expose organizational innovation challenges or failures - Establish the leadership team as a guiding coalition to support the evaluation by providing endorsement and visibility throughout the organization - Present the leadership’s united learning vision to staff involved in evaluation. Invite open dialogue about the tolerance of the leadership and organization to exposing innovation challenges and failures for the purposes of learning - Emphasize that evaluations are assessments of innovations, not individuals or teams - Celebrate failures as learning opportunities; do not associate failures with punitive measures |
Protected time and space to learn with others | Staff’s confidence in their abilities | Increasing Capacity for Evaluation - Prepare for evaluation by ensuring the necessary time, training, and financial resources to support evaluation activities are in place before commencing evaluation - Establish evaluation governance and leadership teams with clear reporting structures and timelines. Enlist the governance and leadership teams to remove barriers for the evaluation team and staff involved in innovation - Position evaluation as part of the “daily work” of staff involved in innovation - In addition to evaluation as daily work, ensure focused time is protected for sharing learnings and reflections about evaluation on a regular basis - Continuously socialize new staff to evaluation work and provide refreshers of evaluation responsibilities on a regular basis to both new and existing staff |
Program Theory Domain: Sequence 1 | ||
Open and transparent contexts with meaningful inclusion of staff input | Feelings of value & ownership in staff | Co-Producing Evaluation - Collaborate with staff doing the work of innovation and evaluation to co-develop an evaluation logic model and make decisions about the data to be collected and its use - Regularly revisit the evaluation approach with those doing the work of innovation and evaluation (including staff, patients, families, and community partners) to ensure its continued fit with the ongoing innovation work - Encourage all staff in the innovation and evaluation teams to raise questions about the evaluation approach and suggest changes that may be necessary to make evaluation data more meaningful, and establish a process for incorporating feedback (from staff, patients, families, and community partners) into the evaluation approach on an ongoing basis - Position members of the evaluation team close to the work of innovation (such as embedded evaluators) to assist in both gathering relevant evaluation data, as well as making sense of learnings on an ongoing basis by those implementing the innovation through bidirectional knowledge exchange |
Continuous assessment of data and collaborative data discussions between leaders and evaluators | Relevance to staff’s work | Implementing Learning Feedback Loops - Establish a plan for collecting, analyzing, reporting, and using data in collaboration with those with the power to take action based on the results of learnings generated from those data - Regularly assess to ensure the format and timing of data delivery continues to be useful for those with the power to take action on evaluation results - Establish a process for taking action to maintain, iterate, adjust, or stop the use of an innovation based on evaluation results - Establish a “feedforward” workflow for removing barriers to taking action on data in collaboration with organizational leaders, should it be needed |
Program Theory Domain: Sequence 2 | ||
Evaluation data enable continuous refinements of innovations | Staff’s observation of positive change | Creating a Sustainability Culture - Establish a process for taking action at individual, team, and organizational levels to translate learnings into actions for innovation sustainability - Continuously assess the fit of the innovation for the challenges it is being implemented to solve, based on ongoing learnings. Take action when evaluation data suggest that innovation refinements are needed - Recognize, collect, and communicate evidence of positive change as a result of innovation and evaluation often - Allow for evaluation to generate new questions or improvement opportunities on an ongoing basis by implementing a system to capture and prioritize new targets for action |
Organization adapts practice and policies based on evaluation data | Staff’s observation of learning as a driver of change | Becoming a Learning Organization - Allow evaluation data to generate disruptions at the organizational level by building in flexibility and responsiveness into leadership teams, which can adapt to ongoing learning from evaluations - Value learning from praxis, intuition, and dialogue as well as learning from “intentionally collected” evaluation data - Establish knowledge-sharing forums where individuals and teams can share and reflect on their experience with innovations and their evaluations to integrate narratives into a collective experience - Intentionally connect individual, team, and organizational learning to change initiatives to demonstrate the impact of learning from evaluation on changes to practice and policies |