Framework to understand the unintended consequences of improvement


Mechanisms of Improvement

Framework to support the development and teaching of balancing measures Improvement Pause – how can it support improvement practice?


What we know

The complexity of the healthcare system means that efforts to improve quality and safety often achieve only limited benefits and can have unintended consequences, which may impact positively or negatively on care processes and outcomes. However, several systematic reviews have shown that there is little evidence that improvement programmes routinely evaluate the presence of unintended consequences either pre or post-implementation, with little specific guidance on how to best account for improvement consequences other than goals.


Completed work

We have conducted three interconnected studies aiming to explore current understanding of measurement of unintended consequences in healthcare improvement, including how the wide range of consequences can be identified, evaluated and interpreted to achieve a more balanced accounting of the overall impact of improvement interventions.


Balancing measures or a balanced accounting of improvement impact: a qualitative analysis of individual and focus group interviews with improvement experts in Scotland

The first study comprised of a qualitative analysis of interview data carried out in two phases, with semi-structured interviews used to formulate a draft conceptual framework for considering all consequences of improvement which was then refined and elaborated during two focus groups. The framework proposes that a balanced accounting should consider goals (expected desirable consequences) and predictable trade-offs (expected undesirable consequences) early in the design of a quality improvement programme and pause to take stock of pleasant (unexpected desirable consequences) and unpleasant surprises (unexpected undesirable consequences) after a period of implementation.

A framework for ensuring a balanced accounting of the impact of antimicrobial stewardship interventions

Drawing on a Cochrane systematic review of Antimicrobial Stewardship Initiative (ASI) in hospitals, the second study aimed to pilot the previously developed framework and examine the range of outcomes measured in published evaluations. We conducted a narrative review and described a more structured framework with specific examples of the range of consequences that ASI can have, in terms of their desirability and the extent to which they were expected during planning. For instance, only 11 (10%) of 110 ITS studies included any information about unintended consequences, with 10 examining unexpected, undesirable outcomes (unpleasant surprises) using case-control, qualitative or cohort designs.


A balanced approach to identifying, prioritising and evaluating all potential consequences of quality improvement: modified Delphi study

The third study consisted of a two-round modified Delphi consultation with 72 experts in round one, and 60 (83.3%) in round two, which elaborates on the previous findings and focuses on exploring and developing consensus in relation to identifying, prioritizing, evaluating and interpreting all consequences of quality improvement, which can ultimately lead to a more balanced accounting of the impact and identified the concept of an ‘improvement pause’.


Translating new knowledge into practice

The next stage of this project is to engender understanding of improvement advisors of a new ‘’balanced accounting’ conceptual model that will improve identification of ‘’effective’’ balancing measures, support improvement advisors to enhance clinical team’s understanding of balancing measures, suggest strategies to incorporate the balanced accounting into the design of quality improvement projects and provide guidance in addressing the unique challenges associated with achieving it.


What this work explored

Exploratory work to adapt and refine the (including improving the language used) to enhance its utility and feasibility (ongoing) :

  • Training materials to support teaching of balancing measures;
  • Prompts for improvement advisors to use to help teams identify more meaningful balancing measures, i.e. 4-6 questions that influence measurement of unintended consequence.
  • Test the impact of an ‘improvement pause’ to reflect on measurement plans and unintended consequences of improvement within the wider system.

The findings from this exploratory work have been refined into a tool to support improvement project reflection – expanding on the ‘Improvement Pause’ concept that was identified within the research.  This approach is currently being tested by a cohort of the Scottish Quality and Safety Fellows – which was due to be completed by April 2020 (but had to be suspended due to COVID-19). 


What this translational element of the study will add

Testing of the concept of the Improvement Pauseis exploring how the Pause can be applied and used, identify the overall role/impact of the Pause, how the Pause was understood by the improvement advisors and translated into frontline settings, how the Pause influenced the understanding and knowledge of unintended consequences of improvement and what are the implications for the wider use and application of this new concept/approach.


Collaborators

This work has been led by Dr Madalina Toma, Research Fellow at the School of Health Sciences, University of Dundee, in conjunction with colleagues within the School and at NHS Education for Scotland, who run the Scottish Quality and Safety Fellowship programme.

For more information please contact SISCC (siscc@dundee.ac.uk)

 

Related Publications:

A balanced approach to identifying, prioritising and evaluating all potential consequences of quality improvement: modified Delphi study (2019)

Balancing measures or a balanced accounting of improvement impact: a qualitative analysis of individual and focus group interviews with improvement experts in Scotland (2017)

A framework for ensuring a balanced accounting of the impact of antimicrobial stewardship interventions (2017)

Links:

The Quality Measurement Framework 

Safer Healthcare (book)

Science of Improvement: Establishing Measures

Introduction to Measurement for Improvement

Scottish Quality and Safety Fellowships