Improvement Science Methods


Achieving a balanced accounting of quality improvement impact: A qualitative analysis of interview data, narrative review and modified Delphi study

The complexity of the healthcare system means that efforts to improve quality and safety often achieve only limited benefits and can have unintended consequences, which may impact positively or negatively on care processes and outcomes. However, several systematic reviews have shown that there is little evidence that improvement programmes routinely evaluate the presence of unintended consequences either pre or post-implementation, with little specific guidance on how to best account for improvement consequences other than goals.

Our Current Focus

We have conducted three interconnected studies aiming to explore current understanding of measurement of unintended consequences in healthcare improvement, including how the wide range of consequences can be identified, evaluated and interpreted to achieve a more balanced accounting of the overall impact of improvement interventions.

What’s Been Happening

The first study comprised of a qualitative analysis of interview data carried out in two phases, with semi-structured interviews used to formulate a draft conceptual framework for considering all consequences of improvement which was then refined and elaborated during two focus groups. The framework proposes that a balanced accounting should consider goals (expected desirable consequences) and predictable trade-offs (expected undesirable consequences) early in the design of a quality improvement programme and pause to take stock of pleasant (unexpected desirable consequences) and unpleasant surprises (unexpected undesirable consequences) after a period of implementation.

Drawing on a Cochrane systematic review of ASI in hospitals, the second study aimed to pilot the previously developed framework and examine the range of outcomes measured in published evaluations. We conducted a narrative review and described a more structured framework with specific examples of the range of consequences that ASI can have, in terms of their desirability and the extent to which they were expected during planning. For instance, only 11 (10%) of 110 ITS studies included any information about unintended consequences, with 10 examining unexpected, undesirable outcomes (unpleasant surprises) using case-control, qualitative or cohort designs

The third study consisted of a two-round modified Delphi consultation with 72 experts in round one, and 60 (83.3%) in round two, which elaborates on the previous findings and focuses on exploring and developing consensus in relation to identifying, prioritizing, evaluating and interpreting all consequences of quality improvement, which can ultimately lead to a more balanced accounting of the impact (currently under review with BMJ Open).

What's Coming Next

Adapting and testing the balanced accounting framework in NHS Tayside

The main aim of this project is to engender understanding of improvement advisors of a new ‘’balanced accounting’ conceptual model that will improve identification of ‘’effective’’ balancing measures, support improvement advisors to enhance clinical team’s understanding of balancing measures, suggest strategies to incorporate the balanced accounting into the design of quality improvement projects and provide guidance in addressing the unique challenges associated with achieving it.

Testing of  the balanced accounting framework with specific ongoing or already delivered projects improvement projects in NHS Tayside, to explore how the framework had been used and applied, what was the overall role/impact of the framework, how was the framework understood by the improvement advisors and translated to staff in frontline settings, how the framework had influenced people’s understanding and knowledge of measuring and monitoring unintended consequences of improvement, were there any changes to measurement and data management as a result of introducing the framework and what are the implications for the wider use and application of this new approach.

Key People


Outputs from the Improvement Science Methods research theme can be accessed on our publications page.

Useful Links