Clinical stakeholder preferences for paediatric cardiac surgery outcome reporting in Australia and New Zealand

Abstract

Background:

Outcome reporting is an essential element of quality assurance. Evaluation of the information needs of stakeholders of outcome reporting is limited. This study aimed to examine stakeholder preferences for the content, format, and dissemination of paediatric cardiac surgery performance data in Australia and New Zealand.

Methods:

Semi-structured interviews were completed with a purposive sample of Queensland stakeholders to evaluate their attitudes and expectations regarding reporting of paediatric cardiac surgery outcomes. The interviews were audio-recorded and transcribed. Two researchers used an interpretive description approach to analyse the transcripts qualitatively.

Results:

Nineteen stakeholders were interviewed including fifteen clinicians, four parents, one hospital administrator, and one consumer advocate were interviewed. Mortality was highlighted as the area of greatest interest in reports by clinical and consumer groups. The majority preferred hospital rather than individual/clinician-level reporting. Annual reports were preferred by clinicians who requested reports be distributed electronically.

Conclusions:

The evidence generated from outcome reporting in paediatric cardiac surgery is highly desired by clinicians, administrators, parents, families, and advocacy groups. Clinical users prefer information to assist in clinical decision-making, while families seek personalised information at crucial time points in their clinical journey.

Approximately seven children are born in Australia every day with Congenital Heart Disease (CHD). ¹ Many will require surgical intervention in their first year of life. Surgery on the paediatric heart is challenging, resulting in variations in outcomes between centres. Differences in outcomes are more apparent in complex procedures, which may be performed infrequently. ^2,3 In the early nineties, investigation into the high incidence of death in babies after cardiac surgery in the United Kingdom resulted in radical reforms, including mandatory benchmarking. ^4,5 Over ten years, reforms significantly improved paediatric cardiac surgical outcomes, with the 30-day mortality rate falling from 4.3% to 2.6%. ⁶

Clinical registries are a commonly used benchmarking tool and are excellent sources of detailed information. They are well recognised as efficient, cost-effective tools to improve clinical outcomes. ^7–9 Registries are particularly beneficial in an era of decreasing research funding, which requires clinical researchers to do more with less. ¹⁰ Harnessing and contextualising the data generated through registries is vital however as data without interpretation is less powerful. ^11,12

The primary output of most registries is an annual report. Annual reports typically feature similar content. They are often detail heavy with limited flexibility to contextualise information to the needs of different recipients. ¹³ Unfortunately, there is little information, on the ideal format of outcome measure feedback. ^14–16

Evidence suggests that most reports from registries are generated without consultation regarding report recipientsʼ requirements and information preferences. In a 2019 study by John et al., ¹⁷ investigators found that many consumers were unaware that outcome reporting was available and that current presentation formats limited understanding. Without input from stakeholders, ability to drive change and design systems to meet stakeholder needs is limited. For registries to adequately meet their desired goal of driving quality improvement, registry generated outputs must be provided in a format that recipients can use. ^18–20 Outcome reporting must include desired, helpful content to inform priorities. ^11,21–23

Registry-based benchmarking has recently been introduced in Australia and New Zealand (ANZ) by paediatric cardiac surgeons to ensure quality control in this region. ²⁴ Benchmarking is accomplished locally via a centralised reporting and outcome analysis registry entitled “ANZCORS.” Methods of the ANZCORS registry have been previously published. ²⁵

This study aimed to examine clinical and consumer stakeholder expectations and preferences regarding the format and content included in outcome data to assist with the design of future ANZCORS reporting. A qualitative methodology was used to identify stakeholder beliefs and values.

Materials and method

A series of in-depth, semi-structured stakeholder interviews were conducted to evaluate stakeholder attitudes and expectations regarding ANZCORS report content and dissemination strategy. Open-ended questions were used to examine priorities for communicating surgical outcome data. An iterative approach was taken in which questioning was adapted based on participant response to extract the most detail possible on emerging concepts.

The study was approved by the local Hospital Research Ethics Committee (HREC/19/QCHQ/49534).

Setting

The study was conducted at a quaternary paediatric hospital providing comprehensive management of approximately 400 cardiac surgical procedures per year. Interviews were conducted in person or via videoconferencing.

Patients

A purposive maximum variation sampling approach was used to gain an understanding of broad stakeholder preferences including parents, surgeons, intensivists, cardiologists, nurses, anaesthetists, executives, and hospital administrators. Clinical participants were selected based on speciality and experience to ensure a representative sample of likely users of the data were involved.

Data collection and analysis

Meetings were led by a facilitator who also acted as a scribe. Interviews lasted on average one and a half hours. All participants provided consent before participation. Each meeting commenced with an introduction to the design and development of the Australia and New Zealand Cardiac Outcomes Registry for Surgery (ANZCORS). Participants were presented with the 2015–2019 ANZCORS report along with a discussion regarding current processes and format of reporting. Following this, the facilitator asked the participants questions listed in Table 1 to identify their views, preferences, and understanding. The interviews were either transcribed, audio recorded, or both.

Table 1. Interview questions

Two researchers conducted a qualitative analysis using an interpretive description approach. Researchers summarised the views of each participant to reflect the range of opinions and provide an indication of the extent of consensus or divergence of views. Subsequently, thematic analysis was undertaken to identify the key underlying concepts and perceptions that informed cliniciansʼ perspectives. In both the descriptive and the thematic analysis, there was a high level of agreement between researchers; where differences occurred, a consensus was achieved through discussion with another member of the research team.

Results

Fourteen clinicians, four parents, one hospital executive, and one local patient representative participated in the interviews. Clinicians interviewed included surgeons, cardiologists, intensivists, anaesthetists, clinical nurses, and allied health workers. Both senior and junior team members were interviewed. Three of the four parents interviewed were parents of children who had recently undergone cardiac surgery and were currently hospital inpatients. Two of these had travelled from regional and remote areas for the treatment.

Table 2 presents an overview of stakeholdersʼ preferences regarding the format and content of outcome reporting. Clinicians collectively expressed positive feedback regarding the provision of reports. Several themes appeared to shape cliniciansʼ views. Factors influencing attitudes included duration of practice, professional background, and position.

Table 2. Interview results

Report stakeholders

Broader ANZCORS report stakeholders were identified in interviews highlighting the importance of the multidisciplinary care team in providing care for children with CHD. Other stakeholders that might benefit from provision of the report included the wider diagnostic team, neonatologists, peri and post-operative care providers, allied health, administrative personnel, and primary care providers).

Content

Most clinicians highlighted post-surgical mortality as the report section of most interest. Other details of interest included team and facility profiles, comparison of site results, and hospital length of stay. There were no apparent differences between stakeholders in these preferences.

Areas of the report where stakeholders expressed a need for further detail included;

• A description of how to interpret statistical results;

• Long-term outcomes (neurodevelopment, the average length of time until regular activity can resume);

• Wider adverse events;

• Patient-related outcomes (severity of disease, age, demographics);

• Risk-adjusted outcome evaluation (currently pending the development of a national risk adjustment model for ANZ);

• Co-morbidities;

• Inclusion of specialist anaesthetic and perfusion metrics;

• Service details including clinical backgrounds, multidisciplinary team members, and accreditation;

• The average length of stay associated with each procedure.

All clinicians interviewed requested the inclusion of both morbidity and mortality data. One individual highlighted that standardising this information would ensure consistency of counselling between team members and reassure patients and families.

Clinicians interviewed requested outcome reporting according to 1) procedure and 2) procedure and diagnosis.

Parents expressed differing priorities for information provision. One parent was interested in the bigger picture and understanding all possible outcomes for their child, such as criteria for future surgeries, as they felt this information was lacking. Another set of parents were most interested in survival and the provision of clear and direct communication. The last was most interested in the diagnosis and what the diagnosis meant. Their most significant concern was getting their child home.

Personalised information was preferred by all parents interviewed. One parent requested that information was presented according to diagnosis. The consumer representative suggested a need to communicate to all literacy levels and demographics.

The patient representative highlighted the importance of connecting with families to identify what information families would like and how they would like to receive it. They also suggested that the team must consider that some information can be alarming, particularly for families in the hospital when they are most vulnerable.

Information on both morbidity and mortality was requested by parents. One parent felt that the provision of risk information was also interesting and reported the following.

"risk gets passed over too much…never know the risk until the problem has occurred. Need to know earlier as you might be able to help identify early warning signs…. parents are the constant…the side effects of congenital heart disease are often not explained until there is a concern." Parent

Format

Clinicians had different preferences for the format and length of the report. They suggested several versions of the report were needed, including a fully detailed report, an executive summary, and a centre-specific report. The majority of stakeholders requested the provision of both a detailed report and a summary.

Stakeholders identified several potential dissemination formats; however, most interviewees preferred electronic communication, including a dashboard. A website was also suggested with different access to be granted according to the designation of the individual.

While some clinicians highlighted the utility of providing hard copies of the report, others highlighted security concerns and costs associated with this approach.

Many clinicians requested that results be presented at local and national meetings. One individual suggested that information be provided orally to patients and families at the time of consent. The hospital executive asked that information be shared in the public domain on social media.

Many clinicians requested the increased use of pictographic visual aids, and several staff suggested that the team enlist the assistance of a graphic designer. One of the families interviewed also suggested that diagrams and visual aids would be beneficial. This was supported by feedback by the local patient representative, who felt that infographics would be a good tool and that graphs may be helpful to show that centres are performing at standard. The patient representative felt that a simplified version with infographics and pictures would be what most individuals would understand.

Several of the parents suggested a hard copy “booklet” would be a helpful way to provide this information. The booklet would include space for individualised pictures of their child’s anatomy. One parent thought that a verbal explanation should be used to supplement information provided in the booklet. Multidisciplinary meetings were identified as a means to communicate this information. One parent felt that the inclusion of this report on the advocacy group website would be unlikely to be seen. In contrast, the patient representative suggested that this was the best place to share this information.

Frequency

The majority of clinicians interviewed preferred the provision of annual reports. It was suggested that more frequent reporting might cause unnecessary alarm if data was insufficiently powered for analysis leading to misinterpretation. Some, however, noted the need for interim analysis and evaluation in the event of outliers.

One parent felt there was no right time to provide information, and another suggested that this information should be provided individually for each patient. The consumer representative thought the surgical period could be very stressful for patients and that antenatal provision may be suitable as this would provide reassurance. One parent felt that during treatment, they wanted to know what was going on and that they would likely use the booklet as a memory. This parent preferred all information to be given together.

"Often during this time, it is hard to process everything, so it is useful to have something to look back on. At the time of diagnosis, you are thinking you are saying goodbye to your child. You are given worst-case scenario, and often it feels you are pushed to terminate. At the time of surgery, it is nice to look at your baby and remember what you did it all for." Parent

"The most significant part of my journey was relocation. It would be nice to know about this and have the impact explained to you, but you are not told much about the process" Parent

Dissemination

Most clinicians interviewed endorsed the notion of comparing their results to a national average rather than comparing each site in an identifiable way. Most welcomed the opportunity of identifying and learning from better performers. Many, however, expressed a need for caution when sharing reports with consumers. Main concerns appeared to relate to misinterpretation of throughput in different centres, which they felt consumers might see as a difference in experience or ability at a less busy centre.

"Some centres are more likely to treat more difficult cases causing a pre-selection of patients. A surgeon can lose confidence if they are targeted or reprimanded unnecessarily" Cardiology consultant.

“To improve outcomes, centres need to know firstly that they need to improve and secondly they need to understand what other centres are doing differently that result in better outcomes. Patients and families however do not need to know these details” Trainee cardiac surgeon

"Parents may have different education levels, so interpretation of statistics is challenging" Cardiac anaesthetist

Discussion

Overall, stakeholders expressed a need for this information and viewed outcome reporting as a positive learning opportunity. Such a view is in line with positive deviance theory, ²⁶ a strength-based approach that focuses on the opportunities for learning from successful behaviours and strategies from others in similar contexts. ²⁶ The interviews highlighted the mixed viewpoints and differing needs of parents and health professionals regarding what should be included in reports and how results should be presented. This aligns with research by van Overveld et al., ²⁷ who highlighted the differing feedback preferences of clinical and consumer stakeholders.

We found that parents primarily wished to receive information most relevant to them and their children. These results are similar to those reported in the van Overveld study, where researchers identified that patients preferred receiving information when aspects had become applicable to them. ²⁷

In the van Overveld study, the authors highlighted clinicians were concerned about the transparency of data and the risk of misinterpretation. ²⁷ Clinicians felt that reputational damage could occur when a hospital was found to have poorer performance or higher rates of complications. ²⁸ This is a common concern when reporting occurs at an individual level. Many clinicians in our study similarly expressed concern about the potential harm of individual identification. They desired to report to be at a hospital level, with comparison against a national average. This preference was also seen in a study of 107 British doctors and nurses. ²⁸ Literature suggests that the experience of the multidisciplinary team may have a more significant influence on outcomes than individual experience. Patients have also been found to prefer hospital-level information finding it more informative than information at a surgeon level. ²⁹

A 2018 cross-sectional survey of parent preference regarding reporting of outcomes in congenital heart surgery found that the most critical information for parents was survival with surgeon experience following. ³⁰ This appears to differ from our findings. Our differing approach to data collection may account for this.

Parents requested contextualisation, individualisation, and clarity in communication. Health professionals expressed a need for 1) detailed information to counsel families, ensuring the standardisation of messaging regarding risk and 2) the ability to evaluate local performance against a national average to ensure the local practice was at an optimal standard. Based on these results, the ANZCORS team will offer several reporting levels, including tailored feedback on centre outcomes against the national average and an executive summary that stakeholders, particularly inpatient counselling, may more broadly use. Lastly, the team aims to complete implementation work with academic support and support of the national patient advocacy agency, HeartKids, to co-design reporting with families for parents/caregivers.

The public reporting of outcome data allows clinicians to evaluate their practice and, when provided to consumers, encourages them to play an active role in decision-making. ^31,32 Increased patient engagement leads to better health outcomes and patient experience. ³³

Limitations of the study

This paper represents a pilot implementation study and involves a small number of patients from only one institution. Saturation was reached for clinical interviews; however, further interviews with consumer representatives are now needed to fully explore themes. Inclusion of patients from other institutions would also be helpful in future studies, to identify if these results apply in other contexts. Finally, additional work is required in order to conclusively evaluate and design reporting mechanisms to meet the needs of broad stakeholder groups.

Conclusions

The evidence generated from outcome reporting in paediatric cardiac surgery is highly desired by clinicians, administrators, parents, families, and advocacy groups. Clinical users desire information to assist in clinical decision-making while families seek individualised information at crucial time points in their clinical journey.