Peer review quality assurance in stereotactic body radiotherapy planning: the impact of case volume

Abstract

Purpose:

Peer review is an essential quality assurance component of radiation therapy planning. A growing body of literature has demonstrated substantial rates of suggested plan changes resulting from peer review. There remains a paucity of data on the impact of peer review rounds for stereotactic body radiation therapy (SBRT). We therefore aim to evaluate the outcomes of peer review in this specific patient cohort.

Methods and materials:

We conducted a retrospective review of all SBRT cases that underwent peer review from July 2015 to June 2018 at a single institution. Weekly peer review rounds are grouped according to cancer subsite and attended by radiation oncologists, medical physicists and medical radiation technologists. We prospectively compiled ‘learning moments’, defined as cases with suggested changes or where an educational discussion occurred beyond routine management, and critical errors, defined as errors which could alter clinical outcomes, recorded prospectively during peer review. Plan changes implemented after peer review were documented.

Results:

Nine hundred thirty-four SBRT cases were included. The most common treatment sites were lung (518, 55%), liver (196, 21%) and spine (119, 13%). Learning moments were identified in 161 cases (17%) and translated into plan changes in 28 cases (3%). Two critical errors (0.2%) were identified: an inadequate planning target volume margin and an incorrect image set used for contouring. There was a statistically significantly higher rate of learning moments for lower-volume SBRT sites (defined as ≤30 cases/year) versus higher-volume SBRT sites (29% vs 16%, respectively; p = 0.001).

Conclusions:

Peer review for SBRT cases revealed a low rate of critical errors, but did result in implemented plan changes in 3% of cases, and either educational discussion or suggestions of plan changes in 17% of cases. All SBRT sites appear to benefit from peer review, though lower-volume sites may require particular attention.

Introduction

The increasing use of high precision radiation therapy (RT) such as stereotactic body radiation therapy (SBRT) emphasises the importance of the ‘process of care’, a framework that facilitates the appropriateness, quality and safety of RT delivered to cancer patients ¹ . Peer review quality assurance (QA) is an essential component of this framework, ensuring quality and safety in clinical management, RT planning and treatment. It has a direct impact on clinical outcomes such as toxicity, local control and overall survival. ^2,3 This widely adopted process involves the evaluation by colleagues of various clinical and RT treatment components including target volumes and organs-at risk (OARs) delineation, intention to treat, dose/prescription, dose distribution, targets/OARs dosimetry and treatment and delivery techniques, and is recommended to be completed prior to starting treatment. ⁴ Historically, 2D simulation films were reviewed with proposed treatment field borders; the advent of 3D planning has increased the complexity of peer review due to the various target/OAR structures and dosimetry involved. ⁵

There are limited data reporting the impact and importance of peer review QA on SBRT cases. ^6,7 Due to the high ablative radiation doses and relatively small margins utilised with SBRT, there is a demonstrated potential for severe toxicity, even in a carefully monitored trial setting. ⁸ Thus, as the role of SBRT increases across a variety of body sites, an awareness of the critical importance of RT quality should be emphasised, with every effort made to utilise both the error-detection capability and educational value of peer review QA wherever possible.

We report the results of a single-institution, retrospective review of peer-reviewed SBRT cases at an academic teaching hospital, describing outcomes collected on suggested plan changes, implemented plan changes and critical errors.

Methods

A review of prospectively recorded outcomes for all peer-reviewed SBRT plans (i.e., extracranial) was performed for cases between July 2015 and June 2018 inclusive. Intracranial radiosurgery plans were excluded. Routine QA practices, detailed below, enabled the identification of suggested plan changes and critical errors. An institutional ethics waiver was granted by the University Health Network, Toronto Research Ethics Board as this was performed as a departmental quality assurance initiative.

RT treatment plans undergo peer review QA in formal weekly or fortnightly chart rounds at our institution, with outcomes prospectively recorded. These are grouped by subsite (spine, gastrointestinal (GI), head and neck (H&N), gynaecological, lymphoma, genito-urinary and lung) with a goal of all cases being reviewed prior to treatment commencement. SBRT cases undergo QA according to their treatment site (e.g., liver SBRT cases for both primary hepatocellular carcinoma and metastases are discussed at GI QA rounds). SBRT cases were defined as greater than 5 Gy per fraction using stereotactic localisation and motion management techniques, steep dose gradients and sub-centimetre margins depending on body site treated.

Plan aspects reviewed included treatment intent/management, prescription, target volumes, OAR delineation, pre-treatment imaging/registration and on-treatment imaging, dose-volume histograms (DVHs) and dosimetry. Prior studies have reported outcomes specifying suggested changes from peer review, and also whether such changes were implemented. ⁹ Previous experience with this approach at the authors’ institution identified ambiguity with the use of ‘suggested changes’ as an outcome. For example, ‘suggestions’ could result from detection of clinically detrimental errors, to a discussion of minor differences in individual practice without definite clinical consequences. The decision to implement or reject a suggestion could be clinician judgement based on factors not known at peer review, or balancing the clinical significance of the change against potential delays in treatment required to execute the change. To disambiguate these outcomes, the results of peer review were prospectively captured as follows:

• • Discovery of actual errors (determined by consensus to be unintended/unacceptable) was reported in the programme’s incident reporting system, noting that:

  1. a. critical errors (requiring immediate plan changes due to potential compromise of patient safety/treatment quality) are expected to be reliably reported

  2. b. non-urgent and non-critical error types maybe less frequently reported

• • Implemented changes to a plan are electronically captured by registering an update to treatment plan documentation.

• • Significant discussion about the case beyond routine discussion during peer review was captured and termed ‘learning moments’, suggesting educational value at minimum.

1. c. These are related to practice/management, dose/fractionation, plan quality/DVHs, contoured volumes and treatment technique

Results

A total of 934 SBRT cases over a 3-year period had complete peer review data for analysis. Of these, the most common SBRT treatment sites were lung (518, 55%), liver (196, 21%) and spine (119, 13%) (Figure 1). In total, 886 out of 934 cases (94.9%) were reviewed prior to radiotherapy commencement, while 44 cases (4.7%) and 4 cases (0.4%) were reviewed during and after treatment, respectively. An additional 358 excluded cases were found to have undergone peer review, but had insufficient outcome data recorded (such as no information regarding which plan aspects were reviewed). These comprised 22% of lung cases, 27% of liver, 32% of spine, 34% of non-spine bone, 21% of nodal, 34% of adrenal, and 72% of prostate cases, as well as 2 out of 6 pancreas, 2 out of 17 soft tissue and 2 out of 4 kidney cases.

Figure 1. SBRT sites reviewed.

The plan aspects reviewed were prescription (96%), treatment intent/management (92%), OAR delineation (87%), plan/dosimetry (77%), pre-treatment imaging/registration (75%), target volume delineation (53%) and on-treatment imaging (3%) (Figure 2). On-treatment imaging was the least-reviewed plan aspect across the three highest-volume SBRT sites, with a review rate of 7% for liver, 3% for spine and 2% for lung. Table 1 shows the breakdown of plan review contents by subsite.

Figure 2. Contents of peer review.

Table 1. Breakdown of reviewed plan aspects, by subsite

Learning moments were identified in 161 cases (17%). These are related to practice/management in 12% of cases, target volume in 11%, dose/fractionation in 11%, treatment technique in 8% and plan quality/DVHs in 6%. The rate of learning moments varied depending on treatment site (Figure 3). Table 2 shows the breakdown of learning moment types per treatment site. The majority of these 161 cases had multiple types of learning moments recorded; 109 had a single learning moment (28%), 26 had 2 (7%), 196 had 3 (51%), 24 had 4 (6%) and 31 had 5 (8%).

Figure 3. Learning moments by SBRT site.

Table 2. Breakdown of learning moments, by subsite

Treatment sites with an average of over 30 cases reviewed per year (lung, liver and spine) had an overall learning moment incidence of 15.8%, while lower-volume sites with 30 or less cases reviewed per year (non-spine bone, lymph node, adrenal, prostate, soft tissue kidney and pancreas) had a learning moment incidence of 28.7% (Figure 4). This difference was statistically significant (p = 0.001).

Figure 4. Learning moments, categorised by subsites and compared with caseload.

Plan changes were implemented in 28 of 934 cases (3%) following peer review. Per body site, the plan change rate was 3.7% for lung, 2% for liver, 1.7% for spine, 18% for prostate and 50% for soft tissue. Unfortunately, documentation of the specific type of change was inconsistent and therefore not possible to accurately report.

A total of two critical errors (0.2%) were identified. One was a liver SBRT plan with an inadequate margin to account for respiratory motion, resulting in a geographical miss. This was due to the assessment of respiratory motion by comparing an exhale breath-hold computerized tomography (CT) to a free-breathing CT, as opposed to an inhale breath-hold CT. This was detected after the first fraction (of 5) had been delivered and was replanned with altered margins prior to recommencing. The second case was a lung SBRT plan with an incorrect imaging sequence co-registered for contouring, resulting in a potential geographical miss of 5 mm. This was detected on the day of the first fraction (of 8), and alterations were made prior to treatment delivery, which was delayed by 2 days.

Discussion

Data regarding the impact of peer review on SBRT planning are limited to two smaller cohorts from time periods in which SBRT utilisation was less frequent, and another more recent cohort of 285 cases. ^10–12 Suggested plan changes from peer review are variably reported in the literature, but commonly are categorised as major changes which require repeat planning and minor changes which do not. ^13–16 A previous review found that suggested plan changes occur in 10% of all cases, with higher rates of change observed for stereotactic radiotherapy and several specific cancer subsites. ⁶ A systematic review from 2017 found similar outcomes, with suggested plan changes in 10.8% of all cases reviewed. ⁷ These both appear to be lower than the rates of change for SBRT cases in the published literature to date. A Canadian report of 40 lung SBRT cases found major contouring changes recommended in 80% of plans, with 18% relating to target volumes and the rest relating to OARs. ¹⁰ A report of stereotactic radiotherapy cases in Australia found suggested plan changes in 22.3% of cases, though the rate of implementation of changes was unclear. ¹² A 2015 single-institution study of 513 SBRT cases found that changes were implemented in 22.6% of cases, with lower rates of change in higher-volume subsites including liver (18%) and lung (17%) cases. ¹¹ The proportion of suggested changes which were not implemented was not reported. These suggest that there is a higher rate of recommended plan changes for SBRT cases compared to non-SBRT cases, which is particularly relevant given the increasing utilisation of SBRT. We found a frequency of 17% for ‘learning moments’, while the rate of implemented plan changes was much lower at 3%.

The proportion of plan changes actually implemented following suggestion of a change was variably reported in the literature. For general radiotherapy cases, Walburn et al. found that radiation oncologist compliance with suggested changes was only 59%, with lower rates of compliance for more significant plan changes—suggesting that radiation oncologistss are reluctant to implement major changes. ⁹ This may also imply that a substantial proportion of suggested changes were deemed to be of educational value rather than direct clinical significance. When considering the high rate of learning moments (28.7%) for our lower-volume treatment sites, it may be inferred that discussions of educational value occur more commonly when reviewing these cases. Time and resource limitations often mean that only selected cases can undergo peer review; thus, one of the factors considered when prioritising cases for review could be case volume. ⁶

We found a statistically significant difference in the rate of learning moments between higher- and lower-volume SBRT sites. This may be due to the presence of established institutional protocols for higher-volume sites and a larger body of evidence and guidelines. ^17–20 This is consistent with findings by Matuszak et al. for their cohort of SBRT cases, with lower rates of change noted for high-volume subsites. ¹¹ Compared with their series, our rate of suggested plan changes was lower at 3% compared with 22.6%. One reason for this may be the timeframe in which the analyses took place—our series is from a more recent time period (i.e., 2015–2018) compared to 2012–2015 in their series. During this intervening period, the evidence for SBRT has grown considerably, as well as utilisation rates and availability of SBRT-specific guidelines and protocols. ²¹ Furthermore, the rate of changes from peer review tends to reduce over time due to the development of standardised protocols and the convergence of individual practices towards agreed standards. ^16,22 A second reason for the discrepancy may be that changes suggested at peer review are not always implemented.^[9] Brundage et al. found that a high proportion of changes suggested from peer review were not actually implemented (51.2%), but instead served as guidance for future cases.^[5] This suggests there is a degree of subjectivity in radiotherapy planning, with the inference that there is a degree of acceptable plan variability for each case. Our data suggest that the rate of absolute critical errors is low; thus, there is a clear difference between what is considered to be outside of the accepted norm, and what is a critical safety issue. Finally, many cases in our department undergo individual one-on-one peer review before being formally presented at peer review rounds. As such, this process may have filtered out some of the changes that would otherwise be suggested at peer review rounds.

The overall rate of critical errors was low at 0.2% (2 cases). Critical errors in our institution were defined as issues which could compromise patient safety—in our series, these included an inadequate planning target volume margin and an incorrect image dataset used for co-registration from which volumes were defined. In both cases, a systematic root cause analysis was conducted to improve processes for subsequent cases. It is unclear from previous reports how frequently errors of this gravity were detected in peer review rounds.^[11,12] It should also be noted that there was a large difference between the rate of actually implemented plan changes and the rate of critical errors (3% versus 0.2%). It is likely that many suggested or implemented plan changes are presented as non-critical suggestions as opposed to compulsory changes deemed necessary for patient safety/quality of care.

As with any retrospective review, there are a number of limitations with our study. It was apparent that not all cases had a full set of peer review codes recorded: out of 1292 cases undergoing peer review, 934 had specific information regarding which plan aspects were reviewed and learning moments. It could be argued that our data may not be completely representative due to this discrepancy. However, these cases represented a minority for most treatment subsites. For prostate, soft tissue and kidney, they represented over 50% of cases—though these subsites comprised less than 5% of all cases. Furthermore, these were subsites which lacked significant evidence and standardised institutional guidelines during the timeframe of this study. Additionally, the time pressure inherent to peer review rounds and the manual nature of recording additional peer review elements in the electronic system may have contributed to incomplete recording of some elements as described. Indeed, time constraints are known to be a significant barrier to the facilitation of peer review.^[23] Furthermore, our institutional system relies on radiation therapists being present at meetings and recording as the radiation oncologists discuss each case. This results in a degree of familiarity required with the discussion and recording process; thus, the reliability of results may be affected if responsible staff are away or there is ambiguity in the discussion. Based on these findings, we are actively reviewing our peer review processes at our institution in accordance with recommendations by Marks et al.^[4]

A further limitation to our study is that the proportion of ‘learning moments’ which were suggested changes versus educational discussion is unknown. Finally, there was no specific recording of implemented plan changes; the only way to determine which plans underwent changes was to search for plan amendments. It is possible that our result thus underestimates the number of actual plan changes. Through undertaking this study, a number of areas for future improvement in our processes have been identified, such as more standardised outcome measures, ‘learning moment’ differentiation between educational discussions and suggested changes, confirmation of implemented plan changes and a more streamlined system of data capture.

Conclusions

Peer review for SBRT cases revealed a low rate of critical errors, but did result in implemented plan changes in 3% of cases, and either educational discussion or suggestions of plan changes in 17% of cases. These appeared to occur more frequently in lower-volume SBRT sites, suggesting that particular attention could be given to these areas. Peer review should remain an integral part of the RT treatment process as the utilisation of SBRT expands over time, with ongoing assessment and optimisation of recording processes.