Article in Review: Perry JJ, Sivilotti MLA, Sutherland J, et al. Validation of the Ottawa Subarachnoid Hemorrhage Rule in patients with acute headache. CMAJ 2017;189(45):E1379-85.
BACKGROUND
Headache is a common presentation, representing 1%–2% of all emergency department (ED) visits.Reference Goldstein, Camargo, Pelletier and Edlow
1
-
Reference Rabinstein
3
Consequently, 1%–2% of the population has an intracranial aneurysm, and 1%–2% of patients presenting to the ED with concerning headaches will have a subarachnoid hemorrhage (SAH).Reference Rabinstein
3
-
Reference Rinkel, Djibuti, Algra and van Gijn
5
In patients without a history of trauma, 80% of SAHs are caused by a ruptured intracerebral aneurysm; prognosis is poor, with a 25%–50% case fatality rate.Reference Rabinstein
3
,
Reference Lawton and Vates
4
Among survivors, 50% experience long-term neuropsychological effects. Rupture of an aneurysm was preceded by a sentinel leak in 10%–40% of patients, and the likelihood of death or disability is four times higher among patients who were misdiagnosed.Reference Rabinstein
3
,
Reference Lawton and Vates
4
Moreover, accurately diagnosing SAH can be challenging, as 41% of patients with SAH are alert when they present, and greater than 50% are oriented or without focal neurological deficits upon presentation.Reference Weir
6
A clinical decision rule (CDR) would assist clinicians to identify patients with SAH without increasing unnecessary testing. Once derived, a CDR should be prospectively validated (ideally in a new clinical setting) before being widely implemented.Reference McGinn, Guyatt and Wyer
7
The Ottawa SAH Rule is a CDR that was previously derivedReference Perry, Stiell and Sivilotti
8
and refinedReference Perry, Stiell and Sivilotti
9
in separate patient cohorts. It was externally validated in a retrospective chart review in the United States.Reference Bellolio, Hess and Gilani
10
The CDR was designed to identify patients with acute non-traumatic headaches who require further investigation for SAH if one or more predictor variables are present (Table 1).
Table 1 CT HEAD mnemonic for the Ottawa SAH Rule
11
POPULATION STUDIED
The study was a prospective, multicentre validation study of the Ottawa SAH Rule. Consecutive patients presenting to six university-affiliated, Canadian tertiary-care hospitals from 2010 to 2014 were eligible for inclusion. The specific inclusion and exclusion criteria are summarized in Table 2.
Table 2 Inclusion and exclusion criteria for the Ottawa SAH Rule
STUDY DESIGN
The study authors planned to enrol 1,200 patients to detect approximately 75 SAHs. Physicians assessing the CDR variables were blinded to the outcome. Patients who met the criteria but were not included by the physician during their ED visit were termed “missed eligible.” Research staff extracted data from the medical records of these missed patients. The accuracy of the physicians’ interpretations of the predictor variables and CDR were assessed by a nurse who was also blinded to the outcome. Although assumed, it was not stated whether the radiologists were blinded to the predictor variables.
OUTCOME MEASURES
The outcomes were well defined a priori as SAH positive or negative. A positive diagnosis required that an SAH was visible on a computed tomography (CT) scan; xanthochromia in the cerebral spinal fluid (CSF) by visual inspection; or >1×106/L erythrocytes in the final tube, with an aneurysm or arteriovenous malformation on angiography. A negative diagnosis required a normal head CT and lumbar puncture. Patients who did not have both investigations were assumed to be SAH negative after a review of all medical and provincial coroners’ records, as well as a validated structured follow-up phone call at one and six months.Reference Perry, Stiell and Sivilotti
8
RESULTS
In this study, 1,153 patients were enrolled, and 590 were “missed eligible.” Of those 1,153 patients who enrolled, eight were lost to follow-up and assumed to be SAH negative, and 67 patients were SAH positive (prevalence of 5.8%). Authors concluded that the Ottawa SAH Rule had a sensitivity of 100% (95% confidence interval [CI] 94.6–100) and specificity of 13.6% (95% CI 13.1–15.8) for diagnosing SAH that corresponded to a +LR 1.16 (95% CI 1.13-1.19) and –LR 0.
COMMENTARY
In this validation study, the Ottawa SAH Rule remained 100% (95% CI 94.6–100) sensitive for the detection of SAH.Reference Perry, Stiell and Sivilotti
8
-
Reference Bellolio, Hess and Gilani
10
With a –LR 0, the widespread adoption of the rule could result in lower missed rates of SAH, as compared with the currently estimated missed rate of 5.4% (95% CI 4.3–6.6).Reference Vermeulen and Schull
12
This study was slightly underpowered with 1,153 patients. In addition, the eight patients who were lost to follow-up were assumed to be SAH negative. Although unlikely, both could result in a falsely elevated sensitivity for the CDR. It is not clear whether the 590 “missed eligible” patients were truly missed or whether they could represent a selection bias. However, the patient characteristics and prevalence of SAH were similar in both groups of patients.
With a low specificity of 13.6% (95% CI 13.1–15.8), this CDR could increase the SAH investigation rates, and an impact analysis would be required to confirm this issue. However, the authors did comment that despite the low specificity, the number of investigations remained unchanged in both the derivation study (85.7%)Reference Perry, Stiell and Sivilotti
8
and this study (87%). The physicians’ interpretations of the predictor variables were excellent, with a Kappa of 0.82 (95% CI 0.76–0.87). The presence of neck pain or stiffness had the most inter-observer disagreement. The physicians were uncomfortable applying the rule to only 9% of the patients, although no reasons were specified.
SAH is associated with significant morbidity and mortality,Reference Lawton and Vates
4
and establishing external validity is extremely important before this CDR can be widely adopted into practice.Reference McGinn, Guyatt and Wyer
7
The derivation and prospective validation of the CDR were rigorously conducted according to strict methodology standards.Reference McGinn, Guyatt and Wyer
7
Statistical methods used to construct and refine the rule were appropriately and accurately described.Reference Perry, Stiell and Sivilotti
8
,
Reference Perry, Stiell and Sivilotti
9
The consistently high sensitivity among the derivation, refinement, external validation, and prospective validation studies adds strong internal validity to the CDR.Reference Perry, Stiell and Sivilotti
8
-
Reference Bellolio, Hess and Gilani
10
Each of the six academic centres used in this study had previously participated in the derivation and refinement studies, but the initial recruitment for these studies was separated by more than 10 years, providing temporal validation.Reference Altman and Royston
13
Although more than 150 physicians were involved, it is likely that they were familiar with the CDR from its development, and their comfort with the rule decreased the external validity for new centres. With that in consideration, the patients were drawn from a heterogeneous population, and the patients represented a wide spectrum regarding the severity of disease at final diagnosis. Moreover, the rule was externally validated in the United States in a retrospective study.Reference Bellolio, Hess and Gilani
10
This provides sufficient evidence that clinicians can use this CDR in a variety of settings, with confidence in its accuracy but with no certainty of improved outcomes. Future research should be directed toward performing an impact analysis to determine whether the rule changes physician behaviour, improves patient outcomes, reduces the number of investigations, or any combination of these issues.
Physicians should still consider other important risk factors for aneurysm rupture, including hypertension, smoking, alcohol abuse, sympathomimetic drug use, black race, and a Hispanic ethnic group before deciding to apply the CDR.Reference Lawton and Vates
4
If the CDR were positive, considering its low specificity, physicians should still consider other life-threatening diagnoses for a headache, such as meningitis, stroke, and tumours, that may require investigations other than a non-contrast CT.Reference Goldstein, Camargo, Pelletier and Edlow
1
,
Reference Rabinstein
3
,
Reference Weir
6
SUMMARY
The Ottawa SAH Rule is a valid CDR that is 100% (95% CI 94.6–100) sensitive for helping to decide who requires further workup for SAH. It is unclear whether the application of this rule would either improve the current care of patients presenting to the ED with sudden onset of an acute non-traumatic headache or decrease unnecessary testing. Physicians who decide to wait for an impact analysis before incorporating the CDR into their practice should, at the very least, consider the predictor variables in their decision-making process.Reference McGinn, Guyatt and Wyer
7
You have
Access
Article in Review: Perry JJ, Sivilotti MLA, Sutherland J, et al. Validation of the Ottawa Subarachnoid Hemorrhage Rule in patients with acute headache. CMAJ 2017;189(45):E1379-85.
BACKGROUND
Headache is a common presentation, representing 1%–2% of all emergency department (ED) visits.Reference Goldstein, Camargo, Pelletier and Edlow 1 - Reference Rabinstein 3 Consequently, 1%–2% of the population has an intracranial aneurysm, and 1%–2% of patients presenting to the ED with concerning headaches will have a subarachnoid hemorrhage (SAH).Reference Rabinstein 3 - Reference Rinkel, Djibuti, Algra and van Gijn 5 In patients without a history of trauma, 80% of SAHs are caused by a ruptured intracerebral aneurysm; prognosis is poor, with a 25%–50% case fatality rate.Reference Rabinstein 3 , Reference Lawton and Vates 4 Among survivors, 50% experience long-term neuropsychological effects. Rupture of an aneurysm was preceded by a sentinel leak in 10%–40% of patients, and the likelihood of death or disability is four times higher among patients who were misdiagnosed.Reference Rabinstein 3 , Reference Lawton and Vates 4 Moreover, accurately diagnosing SAH can be challenging, as 41% of patients with SAH are alert when they present, and greater than 50% are oriented or without focal neurological deficits upon presentation.Reference Weir 6 A clinical decision rule (CDR) would assist clinicians to identify patients with SAH without increasing unnecessary testing. Once derived, a CDR should be prospectively validated (ideally in a new clinical setting) before being widely implemented.Reference McGinn, Guyatt and Wyer 7 The Ottawa SAH Rule is a CDR that was previously derivedReference Perry, Stiell and Sivilotti 8 and refinedReference Perry, Stiell and Sivilotti 9 in separate patient cohorts. It was externally validated in a retrospective chart review in the United States.Reference Bellolio, Hess and Gilani 10 The CDR was designed to identify patients with acute non-traumatic headaches who require further investigation for SAH if one or more predictor variables are present (Table 1).
Table 1 CT HEAD mnemonic for the Ottawa SAH Rule 11
POPULATION STUDIED
The study was a prospective, multicentre validation study of the Ottawa SAH Rule. Consecutive patients presenting to six university-affiliated, Canadian tertiary-care hospitals from 2010 to 2014 were eligible for inclusion. The specific inclusion and exclusion criteria are summarized in Table 2.
Table 2 Inclusion and exclusion criteria for the Ottawa SAH Rule
CT=computed tomography; GCS=Glasgow Coma Scale; SAH=subarachnoid hemorrhage.
STUDY DESIGN
The study authors planned to enrol 1,200 patients to detect approximately 75 SAHs. Physicians assessing the CDR variables were blinded to the outcome. Patients who met the criteria but were not included by the physician during their ED visit were termed “missed eligible.” Research staff extracted data from the medical records of these missed patients. The accuracy of the physicians’ interpretations of the predictor variables and CDR were assessed by a nurse who was also blinded to the outcome. Although assumed, it was not stated whether the radiologists were blinded to the predictor variables.
OUTCOME MEASURES
The outcomes were well defined a priori as SAH positive or negative. A positive diagnosis required that an SAH was visible on a computed tomography (CT) scan; xanthochromia in the cerebral spinal fluid (CSF) by visual inspection; or >1×106/L erythrocytes in the final tube, with an aneurysm or arteriovenous malformation on angiography. A negative diagnosis required a normal head CT and lumbar puncture. Patients who did not have both investigations were assumed to be SAH negative after a review of all medical and provincial coroners’ records, as well as a validated structured follow-up phone call at one and six months.Reference Perry, Stiell and Sivilotti 8
RESULTS
In this study, 1,153 patients were enrolled, and 590 were “missed eligible.” Of those 1,153 patients who enrolled, eight were lost to follow-up and assumed to be SAH negative, and 67 patients were SAH positive (prevalence of 5.8%). Authors concluded that the Ottawa SAH Rule had a sensitivity of 100% (95% confidence interval [CI] 94.6–100) and specificity of 13.6% (95% CI 13.1–15.8) for diagnosing SAH that corresponded to a +LR 1.16 (95% CI 1.13-1.19) and –LR 0.
COMMENTARY
In this validation study, the Ottawa SAH Rule remained 100% (95% CI 94.6–100) sensitive for the detection of SAH.Reference Perry, Stiell and Sivilotti 8 - Reference Bellolio, Hess and Gilani 10 With a –LR 0, the widespread adoption of the rule could result in lower missed rates of SAH, as compared with the currently estimated missed rate of 5.4% (95% CI 4.3–6.6).Reference Vermeulen and Schull 12 This study was slightly underpowered with 1,153 patients. In addition, the eight patients who were lost to follow-up were assumed to be SAH negative. Although unlikely, both could result in a falsely elevated sensitivity for the CDR. It is not clear whether the 590 “missed eligible” patients were truly missed or whether they could represent a selection bias. However, the patient characteristics and prevalence of SAH were similar in both groups of patients.
With a low specificity of 13.6% (95% CI 13.1–15.8), this CDR could increase the SAH investigation rates, and an impact analysis would be required to confirm this issue. However, the authors did comment that despite the low specificity, the number of investigations remained unchanged in both the derivation study (85.7%)Reference Perry, Stiell and Sivilotti 8 and this study (87%). The physicians’ interpretations of the predictor variables were excellent, with a Kappa of 0.82 (95% CI 0.76–0.87). The presence of neck pain or stiffness had the most inter-observer disagreement. The physicians were uncomfortable applying the rule to only 9% of the patients, although no reasons were specified.
SAH is associated with significant morbidity and mortality,Reference Lawton and Vates 4 and establishing external validity is extremely important before this CDR can be widely adopted into practice.Reference McGinn, Guyatt and Wyer 7 The derivation and prospective validation of the CDR were rigorously conducted according to strict methodology standards.Reference McGinn, Guyatt and Wyer 7 Statistical methods used to construct and refine the rule were appropriately and accurately described.Reference Perry, Stiell and Sivilotti 8 , Reference Perry, Stiell and Sivilotti 9 The consistently high sensitivity among the derivation, refinement, external validation, and prospective validation studies adds strong internal validity to the CDR.Reference Perry, Stiell and Sivilotti 8 - Reference Bellolio, Hess and Gilani 10 Each of the six academic centres used in this study had previously participated in the derivation and refinement studies, but the initial recruitment for these studies was separated by more than 10 years, providing temporal validation.Reference Altman and Royston 13 Although more than 150 physicians were involved, it is likely that they were familiar with the CDR from its development, and their comfort with the rule decreased the external validity for new centres. With that in consideration, the patients were drawn from a heterogeneous population, and the patients represented a wide spectrum regarding the severity of disease at final diagnosis. Moreover, the rule was externally validated in the United States in a retrospective study.Reference Bellolio, Hess and Gilani 10 This provides sufficient evidence that clinicians can use this CDR in a variety of settings, with confidence in its accuracy but with no certainty of improved outcomes. Future research should be directed toward performing an impact analysis to determine whether the rule changes physician behaviour, improves patient outcomes, reduces the number of investigations, or any combination of these issues.
Physicians should still consider other important risk factors for aneurysm rupture, including hypertension, smoking, alcohol abuse, sympathomimetic drug use, black race, and a Hispanic ethnic group before deciding to apply the CDR.Reference Lawton and Vates 4 If the CDR were positive, considering its low specificity, physicians should still consider other life-threatening diagnoses for a headache, such as meningitis, stroke, and tumours, that may require investigations other than a non-contrast CT.Reference Goldstein, Camargo, Pelletier and Edlow 1 , Reference Rabinstein 3 , Reference Weir 6
SUMMARY
The Ottawa SAH Rule is a valid CDR that is 100% (95% CI 94.6–100) sensitive for helping to decide who requires further workup for SAH. It is unclear whether the application of this rule would either improve the current care of patients presenting to the ED with sudden onset of an acute non-traumatic headache or decrease unnecessary testing. Physicians who decide to wait for an impact analysis before incorporating the CDR into their practice should, at the very least, consider the predictor variables in their decision-making process.Reference McGinn, Guyatt and Wyer 7
Acknowledgements: JP, MS, and EL all contributed to the inception of the draft and reviewed the final copy for submission.
Competing interests: JP, MS, and EL have no conflicts of interest to declare. This research received no specific grant from any funding agency, commercial, or not-for-profit sectors. JP takes responsibility for the submission as a whole.