Validation of the Passive Surveillance Stroke Severity Score in Three Canadian Provinces

Abstract:

Background:

Stroke outcomes research requires risk-adjustment for stroke severity, but this measure is often unavailable. The Passive Surveillance Stroke SeVerity (PaSSV) score is an administrative data-based stroke severity measure that was developed in Ontario, Canada. We assessed the geographical and temporal external validity of PaSSV in British Columbia (BC), Nova Scotia (NS) and Ontario, Canada.

Methods:

We used linked administrative data in each province to identify adult patients with ischemic stroke or intracerebral hemorrhage between 2014-2019 and calculated their PaSSV score. We used Cox proportional hazards models to evaluate the association between the PaSSV score and the hazard of death over 30 days and the cause-specific hazard of admission to long-term care over 365 days. We assessed the models’ discriminative values using Uno’s c-statistic, comparing models with versus without PaSSV.

Results:

We included 86,142 patients (n = 18,387 in BC, n = 65,082 in Ontario, n = 2,673 in NS). The mean and median PaSSV were similar across provinces. A higher PaSSV score, representing lower stroke severity, was associated with a lower hazard of death (hazard ratio and 95% confidence intervals 0.70 [0.68, 0.71] in BC, 0.69 [0.68, 0.69] in Ontario, 0.72 [0.68, 0.75] in NS) and admission to long-term care (0.77 [0.76, 0.79] in BC, 0.84 [0.83, 0.85] in Ontario, 0.86 [0.79, 0.93] in NS). Including PaSSV in the multivariable models increased the c-statistics compared to models without this variable.

Conclusion:

PaSSV has geographical and temporal validity, making it useful for risk-adjustment in stroke outcomes research, including in multi-jurisdiction analyses.

Résumé :

Validation du score de gravité de l’accident vasculaire cérébral de surveillance passive dans trois provinces au Canada.

La recherche sur les résultats des accidents vasculaires cérébraux (AVC) nécessite un rajustement du risque du degré de gravité, mais cette mesure souvent n’existe pas. Le score de gravité de l’AVC de surveillance passive (Passive Surveillance Stroke SeVerity ([PaSSV]) est une mesure du degré de gravité des AVC reposant sur des données administratives, qui a été élaborée en Ontario, au Canada. L’étude ici décrite visait donc à évaluer la validité externe du score PaSSV dans le temps et dans l’espace en Colombie-Britannique (C.B.), en Nouvelle-Écosse (N.É.) et en Ontario.

Pour ce faire, l’équipe de recherche a utilisé des données administratives liées de chacune des provinces participantes afin de repérer les adultes qui avaient subi un AVC ischémique ou une hémorragie cérébrale, entre 2014 et 2019, et a calculé leur score PaSSV. Les chercheurs et les chercheuses se sont appuyés sur des modèles des risques proportionnels de Cox pour évaluer l’association du score PaSSV avec le risque de mort sur une période de 30 jours et le risque d’admission dans un établissement de soins prolongés par cause, sur une période de 365 jours. Enfin, les valeurs discriminatives des modèles ont été évaluées à l’aide des valeurs statistiques de concordance d’Uno, par comparaison des modèles avec ou sans score PaSSV.

Au total, 86 142 dossiers de patient ont été retenus dans l’étude (n = 18 387 en C.B.; n = 65 082 en Ontario; n = 2 673 en N.É.). Les scores PaSSV moyen et médian étaient comparables dans toutes provinces. Un score PaSSV élevé, correspondant à un faible degré de gravité, a été associé à un risque moindre de mort (rapport de risques instantanés [RRI] et intervalles de confiance à 95 % : 0,70 [0,68-0,71] en C.B.; 0,69 [0,68-0,69] en Ontario; 0,72 [0,68-0,75] en N.É.) et d’admission dans un établissement de soins prolongés (0,77 [0,76-0,79] en C.B.; 0,84 [0,83-0,85] en Ontario; 0,86 [0,79-0,93] en N.É.). Le fait d’inclure le score PaSSV dans les modèles plurifactoriels a eu pour effet d’accroître les valeurs statistiques de concordance d’Uno par rapport à celles obtenues dans les modèles sans l’intégration de cette variable.

L’étude a permis de démontrer la validité externe du score PaSSV dans le temps et dans l’espace, ce qui en fait un instrument utile de rajustement du risque dans les recherches sur les résultats des AVC, y compris dans les analyses touchant différents territoires de compétence.

Background

Stroke is a leading cause of mortality and morbidity. ¹ Ongoing evaluation of the organized stroke systems of care established across Canadian provinces is necessary to ensure excellence in care and outcomes. ² Such studies typically use linked administrative health data to include large populations over long time periods. ³ However, accurate and fair comparisons of patient outcomes require risk adjustment for baseline stroke severity because it is one of the most important predictors of outcomes. ^4–6 An important limitation of stroke research using administrative data is the lack of a measure of baseline stroke severity. Even in clinical databases created using primary data collection, stroke severity is often missing. ^5,7,8

The Passive Surveillance Stroke SeVerity (PaSSV) score was derived using Ontario administrative data, and was found to be associated with 30-day all-cause mortality after stroke with a similar magnitude of effect as for observed stroke severity ascertained from clinical data in the Ontario Stroke Registry. ⁹ The PaSSV score has the potential to be used as a risk-adjustment tool for multi-jurisdiction stroke outcome comparisons across Canadian provinces. However, health data structure is different in each province, necessitating dedicated external validation analyses for each province. ¹⁰

We calculated the PaSSV score using administrative data in British Columbia (BC) and Nova Scotia (NS) between 2014 and 2019, and we assessed its association with stroke mortality and admission to long-term care in each province. These analyses were also carried out in Ontario to assess validity PaSSV over time. We hypothesized that PaSSV will show similar validity across provinces and over time.

Methods

Study cohort

We used diagnosis codes from the Canadian Institute for Health Information’s Discharge Abstract Database ¹¹ (DAD) to identify hospitalizations with a most responsible diagnosis of ischemic stroke (H34.1, I63 except I63.6, I64) or intracerebral hemorrhage (I61) between April 1, 2014 and March 31, 2019. These codes have been shown to have high accuracy with positive predictive value of 92% for intracerebral hemorrhage and 97% for ischemic stroke. ¹² The beginning of the cohort accrual period falls after the period during which the PaSSV score was derived in Ontario (2002–2013), thereby allowing us to assess the temporal validity of the score in Ontario, as well as its geographic external validity to other provinces.

To exclude elective admissions, we only included records where the admission was through an emergency department and the corresponding National Ambulatory Care Reporting System (NACRS) record was available. ¹³ While the NACRS coverage of emergency departments is complete in Ontario, it is incomplete in BC and NS. ¹³ According to the 2019–20 CIHI estimates, only 27.8% (30 of 108) of emergency departments in BC and 21.1% (8 of 38) in NS were mandated to report to NACRS, accounting for an estimated coverage of 71% of all emergency visits in BC and 49% in NS. ¹⁴ We also excluded patients aged < 18 or > 105 years, those with an invalid health card number, and those who experienced a stroke while hospitalized for a different reason. Among individuals with multiple eligible events, we only included the first one (Table 1). Given inter-hospital transfers are common in stroke, we created an episode of care for each hospitalization to avoid double counting events. The index date was the first day of the episode of care. In Ontario, we used ICES’ (previously Institute for Clinical Evaluative Sciences) standard definition: any admissions within 6 hours of the previous discharge, any admissions within 12 hours of the previous discharge where discharge codes indicate transfer between two acute care hospitals, and any admissions within 48 hours of the previous discharge where the “institution from” and “institution to” numbers match. In BC and NS, we included all emergency department visits within 48 hours of the hospitalization as long as these were also within 24 hours of each other.

Table 1: Cohort creation flow in each province

NACRS = National Ambulatory Care Reporting System; CTAS = Canadian Triage and Acuity Scale.

Passive surveillance stroke SeVerity (PaSSV) score

The PaSSV score was developed in Ontario, where observed stroke severity was available in the Ontario Stroke Registry between 2003 and 2013. ⁹ The Ontario Stroke Registry is a population-based clinical stroke registry with information on the Canadian Neurological Scale (CNS) score, a stroke severity scale ranging from 1.5 to 11.5 where a lower score indicates higher severity, and this clinical score is always an integer or ends in 0.5. ¹⁵ The PaSSV score was derived by fitting a multivariable linear regression model in which we regressed the CNS as a continuous variable on predictor variables obtained from administrative databases. ¹⁶ The components of PaSSV include information from the emergency department (Canadian Triage and Acuity Scale score, arrival by ambulance), hyperacute stroke care (transfer to a higher-level stroke centre, mechanical ventilation within two days of the index date) and the International Classification of Diseases 10^th revisions Canadian codes for stroke symptoms (Supplemental Table 1).

Outcomes

The main outcome was the time to all-cause mortality within 30 days of the index date obtained from the BC vital statistics ¹⁷ , NS vital statistics and Insured Patient Registry and the Ontario Registered Persons Database. We also evaluated the association between PaSSV and admission to long-term care within 365 days among patients who were not in a long-term care facility at baseline. Admission to long-term care was identified using the Home and Community Care database and prescriptions dispensed under the PharmaNet long-term care plan in BC, the Continuing Care Reporting System in Ontario, and the Eligibility Group database in NS.

Research ethics, privacy and data access

Datasets were linked deterministically using unique encoded identifiers in each province. In Ontario, data were linked and analyzed at ICES and the use of data in this project was authorized under section 45 of Ontario’s Personal Health Information Protection Act without requirement for review by a Research Ethics Board. In British Columbia, data were linked at Population Data BC and analyzed at the UBC Centre for Health Services and Policy Research. Access to data provided by the Data Stewards is subject to approval but can be requested for research projects through the Data Stewards or their designated service providers. The following data sets were used in this study: consolidation (census geocodes, demographics, registry), Home and Community Care, DAD, NACRS, PharmaNet, vital statistics and medical services plan payment information file. You can find further information regarding these data sets by visiting the PopData project webpage at: https://my.popdata.bc.ca/project_listings/22-001/collection_approval_dates. All inferences, opinions and conclusions drawn in this publication are those of the author(s), and do not reflect the opinions or policies of the Data Steward(s). In Nova Scotia, data were linked and analyzed at Health Data Nova Scotia with approval from the Nova Scotia Health Research Ethics Board (REB file #1027160).

Statistical methods

We described the patient characteristics and stroke severity using PaSSV in each province. We calculated the PaSSV score using the previously published beta coefficients. ¹⁶ The theoretical range of PaSSV is from −2.3 to 13.1 and unlike the clinical CNS score, PaSSV is not constrained to be either an integer or to end in 0.5, and it was modeled as a continuous variable in our analyses. We estimated the hazard ratios (HR) and 95% confidence intervals (CI) for 30-day all-cause mortality after stroke using two Cox proportional hazards models, one with PaSSV and one without, in each province. All models adjusted for age (continuous), sex, Charlson comorbidity index (dichotomized < 2 versus ≥ 2) with a 5-year look-back period, ¹⁸ and stroke type (ischemic versus intracerebral hemorrhage). We compared the models’ discriminative value using Uno’s c-statistics and 95% CI. ¹⁹ We repeated the analyses with the outcome of admission to long-term care within 365 days using adjusted cause-specific hazard models in order to account for the competing risk of death. Analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, North Carolina).

Results

We identified 25,933 hospitalizations for ischemic stroke or intracerebral hemorrhage in BC, 78,412 in Ontario, and 6,385 in NS. After applying the exclusion criteria, 18,387 (70.9%) hospitalization records from unique patients were included for analysis in BC, 65,082 (83.0%) records in Ontario, and 2,673 (41.9%) records in NS. As shown in Table 1, the proportion of hospitalizations excluded for each exclusion criterion was similar across provinces except for linkage to an emergency department record in NACRS. A higher proportion of hospitalizations in BC and NS, than in Ontario were excluded due to lack of linkage to NACRS: 17.9% (n = 4,654) of patients in BC, 3.6% (n = 2,810) in Ontario, and 48.3% (n = 3,082) in NS.

Table 2 shows the patient baseline characteristics by province, the composite PaSSV score, as well as the frequency of the individual components of PaSSV. There were more patients with a Charlson comorbidity index of ≥ 2 in Ontario than in BC or NS. The mean and median PaSSV scores were similar in the three provinces, but there were differences in the components of the composite score. Compared to the other two provinces, a lower proportion of patients in BC were triaged to the highest acuity CTAS score or underwent transfer to a hospital with a higher level of stroke care, but a higher proportion arrived by ambulance. Use of mechanical ventilation and stroke symptoms recorded in administrative data were similar across provinces.

Table 2: Baseline characteristics by province

In BC, 2,678 (14.6%) patients died within 30 days and among the community-dwelling patients at baseline, 2,355/17,536 (13.4%) were admitted to long-term care within 365 days. In Ontario, 9,199 (14.1%) died and 9,045/61,420 (14.7%) were admitted to long-term care. In NS, 349 (13.1%) patients died and 200/2,587 (7.7%) were admitted to long-term care. In multivariable models, every 1-unit increase in the PaSSV score, where a higher score indicates lower stroke severity, was associated with a lower hazard of death (HR and 95% CI 0.70 [0.68, 0.71] in BC, 0.69 [0.68, 0.69] in Ontario, 0.72 [0.68, 0.75] in NS) and lower hazard of admission to long-term care (0.77 [0.76, 0.79] in BC, 0.84 [0.83, 0.85] in Ontario, 0.86 [0.79, 0.93] in NS), shown in Table 3. Adding PaSSV as a covariate in the multivariable models for 30-day mortality improved the models’ discriminative ability as demonstrated by higher c-statistics (Table 3). We made a similar observation for the 365-day long-term care outcome, but the increase in c-statistics was to a lesser extent.

Table 3: Uno’s c-statistics [95% confidence intervals] and hazard ratios [95% confidence intervals] for multivariable models for all-cause mortality within 30 days and admission to long-term care within 365 days with and without the PaSSV variable

A higher proportion of patients were excluded in BC (29.1%) and NS (58.1%) than in Ontario (17.0%) primarily based on the lack of linkage to an emergency department visit in the NACRS database preceding the admission. Most patients without NACRS records had a flag in the DAD database that the hospitalization was preceded by an emergency department visit (96.5% in BC [4,307/4,465] and 80.6% in NS [2,484/3,082]), indicating that most of these hospitalizations were not elective in nature. We observed several differences in patient characteristics comparing those with versus without NACRS record, the most striking one being that there was a higher proportion of rural residents among those without a NACRS record: 40.2% of patients without a NACRS record in BC and 42.7% in NS were living in rural areas, compared to only 6.2% of those with a NACRS record in BC and 31.0% in NS were living in rural areas (Supplemental Table 2).

Discussion

We successfully calculated the PaSSV score using provincial linked administrative data in 86,142 patients who were admitted to hospital with an ischemic stroke or intracerebral hemorrhage over a 5-year period in three Canadian provinces and made several interesting observations. First, the composite PaSSV score was similar among patients in all three provinces. This is consistent with our clinical expectation that stroke severity would be similar across provinces. Our study confirms that the global PaSSV score is not affected by potential variations in clinical practice patterns or coding practices in different Canadian provinces. Second, we showed that a higher PaSSV score (lower stroke severity) was associated with a lower hazard of death within 30 days and admission to long-term care within 365 days, again consistent with the clinical expectation of that lower stroke severity is associated with better outcomes. Third, we showed that adding PaSSV to multivariable models to predict death or long-term care admission improved the models’ discriminative ability compared to those without PaSSV. The c-statistic and 95% CI for the 30-day all-cause mortality models in all three provinces in the current study were similar to the c-statistic of that published in the original PaSSV derivation cohort (0.76 [0.75, 0.76]). ⁹

These observations suggest that the PaSSV score, initially developed in Ontario, has geographical and temporal external validity. The ability to account for differences in stroke severity is important for stroke outcomes research and quality improvement initiatives within each province and when comparing care and outcomes across jurisdictions. For example, prior work showed that patients being evaluated in comprehensive stroke centers with advanced stroke care and treatments were more likely to be experiencing more severe strokes compared to those treated in primary stroke centers or non-designated centers, and accounting for PaSSV reclassified 18.5% of 157 acute care hospitals across Ontario with regards to their risk-standardized stroke mortality performance compared to a model without PaSSV.(16) PaSSV has also been used for risk-adjustment in a recent American study on population-based access to thrombectomy. ²⁰

A critical limitation is that PaSSV requires linkage to the NACRS database, which contains important information related to the emergency department visit that reflect stroke severity, including the emergency triage acuity, arrival by ambulance and inter-hospital transfers. Stroke is a medical emergency and as expected, most hospitalizations were through an emergency department in BC and NS, even when there was no NACRS record. Prior work using Alberta administrative data, where reporting to NACRS is complete, showed that PaSSV is associated with clinical outcomes after stroke. ²¹ However, in BC and NS, reporting is incomplete, and hospitals that do not report to NACRS tend to be smaller centers located in rural areas, and as a result, patients with stroke living in rural area were more likely to be excluded from our analyses. This is particularly regrettable because rural residents have been shown to have reduced access to standard stroke investigations and services. ²² Incomplete reporting to NACRS in BC and NS creates additional challenges to province-wide quality evaluations. Analyses in NS were particularly affected because only 41.9% of patients with stroke could be included in this study. Thus, our results may not be generalizable to the entire province. We hope our findings will encourage more provinces to fully mandate reporting to NACRS for all emergency departments.

In addition to incomplete NACRS coverage, our study had other limitations. We did not have access to observed clinical stroke severity and therefore could not directly compare PaSSV to observed stroke severity, but these comparisons have been previously reported. ^9,21 We also acknowledge that certain components of the composite PaSSV score, such as inter-hospital transfer to a hospital with higher level of care or use of mechanical ventilation, reflect hyperacute stroke care. Nevertheless, hyperacute care decisions in clinical practice are most often guided by the severity of the stroke and we showed that these can be used as proxy measurements for stroke severity.

Conclusion

Our findings suggest that PaSSV, a measure of stroke severity derived from administrative data, has geographical and temporal external validity across multiple Canadian provinces, making it a valuable tool for risk-adjustment in stroke outcomes research. We recommend the use of a clinical measure of observed stroke severity where possible, but in the absence of this information, the use of PaSSV through linkage with administrative health data is an alternative validated option.