The Canadian Triage and Acuity Scale (CTAS) is a five-level triage tool that is used to help prioritize the order in which emergency department (ED) patients should be seen. The objectives of this study were to determine the interrater and intrarater agreement of the 2008 CTAS guideline revisions by triage nurses and to compare agreement between triage nurses working in a small community ED and an academic ED.

Seventy-eight triage nurses assigned CTAS scores and free-text presenting complaints for 10 paper-based case scenarios. For five scenarios, the CTAS score should have remained unchanged from previous guidelines, whereas the other five scenarios should have been triaged differently based on the 2008 CTAS first-order modifiers. Thirty-three participants repeated the questionnaire 90 days later, and intrarater agreement was measured.

There was a higher level of agreement (κ = 0.73; 95% CI 0.68–0.79) for the five case scenarios, which relied on the older 2004 guidelines compared to the scenarios where the 2008 guidelines would have suggested a different triage level (κ = 0.50; 95% CI 0.42–0.59). For the 10 case scenarios analyzed, the free-text presenting complaints matched the Canadian Emergency Department Information System (CEDIS) list 90.1% of the time (κ = 0.80; 95% CI 0.76–0.84).

The reliability of CTAS scoring by academic and community ED nurses was relatively good; however, the application of the 2008 CTAS revisions appears less reliable than the 2004 CTAS guidelines. These results may be useful to develop educational materials to strengthen reliability and validity for triage scoring using the 2008 CTAS guideline revisions.