Search

Summary

In Chapter 11, the author takes readers through the process of creating and using holistic and analytic rating scales for performance assessments. The author uses an example of an assessment that a team of language teachers created for measuring the abilities of young Saudi learners who were learning English as a second language in the United States. The author discusses multiple approaches to creating rating scales, including adapting existing scales, patterning scales after course standards, a theory-based approach where developers use language and assessment theory to create scales, and a performance-driven approach where developers use test taker language samples to identify distinguishing characteristics among test taker abilities. The author discusses three sample writing performances of the young Saudi learners and shows how the teachers used them to create the scales. The chapter concludes with a discussion of the process of creating and using computer scoring systems for performance assessments.

Introduction

The six-item Positive And Negative Syndrome Scale (PANSS-6) is short psychometric valid scale quantifying the severity of core schizophrenia symptoms. Using PANSS-6 to guide treatment decision-making requires that staff members’ ratings are valid and reliable.

Objectives

The objective of the study was to evaluate whether such valid and reliable PANSS-6 ratings can be obtained through a video-based training program.

Methods

One-hundred-and-four staff members from Aarhus University Hospital - Psychiatry, Denmark participated in the training. Participants conducted baseline PANSS-6 ratings based on a video of a patient being interviewed using the Simplified Positive And Negative Symptoms interview (SNAPSI). Subsequently, a theoretical introduction video was displayed followed by five SNAPSI patient interviews. After each SNAPSI video, individual ratings were performed before a video providing the gold standard scores was displayed. The validity of ratings was estimated by calculating the proportion of participants not deviating from the gold standard scores with >2 points on individual items or >6 points on the PANSS-6 total score. Reliability was evaluated after each step in the training by means of Gwet’s Agreement Coefficient (Gwet).

Results

By the end of the training, 72% of the participants rated within the acceptable deviations of the gold standard, ranging from 60% (nurses) to 91% (medical doctors/psychologists). The reliability improved (Gwet baseline vs. endpoint) for all PANSS-6 items, except for Blunted affect.

Conclusions

The majority of the staff members conducted valid PANSS-6 ratings after a brief standardized training program, supporting the implementation of PANSS-6 in clinical settings to facilitate measurement-based care.

Conflict of interest

Dr. Opler is a full-time employee of MedAvante-ProPhase Inc. Dr. Correll has been a consultant and/or advisor to or have received honoraria from: Acadia, Alkermes, Allergan, Angelini, Axsome, Gedeon Richter, Gerson Lehrman Group, Indivior, IntraCellular T

Search Results

Refine search

Refine search

Actions for selected content:

2 results

Chapter 11 - Scoring Performance Assessments: Rating, Rating Scales, and Raters

Summary

Standardized training in the rating of the six-item positive and negative syndrome scale (PANSS-6)

Search Results

Refine search

Refine search

Actions for selected content:

Save Search

2 results

Chapter 11 - Scoring Performance Assessments: Rating, Rating Scales, and Raters

Summary

Standardized training in the rating of the six-item positive and negative syndrome scale (PANSS-6)