Cognitive remediation in large systems of psychiatric care

Abstract

Introduction

With the increasing enthusiasm to provide cognitive remediation (CR) as an evidence-based practice, questions arise as to what is involved in implementing CR in a large system of care. This article describes the first statewide implementation of CR in the USA, with the goal of documenting the implementation issues that care providers are likely to face when bringing CR services to their patients.

Methods

In 2014, the New York State Office of Mental Health set up a Cognitive Health Service that could be implemented throughout the state-operated system of care. This service was intended to broadly address cognitive health, to assure that the cognitive deficits commonly associated with psychiatric illnesses are recognized and addressed, and that cognitive health is embedded in the vocabulary of wellness. It involved creating a mechanism to train staff to recognize how cognitive health could be prioritized in treatment planning as well as implementing CR in state-operated adult outpatient psychiatry clinics.

Results

By 2017, CR was available at clinics serving people with serious mental illness in 13 of 16 adult Psychiatric Centers, located in rural and urban settings throughout New York state. The embedded quality assurance program evaluation tools indicated that CR was acceptable, sustainable, and effective.

Conclusions

Cognitive remediation can be feasibly implemented in large systems of care that provide a multilevel system of supports, a training program that educates broadly about cognitive health and specifically about the delivery of CR, and embedded, ongoing program evaluation that is linked to staff supervision.

Introduction

Cognitive remediation (CR) is an evidence-based, recovery-oriented behavioral intervention that focuses on reducing cognitive deficits and enhancing life skills in people with serious mental illnesses (SMI). CR addresses the specific cognitive capabilities that are frequently impaired in people with SMI. Distinctly different from cognitive behavioral therapy, which focuses on the content and form of thoughts and beliefs, CR targets the information processing skills that underlie thinking; common targets include attention, memory, executive functioning, working memory, speed of information processing, and reasoning, and may include social cognitive processes such as facial affect recognition and theory of mind. ¹ Cognitive impairments are persistent, unresponsive to pharmacologic treatment, and constrain psychosocial achievement at tremendous financial, social, familial, and personal cost. ^{2 , 3} Cognitive deficits add to illness burden by imposing significant limitations on the ability to adhere to pharmacotherapy; benefit from psychosocial programs; and function in the social, vocational, and educational venues integral to community integration. ^{4 – 8} Cognition is thus a key intervention target because cognitive skills are needed to help people with SMI achieve their recovery goals—to work, go to school, live independently, and socialize. With improvement in cognition comes hope, greater social connectedness, improved relationships, empowerment, independence, motivation, and rebuilding of supports. ⁹ The ultimate purpose of CR is to enable recovery.

An increased focus on cognitive health and the implementation of evidence-based CR complements care practices that address physical and mental health in routine practice settings. CR programs are scalable, making them cost-effective ¹⁰ in addition to being efficacious. The literature supporting CR efficacy for SMI has been reported in several meta-analytic studies. Consistent evidence from randomized controlled trials demonstrates significant, durable improvements in cognition and daily functioning in people with schizophrenia ^{11 – 16} with additional evidence indicating that cognitive gains translate to improvements that impact quality of life. ¹⁷ Cognitive remediation is linked to improvement in social functioning, psychiatric symptoms, and vocational outcomes. ^{18 – 22} Further, emerging evidence indicates that CR programs are effective for first episode psychosis ^{23 – 25} and for people with affective disorders. ²⁶ Thus, CR has the potential to effectively and sustainably improve cognitive processes across populations with SMI by harnessing the potential for neuroplasticity that exists even when psychiatric illness impacts brain functioning, thereby enabling the use of adaptive cognitive skills in personally meaningful ways.

Emerging data on the consumer’s subjective experience of CR complements the empirical evidence supporting its efficacy. The learning benefits reported by participants include gains in specific cognitive domains, as well as in strategy use and skills to aid social abilities and everyday functioning. ^{27 , 28} The social experience of participating in CR in a group setting, as well as the positive, motivating influence of the therapist, have been reported as impacting engagement, self-efficacy, and learning outcomes. ^{27 , 29 , 30}

The corpus of data has generated great enthusiasm about CR, and the field continues to gain momentum. Scientific advances at the intersection of cognitive neuroscience and psychiatric rehabilitation allow for innovation in cognitive learning platforms and opportunities to develop and test personalized CR approaches. While laboratory-based developments aim to enhance cognitive learning, ongoing work in the field aims to enhance the treatment’s accessibility and its individual-level effectiveness in improving both cognition and recovery outcomes. ^{31 , 32} Indeed, there is increasing enthusiasm for implementing theoretically and empirically grounded CR approaches in large systems of psychiatric care. However, there is a lack of clarity about a number of factors that impact how CR can be feasibly implemented in clinical settings. Common questions include the following:

• ∙ What exactly is CR and what happens in a CR session?

• ∙ Are there different CR approaches, and if so, how does one choose?

• ∙ Who will be eligible to receive the service?

• ∙ Does it work for everyone?

• ∙ Is it covered by insurance?

• ∙ Who would provide it?

• ∙ How do we implement it?

In July 2014, New York State’s Office of Mental Health (OMH) and its academic partner, the Department of Psychiatry at Columbia University, created a Cognitive Health Service that could be implemented throughout the state-operated system of care. OMH serves over 700,000 people annually in New York, a state of about 19.5 million. In addition to licensing mental health agencies across New York State, OMH operates emergency, inpatient, outpatient, forensic, crisis, and residential services for adults and children, as well as 2 research institutes. As the largest state mental health system in the United States, it is relatively unique in breadth of geographic regions, and cultural and linguistic groups served. The Cognitive Health Service was intended to broadly address the cognitive health needs of patients, to assure that the cognitive deficits commonly associated with psychiatric illnesses were being recognized and addressed, and that cognitive health was embedded in the vocabulary of wellness. This involved creating protocols to better recognize cognitive health impairments and establishing the first known statewide-level implementation of cognitive remediation programs. ³²

OMH intends to eventually make CR available on inpatient and outpatient services for children and adults. However, it was decided to stage implementation beginning with outpatient clinics serving adults, ages 18–65, with the predominant diagnoses consistent with a serious mental illness. Currently OMH has 16 adult Psychiatric Centers (PCs), which provide care to almost 20,000 unique adult consumers across 62 clinics in rural and urban settings.

The purpose of this article is to describe the implementation process, with the goal of informing future implementation and dissemination efforts. Qualitative and quantitative data demonstrate the contextual challenges associated with the implementation of CR and the strategies used to enhance feasibility of uptake in a large system of psychiatric care.

Conceptualization and Organization

Overview

The initiative to implement CR in outpatient clinics throughout the state required significant organization and administrative support. A 3-tiered model of oversight governed the implementation process, coordinated through the Office of the Medical Director and the Office of State Operated Adult Services; leadership was represented at the state level by Central Administration, as well as within each local facility and clinic. Figure 1 illustrates the responsibilities assumed by each. Prior to implementation, a leadership team (authors AM, MDE, and LIS) first conceptualized the treatment model and presented a white paper and budget plan. During implementation this team identified local and central leaders, content experts, and vendors; located initial OMH state-operated adult outpatient sites for a staggered rollout of CR; and readied facilities for the necessary staffing, training, and infrastructural requirements as required by the treatment model.

Figure 1 Model of implementation oversight.

The conceptual model of CR

A first step of the leadership team was to align the model of CR that would be implemented with the overall goals of outpatient psychiatric care. Conceptualization of the treatment model helped to define a culture of cognitive health at OMH and operationalize methods of client referral, screening, and assessment, as well as staffing needs.

At OMH, a primary goal of outpatient treatment is to provide recovery-oriented, patient-centered treatment to facilitate maximal productive engagement in the community with as much independence as possible. CR services were being provided to further that goal. Research indicates that CR is more likely to impact cognitive symptoms, community engagement, and functional outcome when it is provided in a rehabilitation context. ^{15 , 33} Rehabilitation programs emphasize a multidisciplinary approach, skills acquisition, independent living, consumer advocacy, and pharmaceutical treatment in combination with behavioral intervention. The preferred model of CR was one that could integrate easily within the outpatient rehabilitation context and that emphasized the improvement of cognition as a way to enhance attainment of overall recovery goals. The CR service was expected to complement and enhance current treatments. Maximizing wellness, addressing cognitive deficits, and improving patient-centered recovery goals were identified as the 3 pillars of the CR initiative. OMH central leadership named the initiative Cognitive Remediation to Promote Recovery (CR2PR), to reflect that it would serve patients with SMI identified as having cognitive impairments that hamper recovery goal attainment. With this name, the initiative tied cognitive health to functional outcome, and squarely placed CR as a recovery oriented treatment.

A second goal of outpatient treatment at OMH is to improve the quality of behavioral and cognitive health of all community-dwelling public sector clients. The OMH leadership team advocated for a CR program with a “public psychiatry” focus, whereby most adults within the state-operated system would be eligible to receive services. The program therefore had to be accessible in rural and urban areas, and available to the multilingual populations served, with few exclusion criteria. The CR program needed to be flexible so it could be adapted to client and clinic alike. Recognizing that avolition is a primary issue for public sector clients with severe mental illnesses, ^{34 , 35} and that low motivation interferes with treatment compliance and learning of skills to promote recovery, ^{36 , 37} the model of CR needed to address impairments in motivation alongside cognitive dysfunction. The goal that participation in the CR service be experienced as enjoyable by the clients was important to facilitate treatment engagement and learning outcomes.

Finally, feasibility factors were considered when choosing a model of CR implementation. The service needed to fit within the constraints of Current Procedural Terminology (CPT) coding requirements so that it would be billable and therefore self-sustaining. Staffing, technical requirements such as web connectivity, and space requirements also informed the choice of CR model. Research on home-based computer or iPad delivered CR is an exciting frontier, but at this juncture such models of delivery are incongruent with the realities of internet access and codable/billable service implementation in the public sector. On the other hand, models of CR that conform to CBT group-based treatment are codable and thus sustainable as an implemented service.

The Neuropsychological and Educational Approach to Remediation (NEAR) ²⁸ is the approach considered to best fit these guidelines. NEAR has been used in multiple clinical settings, geographic regions, cultures, age groups, and languages and thus provides the flexibility and “public psychiatry” mandate of being a CR service that is client inclusive and accessible. With ready-made clinician training curriculums and manuals, inclusive of web-based training platforms, there is a systematic and efficient process to train clinicians. NEAR implementation kits provide training in the set-up of quality assurance and treatment fidelity monitoring so that programs start with a mechanism in place to continually improve service delivery.

The theoretical framework of NEAR is based on neuropsychology, educational psychology, learning theory, and cognitive psychology. In NEAR, psychiatric rehabilitation is viewed as a learning process, and the client is considered a learner—a participant in a learning process. The theoretical foundation of NEAR emphasizes motivation, and the teaching techniques are designed to promote motivation to learn. Each session is designed to be a positive, personalized, recovery-oriented learning experience that targets the cognitive skills identified as impeding progress toward individuals’ recovery goals.

Specifically developed for psychiatric patients, NEAR is a group-based treatment that provides highly individualized learning by allowing each person in the group to work at his or her own pace on tasks carefully chosen to be engaging and to address particular cognitive needs. Sessions are conducted in groups of 6–8 clients, which has both therapeutic and administrative advantages. In a group, a sense of community often develops among the clients because they share the same valued activities of learning and self-improvement. Because clients work at different tasks and rates, the groups have a rolling admission. It is not administratively advantageous or clinically responsive to wait to start a group until all 6 people have been screened for admission. Rather, the groups slowly develop as the clinician enrolls more people. Clients participate in sessions at least twice each week, and the sessions are typically 60 minutes long. About three-fourths of the session involves individual cognitive activities on the computer, and one-fourth is spent in a bridging group, which is a verbal group for all the participants to discuss how the computer activities relate to real-world situations. The bridging group discussions also promote metacognition by teaching how cognitive skills are used in everyday life, and by providing in vivo activities to promote cognitive skill development.

Creating a culture of cognitive health

Implementation of CR in a large system of care begins with creating a culture of cognitive health which brings awareness to cognitive impairment as a barrier to recovery and a key target of intervention. This often requires education to develop a common language around cognition that is shared by administrators, clinical care providers, patients, and their families. Shared understanding of what is meant by cognitive symptoms and their impact in real-world contexts fosters coordinated, person-centered care by providing a unified context for recognizing and addressing cognitive problems from multiple vantage points. Cognitive awareness promotes the integration of cognitive health building practices, including but not limited to CR, while also impacting down-stream care across behavioral health disciplines (eg, attention to pro-cognitive pharmacologic treatment, improved physical health, vocational and educational support). The culture of cognitive health and a common cognitive lexicon facilitates referrals to a CR program, shared decision making for cognitive intervention(s), recovery-oriented treatment planning, the therapeutic alliance, and outcomes assessments to ensure program efficiency and effectiveness. ³⁸ The CR2PR initiative sought to create a culture of cognitive health throughout OMH and relied on clinic leadership to uphold the local culture, thereby facilitating the development and growth of each CR program.

OMH created THINKING WELL, a model educational campaign that focuses on the need for cognitive health to make a good recovery. This initiative includes education about cognitive health and wide-scale enhancements for the delivery of treatments that promote optimal cognitive health. Components of THINKING WELL include literature and informational sheets for providers, patients, and families; a website for providers; and statewide archived Grand Rounds, which are web-based free lectures and clinician focus groups to inform OMH staff about implementation efforts.

Gateway to CR services

When implementing a service, key questions concern the treatment gateway: Who will be eligible to receive the service? How will consumers be identified, referred, and screened? What assessments will be needed? In keeping with the recovery orientation of the treatment model, the referral process was to identify all clients with cognitive problems interfering with attainment of recovery goals. The central question was: Does this person have memory, attention, processing speed, or other cognitive problems that are making it difficult to go back to school or work, manage symptoms, or function independently in their home? People with intellectual disabilities or less than a 4th grade reading level were excluded from CR and instead referred to services for people with developmental disabilities. Clients who would be better served by a CR program for people with neurologic disorders (eg, Traumatic Brain Injury) were to be appropriately routed to care and not referred to CR2PR, which was designed for people whose cognitive problems stem from psychiatric disorders. Actively symptomatic clients were welcome as long as they had sufficient behavioral control to participate in a 1-hour group session.

Operationalization of the referral and screening process aimed to facilitate a steady stream of appropriate referrals, with few barriers to compromise accessibility and efficiency. Referrals from all clinical staff as well as clients and support systems would be accepted. The screening process was to adhere to shared decision making guidelines, whereby client and clinician identify and agree on recovery-oriented goals and subsequent steps to promote goal attainment.

The staffing model

The CR program was designed to be sustainable by billing for testing and the CR group, with one group meeting twice weekly with 6–8 clients, requiring a total of .15 full-time effort cumulatively across providers and administrators. The staffing model recognized that each clinic where CR2PR was to be implemented needed a “champion” to be the primary leader for the CR service and an advocate for the integration of CR as a new clinical practice. Psychologists were selected as champions for this initiative, as within OMH they are recognized as the team member with the theoretical and clinical background to easily learn and help other team members understand the importance of addressing cognitive health, as well as having expertise in cognitive assessment and behavioral therapies. Psychologists thus would be both the champions and the persons who performed cognitive assessments to inform the treatment plan. All other clinical staff with at least a master’s level of education in clinical psychiatric care (nurses, physicians, psychologists, social workers, mental health counselors) were eligible to conduct CR sessions. Finally, a clinical administrator was identified as the team member who facilitated the implementation and sustainability of the service.

Implementing CR Services for Outpatients

Identifying sites

The OMH central office leadership team took a stepped approach to identifying which of the 62 outpatient clinics within 16 statewide adult Psychiatric Centers (PCs) were best suited to provide CR2PR. In step 1, phone calls with PC administrators and medical directors identified 13 PCs across the state as having the staffing and interest to implement CR2PR. In step 2, clinics within these PCs were considered for CR2PR implementation based on leadership, staffing, the availability of skills training and recovery-oriented services, the infrastructure to support CR2PR activities, and the need for geographic diversity of sites, as well as socioeconomic, cultural, and educational diversity of the client population.

CR requires specific conditions that affect its implementation in real-world settings that may differ from other evidence-based practices. Clinics interested in being a CR2PR site completed a Facility Needs Assessment (Table 1) which helped to concretize program needs, and alerted sites and central office when the facility had to make adaptations in order to implement CR2PR. Of the conditions queried, the most common adaptations were the need to acquire more computers and establish web connectivity for computer-based cognitive exercise. In large public systems of care, this can be a time-consuming process, since internet security issues must be managed.

Table 1 Facility needs assessment

Ultimately, 16 clinics were identified as CR2PR implementation sites. These 16 clinics range in size, with the smallest serving 23 clients and the largest serving 299 per week. The mean weekly census across clinics is 138 clients and 92–100% of clients are diagnosed with a SMI. On average, the percentage of patients carrying a diagnosis of schizophrenia or other psychotic disorder is 66.3% (range 40–87%). Adults age 18–64 years constitute over 80% of the population served in all but one clinic, where 60% of the clients were ages 65 and over. Ten clinics are in urban areas, and 6 are in rural areas. One clinic serves a majority (58%) of non-English-speaking clients, while the other 15 clinics serve a majority of English-speaking clients (range: 79–100%).

Implementation was done in two phases. In 2015, Phase 1 implementation commenced at 9 OMH-operated adult outpatient clinics that were attached to 8 unique OMH-operated PCs. In 2016, Phase 2 was launched at 7 additional OMH-operated clinics whereby 5 new PCs received the program and 2 pre-existing Phase 1 PCs implemented CR2PR in a second clinic. One Phase 2 clinic chose to delay clinical implementation until 2017 to become a site for a randomized clinical trial, which offered 2 types of CR (see clinicaltrials.gov NCT01945333), structured in the same format as the type of CR used in CR2PR clinics. By 2017, CR2PR was available at 13 of the 16 adult PCs, and 14 of the 16 chosen clinics were operating with CR2PR (including the clinic that transitioned from being a research site). Clinic closures impacted several CR2PR programs.

Staff training

A standardized, efficient staff training plan is needed when implementing CR in large systems of care. Experts in assessment and CR were identified as content leaders to provide training. To inform training needs, focus groups were conducted with staff before starting CR2PR. Clinicians reported an awareness of cognitive problems among clients, but uncertainty about etiology and treatment; although, for those who were aware of CR, the scientific evidence supporting its efficacy was perceived as moderate to strong. Clinicians were unaware of how to use cognitive assessments to create a CR treatment plan, how to conduct CR sessions, how to use web-based cognitive training exercises, and how CR can be a means to promote recovery. Questions about who to refer and who would benefit from CR were reported. The staff training was designed to provide education to define what cognitive health is, what CR is, and who benefits; how to conduct assessments for program entry; and how to conduct CR sessions. To ensure a solid referral base, educational activities for all clinic staff focused on how cognition impacts daily functioning and how CR can promote recovery.

Staff training draws clinicians away from other important duties, so it is essential that it be conducted efficiently. Multiple venues and resources were utilized (Table 2). CR2PR training included 1 in-person day of training for CR clinicians and champions, offered at 2 geographic locations. This was supplemented with webinars; internet-based learning at www.teachrecovery.com, a website built for CR training; monthly telephonic supervision; and lectures and workshops on assessment and CR session skills at Columbia University’s sponsored annual Cognitive Remediation in Psychiatry conference. Clinical administrators were provided written and verbal instructions, and periodic supervision on how to support the program and supervise staff. Because New York is a large state, it was important to minimize expensive and time-consuming travel to in-person training. Furthermore, with a 25% rate of staff turnover across sites, the web-based and telephonic training programs, coupled with train the trainer classes, could accommodate a constant influx of new clinicians.

Table 2 Training model

The training package specific to NEAR includes slide decks to teach didactics; forms for administrator, clinician, and client use; manuals; and competency exams to check knowledge. These training materials were adapted for use at OMH clinics to align with the state-operated quality assurance and record keeping standards. To facilitate dissemination of information for training and supervisory purposes, a website with CR2PR-related teaching and supervision materials was created on OMH’s internal website. Statewide Grand Rounds and site-specific in-services were used to disseminate information that would facilitate clinician knowledge about who to refer to CR2PR.

Program start-up at each site using web-based, face-to-face, and manualized implementation strategies took an average of 6 months. In Phase 1, 36 OMH staffers were trained to understand the details of cognitive health and to identify cognitive impairments. Additional training on cognitive assessment was provided for psychologists. In Phase 2, 42 new CR2PR staff members were trained to provide CR2PR. At least 30 CR2PR staff members attended the OMH sponsored Cognitive Remediation Conference annually from 2015 to 2017.

An essential part of the training was to familiarize the CR2PR clinicians with the use of web-based cognitive exercises and, in complement, how to provide coaching and support. Since CR2PR is highly personalized, the clinician selects CR exercises based on areas of cognitive strength and weakness revealed by the cognitive assessment, thus accounting for individuals’ learning needs and recovery goals. As a standard CR2PR training activity, clinicians sampled and analyzed exercises with respect to the primary cognitive targets (eg, attention, executive functioning, working memory, processing speed, verbal memory), prerequisite skills (eg, computer familiarity), task adaptability, level of therapist involvement, and overall strengths and weaknesses. Their task analyses were reviewed in supervision to ensure they understood how to choose an exercise for any given client, given their cognitive, learning, and recovery needs.

To evaluate this training component and inform further training needs, the ratings of 24 exercises from 3 web-based training programs were examined from 21 clinicians (eg, psychologists, social workers, nurses) in Phase 1 implementation sites. The specific quality improvement focus was whether clinicians could identify the cognitive target of each exercise. For example, did they recognize a verbal memory training task as different from a processing speed task? While the clinicians’ task analyses appropriately labeled exercises as targeting executive functions (eg, reasoning, problem-solving), memory and attention were often inaccurately under- or over-identified as cognitive targets. Without training, clinicians struggled to identify tasks that would help a client improve their memory or attention.

Without better recognition of cognitive skills in practice, clinicians may neglect to select the most appropriate exercises for a treatment plan, or inaccurately employ other exercises to meet specific cognitive needs. Treatment plans may be less efficient, personalized, effective, and therefore less recovery-oriented. Clinicians’ ability to match exercises and cognitive targets is also necessary to facilitate clients’ meta-cognitive awareness of their own cognitive processes during CR and for teaching appropriate strategies for executing cognitive tasks. This training program evaluation study informed a need for ongoing in-depth training about cognition and the value of cognitive task analysis as a training tool.

Staff receptivity to CR

Staff receptivity to incorporating a new treatment modality was encouraged using basic principles of motivation:

1. 1. Clinicians’ perceptions of competency were supported by ensuring experiences of success in meeting feasible training benchmarks and by discussing positive clinical outcomes.

2. 2. Interest was promoted by employing training techniques that use contextualization, personalization, and control (eg, role-plays, discussing clinicians’ clinical experiences).

3. 3. Supervision provided an experience of caring.

4. 4. Clinicians’ autonomy was promoted by giving choices in the benchmarks and use of materials, and by celebrating creativity while maintaining treatment fidelity.

Staff were offered the chance to participate in CR2PR but respected if they declined. Those who went through the motions of engaging in the training or setting up a program without producing clinical results were encouraged to get involved in other activities to give staff with interest the opportunity to participate. Ongoing supervision of staff and management emphasized interest in the program as opposed to compliance with regulations. Opportunities to showcase their work within and outside of the system were provided.

CR2PR program phases

CR2PR was conceptualized as having 3 program phases: entry, treatment, and exit (Figure 2). The entry phase entails a referral, screening, and assessment. Referrals can be made by any clinical team member by completing a referral form, and the CR clinicians are encouraged to remind staff about the CR2PR program during staff meetings and clinical case conferences. Clients are able to self-refer, in which case their care coordinator is asked to complete the referral form. The initial screening is a brief meeting between the CR2PR clinician and client to review their goals, availability, and interest in the program. The cognitive assessment includes a brief history and 40 minutes of cognitive tests.

Figure 2 CR2PR treatment model.

The CR2PR treatment phase generally consists of 30 sessions conducted twice weekly, although length of treatment can be tailored to individual needs. Clients are asked to commit to attending blocks of 10 sessions that are conducted in groups of 6–8 participants. Each session is 60 minutes, including 45 minutes devoted to restorative web-based cognitive training exercises and 15 minutes of bridging group discussion. Admission to the group is on a rolling basis, and during the computer exercise portion of the session, participants work at their own rate on exercises identified as useful to them based on their assessment and recovery goals. One clinician leads the group.

The exit phase is marked with a graduation ceremony where clients are given certificates of completion. Review of progress and next steps engages all group members and serves to promote motivation and engagement in learning.

Formal assessment

The CR2PR program includes a structured assessment of cognitive functions to accompany subjective reports of cognitive problems. The goal of the assessment is to provide objective evidence of cognitive deficits to guide the CR2PR treatment plan. While brief assessment batteries exist, none met the feasibility and clinical criteria for this project. Therefore, the CR2PR clinical director (author AM) worked with an outside content expert to refine a battery for OMH cognitive health services. Brief Assessment of Cognition in Schizophrenia (BACS) ³⁹ and MATRICS Consensus Cognitive Battery (MCCB) ⁴⁰ subtests are used in a clinician-administered brief assessment of premorbid intelligence, working memory, verbal memory, attention, executive functioning, and processing speed. An outside vendor was contracted to train psychologists in the administration and scoring of the battery, and to develop a scoring platform that would generate an automated report used to identify pertinent cognitive targets. Clinician training in assessment was done entirely over the web, and certificates were given to those psychologists who passed the competency test. Once certified, psychologists were trained to develop treatment plans using the test results. One or 2 psychologists from each of the 16 clinics were certified as testers.

Quality assurance monitoring

Continuous quality improvement (QI) strategies were built into the CR2PR initiative to maintain programmatic fidelity, efficacy, and efficiency. At each CR2PR site, the clinicians designated as “champions” oversaw the implementation of QI data collection. Staff were trained to systematically track appointments, session attendance, and client progress, enabling monthly monitoring and the collection of data to generate a quarterly report on referrals, intakes, utilization, and discharges, as well as outcomes. Clinician and administrator perceptions of the implementation process are discussed during supervision with author AM. Service data and supervision of clinicians are used to monitor fidelity. Consistent with guidelines for program evaluation in clinical settings, ^{41 , 42} outcomes include a client-completed satisfaction scale and a clinician report on recovery progress both completed after 30 sessions. The outcome questionnaires, as well as utilization data from Phase 1 were previously published, ⁴³ and the addition of data from Phase 2 supported the perceived effectiveness of the treatment and high level of satisfaction with the program. ⁴⁴

As of summer 2017, QI monitoring at 13 sites (the 14th site was still transitioning from being a research to a clinical site) determined that 61% of sites were keeping their fill levels at 80%. At each site an average of 4.85 patients were enrolled in the CR2PR program at any given time; the goal was for an average of 6 people to be enrolled. On average each site received 4.15 referrals per quarter, and referrals were twice as likely to come from a non-physician than physician. Attendance at sessions was good, with 70% of scheduled sessions attended.

Treatment fidelity

Implementation of CR in large systems of care requires monitoring of treatment fidelity to ensure a standardized quality of service provision and to provide concrete feedback to clinics and clinicians to remedy service issues. A chosen fidelity measure must operationalize what is done during CR as well as how it is done, reflecting adherence to an empirically based model, as well as clinician competence in enacting the treatment principles and components. A fidelity measure should also capture how CR differs along critical dimensions from other psychosocial interventions. This enables the effects of CR to be specifically evaluated in terms of its effectiveness in routine care settings. While there is at present no standard CR fidelity scale, a measure developed for use in a clinical trial was selected and adapted for CR2PR. This measure was chosen, given a number of factors that will facilitate uptake of evidence-based CR in routine practice settings. First, key elements included in the measure are informed by the available evidence as impacting cognition and generalization. Second, the items and criteria are applicable, or are adaptable, to the realities of OMH clinic settings, while keeping within empirically based guidelines. Third, the scale is feasible to implement in the context of routine quality assurance monitoring and has utility value. The components are identifiable to a rater via records review, supervision, or live observation. Items are ratable at repeated time points, across phases of treatment. The scoring system is straightforward and easy to interpret to provide feedback to CR clinicians. Finally, having been developed for a multi-site clinical trial, there is a standardized procedure for rater training, which is similar that used in training CR clinicians.

The fidelity scale used throughout OMH clinics maintains the original scale format as a checklist of 4 criteria for each of 6 items such that the item score reflects the number of criteria met (0 to 4), yielding a possible total score (0 to 24) and an overall average item score. Three items—treatment structure and session organization, composition of the treatment group, and clinician competence—measure basic design elements informed by the available evidence-base (eg, dosage and treatment intensity). Criteria were, however, modified to reflect clinic-specific parameters, such as the duration of sessions and session components, and group size. Three items—computer-based learning activities, bridging group, and the learning context—reflect empirically based, defining elements of CR. Criteria measure the extent to which there is sufficient cognitive practice with explicit links to daily functioning and rehabilitation goals in support of generalization, empirically derived techniques are used to support intrinsic motivation, and the therapeutic context is conducive to learning. The criteria reflect group content and therapeutic techniques that distinguish CR from other skills-based interventions.

Fidelity ratings were attained by review of electronic utilization reports submitted quarterly and monthly phone supervision with CR clinicians. Within the first quarter of implementation, the most frequent lapse in treatment fidelity concerned identification of rehabilitation goals, which is important for linking CR to psychiatric rehabilitation planning and treatment. This was corrected via clinician supervision, resulting in 100% treatment fidelity at each clinic for all elements by the next quarter. Subsequent lapses in fidelity arose when there were gaps in web connectivity at selected sites, or when there was staff turnover. These temporary problems were corrected with supervision and local and/or central office support. CR treatment fidelity can thus be feasibly assessed in routine care practices and inform clinical supervision to promote systematic implementation of evidence-based CR practice.

Discussion

This article describes the process of implementing CR in a large public psychiatric system of care. There is considerable consumer and provider enthusiasm for CR, but relatively little is known about how to best implement it in mental health care systems. Research has highlighted the importance of offering CR within a larger recovery context ^{15 , 33} so that the cognitive improvements translate into functional gains. Other essential implementation elements identified by CR implementation initiatives in other countries include the need for a common language among stakeholders to discuss cognitive health, the importance of timing participation in CR so it maximizes the overall recovery process, and the value of having strong program leadership. ³⁸ However, the exact steps of implementation remain a mystery to many stakeholders, making a process paper timely.

The described implementation effort included focus groups, needs assessments, and ongoing program evaluation, which inform the lessons that can be learned. We believe these lessons are as follows:

1. 1. Stakeholders (clients and staff) need familiarization with cognitive health services to support referrals and utilization.

2. 2. The context or the quality of the environment where CR is implemented depends upon multiple factors, from leadership to IT systems to staff availability.

3. 3. CR programs are not “turn-key,” in that clinicians require specific training.

4. 4. Ongoing program evaluation is critical for fidelity, trouble-shooting, and sustainability.

CR is one of many ways to address cognitive health, and both staff and clients require education to identify who is most likely to benefit. Educational campaigns help assure that everyone understands what cognitive skills will be addressed in CR and why, what the treatment involves, and the outcomes that can be expected. While we found that utilization of the service and satisfaction surveys reflected the acceptability of CR2PR, the program fill rates were negatively impacted by lower than expected referrals. With constant staff turnover, there is a need for ongoing education about cognitive health and the role CR can play in recovery.

This project also highlighted the multiple factors that can impact whether the CR environment is supportive enough to sustain the program. Dynamic leadership is key but so is having a culture where cognitive health is recognized by providers. Technology support is essential to ensure consistent to access the web-based cognitive exercises. Adequate staffing to run the CR program is obviously needed, but the effects of staffing in areas that complement and support the service are also impactful.

CR is a relatively new treatment modality, and mental health clinicians are not taught to provide CR in school; they must learn about it on the job. Implementation of CR in systems of mental health care requires operationalizing training for CR clinicians. They require foundational knowledge about cognitive skills and CR, as well as training in the use of cognitive exercises and compensatory techniques, and in the teaching techniques that promote motivation to learn. Training needs to proceed at a pace that allows the clinician to practice what is learned, and ongoing supervision is essential for treatment fidelity. To make training assessable and effective, it optimally combines face-to face and remote learning, and uses competency exams and fidelity monitoring to assess learning.

Ongoing program evaluation is critical for fidelity and program sustainability. Staff can be trained to provide program evaluation data from the outset, and supervisors can be trained to use this data to monitor fidelity and sustainability. Program evaluation data should monitor fill rates, utilization, acceptability, and outcomes. Facility needs assessments and site self-evaluations can serve to mobilize local and central support systems to address issues that may adversely impact the delivery of CR.

Conclusion

New York State’s OMH cognitive remediation initiative, CR2PR, demonstrates the feasibility of implementing CR in a large, geographically, culturally, and linguistically diverse system of care. It is important for research to continue to address ways to enhance CR efficacy, but we are now also at a juncture when research can focus on the factors that impact implementation. Process studies like this will hopefully prove useful in future CR implementation research.

Disclosures

Alice Medalia, Matthew Erlich, and Lloyd Sederer have nothing to disclose. Alice Saperstein reports salary support from the Pibly Fund during the conduct of the study.