Practicing safety: a quality improvement intervention to test tools to enhance pediatric psychosocial care for children 0–3 years

Abstract

Background

Child maltreatment is a significant public health issue in the United States. Yet, fewer than half of pediatricians discuss behavioral, developmental, or parenting issues with parents.

Objective

This paper describes the testing of bundles of tools and processes, part of a larger intervention, Practicing Safety, targeted at changing physician and staff behavior to identify families at risk for child maltreatment, provide anticipatory guidance, refer to community resources, and follow-up and track at-risk families. The intervention was implemented with 14 pediatric primary care practices throughout the United States; the study was completed in 2011.

Methods

A within-subjects repeated measures pre-post follow-up design was used to evaluate the intervention. Baseline and repeated measurements of pediatric practices’ processes were collected using qualitative and quantitative methods. In total, 14 core improvement teams from across the country tested three bundles of tools (maternal, infant, toddler) within a quality improvement framework over seven months.

Results

Quantitative results showed statistically significant adoption of tools and processes and enhancement of practice behaviors and office environmental supports. The increase in tool use was immediate and was sustained for six months after implementation. Qualitative data provided insight as to how meaningful the intervention was to the core improvement teams, especially with more complicated behaviors (eg, engaging social workers or community agencies for referrals). Barriers included lack of community resources. Findings showed unanticipated outcomes such as helping practices to become medical homes.

Conclusion

Lessons learned included that practices appreciate and can adopt brief interventions that have meaningful and useful tools and process to enhance psychosocial care for children 0–3 and that do not place a burden on pediatric practice. An innovative, quality improvement strategy, intuitive to pediatricians, with a brief intervention may help prevent child maltreatment.

Introduction

When we think of patient safety, child maltreatment prevention does not usually come immediately to mind. Yet the incidence and prevalence of child maltreatment, often unrecognized, is a significant public health issue in the United States. In 2013, Child Protective Services determined 9.1 out of 1000 children were victims of child abuse and neglect [US Department of Health and Human Services (U.S.D.H.H.S) et al., 2015]; young children were the most vulnerable with 27.3% of victims aged less than 3 (Norman et al., 2012; U.S.D.H.H.S et al., 2015). The National Survey of Children’s Exposure to Violence found a lifetime victimization rate of 25.6% with 13.8% of participants experiencing maltreatment in the past year (Finkelhor et al., 2013). Of children who have been neglected or emotionally abused, 80% were victimized by a parent/guardian (Gilbert et al., 2009). Long-term effects of childhood abuse include toxic stress and a sustained negative impact on adult health-related quality of life (Corso et al., 2008).

Factors increasing a child’s risk for maltreatment include a child’s emotional/behavioral issues or developmental disabilities, parents’ mental illness, substance abuse, limited knowledge or unrealistic expectations of normal child development and behavior in addition to poverty, single-parent home, and intimate partner violence (Flaherty and Stirling, 2010). Yet, fewer than half of all US pediatricians screen to detect development, behavioral, or social issues with standardized tools or discusses, with the majority of parents, issues such as how a child communicates his/her needs (0–9 months), toilet training (10–18 months), and getting along with others (Margolis, 2004; National Survey of Children’s Health (NSCH), 2013). Pediatricians may not recognize parental frustration related to discipline and crying or maternal depression as factors that may drive the risk of maltreatment. Less discussed topics include issues related to family needs and community resources such as social support, financial needs, and violence in the home or community (Hagan, 2001; Olson et al., 2004). The Adverse Childhood Events research demonstrate that child abuse, neglect, and other trauma, that disrupt the parent–child relationship, are significantly associated with adult health problems including alcoholism, depression, drug abuse, eating disorders, obesity, high-risk sexual behaviors, sexually transmitted infections, smoking, suicide, and certain chronic diseases (Felitti et al., 1998; Norman et al., 2012). Therefore, enhancing well-child visits may have long-term impacts on health and quality of life in the population of children as they age (Shonkoff and Garner, 2012).

The Institute of Medicine (IOM) report ‘Crossing the Quality Chasm’ lists the core needs for healthcare to be safe, effective, patient-centered, timely, efficient, and equitable (Richardson et al., 2001; Bethell et al., 2004). Although the report prescribes a blueprint for improving child healthcare interventions, the core needs have yet to be routinely incorporated into well-child care (Randolph et al., 2005). The IOM report can be used as a guide for pediatric quality improvement to reduce child maltreatment risks. The issue of safety and avoiding child injuries includes clinician recognition of familial stressors affecting the physical and emotional health and safety of children 0–3 years old. Effective care for children includes identifying the behavioral, developmental, and social functioning problems encountered in pediatric practice and linking at-risk children and their families to community psychosocial resources (Roghmann and Haggerty, 1972; Halfon et al., 2004). Patient/family-centered care for the youngest children requires assessing the family’s needs and understanding the importance of parental well-being, knowledge, and support. Timeliness of care includes increasing standardized developmental and behavioral screening often delayed due to time restraints (Halfon et al., 2004) or lack of confidence (Weitzman and Wegner, 2015). Use of electronic medical record (EMR) systems or alternative staffing roles and responsibilities utilizing physician extenders to perform assessments may improve timeliness. The use of EMRs and personal health records may also provide more efficient care by documenting and providing information electronically to parents/caregivers prior to, during, and in between visits (Richardson et al., 2001). Finally, child maltreatment assessment and safety prevention are often provided to families inconsistently or not at all depending on their personal characteristics (eg, race/ethnicity, socio-economic status). Use of universal and standardized screening/assessment, anticipatory guidance, referral, and follow-up tracking are strategies to provide equitable care to all patient populations (Schuster et al., 2000; Abatemarco et al., 2008).

Incorporating the IOM’s core health values, Practicing Safety (PS) is a pediatric intervention developed to promote routine and enhanced well care in the child’s medical home, that focuses on the primary prevention of child abuse and neglect by engaging all patient families and the community (Abatemarco et al., 2008). Practicing Safety processes are based theoretically on participatory action research (Simons and MacDonald, 2006), reflective adaptive practice (Crabtree et al., 2001), and complexity science (Stroebel et al., 2005). There are three bundles of tools and processes within PS (1) an infant bundle focused on crying, (2) a maternal bundle focused on maternal depression and attachment and bonding, and (3) a toddler bundle focused on toilet training and discipline. Each bundle includes a suggested set of tools and processes to be adapted to the individual pediatric practice and includes: assessment/screening, enhanced anticipatory guidance (ie, prescribed questions, prompts, and counseling), referral procedures, and follow-up protocols.

This paper describes the results of an implementation science study to evaluate the uptake of three bundles of tools, processes, and resources – one each for mother, infant, and toddler – within a quality improvement (QI) framework (Boonyasai et al., 2007). We sought to determine whether practices could implement tools for assessment/screening, anticipatory guidance, and documentation of tools and resources shared with families, as well as implement practice-level policy change through the use of QI methods and processes.

Methods

Practice recruitment

Pediatric practices were recruited from the membership of the American Academy of Pediatrics (AAP)’s Quality Improvement Innovation Network (QuIIN), which is comprised of pediatric practices throughout the US. Pediatric primary care practices are similar to general family/primary care practices only their patients are between the ages of birth to generally 18 years of age. Approximately 100 practices are members of the QuIIN network and are willing to participate in quality improvement projects that address the practices needs. We sought to recruit a diverse sample of practices by geography (urban, rural, and suburban), practice size, type of organization, and patient populations (Table 1). Physicians and their core improvement teams were asked to commit to test changes and make improvements in their care and delivery of topics related to child abuse and neglect prevention using the PS toolkit. Applicants identified a multidisciplinary core improvement team of three members – a physician, manager, and nurse or administrative or office support staff member – and documented commitment of the lead physician to support this project. For this project, practices assembled a core improvement team consisting of three members (clinical and non-clinical staff), including a QuIIN member primary care clinician. Other team members could include nurses or other clinicians and administrative or office staff members. Teams were expected to share successes, challenges, and lessons learned with other participant practices. Based on these criteria, 14 practices volunteered and were recruited into the study. Each core improvement team member signed a consent form that included specific, agreed-upon responsibilities listed in Table 2.

Table 1 Practice ownership, medical chart type, insurance status, and patient characteristics (n=14 practices)

Table 2 Each core improvement team agreed to the following expectations and activities

Pre-study activities

Core improvement teams participated in an orientation conference call facilitated by the project team. Baseline data were collected by the teams to assess their current level of standardized assessment/screening, anticipatory guidance, and parental education tools for psychosocial development issues (ie, maternal depression, crying, bonding/attachment, discipline, and toilet training). The teams completed and submitted a practice-based pre-inventory survey to assess existing systems.

Study design

This QI research used interrupted time series design in an attempt to establish a functional relationship between process changes over time within a system and variation in measured outcomes. Baseline and repeated measurements of the pediatric practices’ care processes, using patient chart review, were conducted over six months (baseline and monthly). Practices agreed and signed contracts stipulating they would pull random charts monthly. Since there was variation from month to month in the data from the chart reviews we believe that practices did follow instructions based on the agreement to pull charts randomly. The baseline and repeated measures allowed us to track changes in performance of the use of the bundled tools and processes associated with: assessment/screening, anticipatory guidance, resources, and referrals by pediatric physicians and staff with parents/caregivers (see Table 3).

Table 3 Tools and practices included in each of the maternal, infant, and toddler bundles

ACOG=American Congress of Obstetricians and Gynecologists; AAP=American Academy of Pediatrics.

In addition to the survey questions about referrals we also discussed, in the monthly calls, strategies that could help the practices track referrals (eg, use of spreadsheets or a registry application; assign staff to track/follow-up on referrals made at prior visit). The monthly report included a question on the practices ability to conduct follow up regarding referral(s) made at prior visit(s), with a Likert scale response of 1–5 (5=strongly agree, 1=strongly disagree). We used this question and the response to gauge the importance of this topic to be included in the monthly calls.

Post-intervention, core improvement teams conducted the final chart review repeated measurement, a post-inventory survey to assess current practice-based systems, and a tool evaluation survey.

Study methods and procedures

During the seven-month project (one pre-work month and six one-month action/testing periods), teams used a systematic approach to the prevention of child abuse and neglect through the use of quality improvement science methodology and implementation of three bundles (infant, mother/caregiver, and toddler) of tools and improvement processes. The core improvement teams, working in a learning collaborative, participated in a one and a half-day learning session and participated in monthly facilitated conference calls with other teams to share successful strategies and solutions for challenges. The learning session-oriented clinical teams to the PS intervention and use of bundles, trained them on data collection methods and project protocols, and explained the use of the model for improvement for testing and implementing practice-based process improvements. Since the study tools and surveys were piloted in this study, revisions of the tools were made during the team training with input from the teams. Teams collected data monthly from 20 random patient charts for each set of process measures and submitted monthly progress reports to share their tested cycles of change, challenges, and what they learned.

Teams were provided with monthly run-charts of their data to see their successes, where gaps existed and where their efforts could be focused. In addition, teams were able to compare their data to the other individual practices and the overall mean of all 14 practices. Measurement of progress and documentation of small tests of change using the project-specific tools and processes were required. The project timeline and activity framework is shown in Figure 1.

Figure 1 Timeline and activity quality improvement (QI) framework for core improvement teams. PDSA=Plan-Do-Study-Act

Data collection tools and outcomes

Each month teams collected and submitted two sets of chart documentation forms: 10 de-identified records for infants seen for their two-month well-child visits and 10 de-identified records for toddlers seen for their 18-month well-child visits. These data were used to determine an average frequency of use for each PS bundle of tools and care processes (assessment/screening, anticipatory guidance, and tool and resource use). There were 17 possible outcomes across the three bundles for each practice each month based on the age-specific chart review (Table 3).

Teams also completed monthly progress reports and were taught to measure their progress using the Plan-Do-Study-Act (PDSA) cycles. The PDSA is conducted by testing a change using the four steps of planning, trying out the method, observing the results, and then implementing what the team learned. PDSA is considered a method to implement change based on action-oriented learning. The PDSA is an easy to use method for implementing quality improvement projects. In our study core improvement teams identified an aim and developed measures to determine whether a change led to an improvement, then implemented the change in the real work setting. An inventory survey of tool use and implemented processes was conducted at baseline and at the completion of the project to determine pre-post changes in formal practice guidelines and policies within each team’s practice. This self-report survey was completed by the core improvement team members and was comprised of 35 dichotomous items (see Table 4) such as, ‘This practice screens every family about crying.’ These items consisted of five domains: (1) use of anticipatory guidance (four items); (2) practice guidelines (three items); (3) documentation systems (12 items); (4) practice policies (12 items); and (5) screening policies (four items). All surveys and documentation were completed by one or more members of core improvement team.

Table 4 Pre-post comparison of aggregated items in the self-report inventory survey by practices (n=12)

All significance tests are paired t-tests. Two practices did not complete the post-inventory survey despite continuing to be included in the monthly chart review analyses.

An interview guide was developed for the post-intervention in-depth interviews with each lead physician. The interviews were designed to provide additional insight into improvements made, challenges faced, and impact of the PS bundles on patient care and on practice transformation. The lead physician on each improvement team participated in the post-study, in-depth interview with the project evaluators.

Analysis

Quantitative

The analyses focused on detecting changes in the practices’ documentation percentages across care processes (assessment and screening, anticipatory guidance, and resource and tool use) for each of the three bundles (maternal, infant, toddler) (Table 3). First, each practice’s run chart was visually inspected for every outcome prior to any statistical analysis. This visual inspection suggested that analysis using three time points corresponding to baseline, three months and six months would adequately capture the nature of the changes seen over time. A series of one-way within-subjects ANOVA’s were performed. The primary outcome was the overall measure that averaged the percent of documentation across all bundles and care processes, and then each of the nine specific care processes documentation percentage was tested. Because of the multitude of outcomes, a more conservative α of 0.01 was used. The assumptions of normality, and sphericity for the ANOVA were tested using the Shapiro–Wilk and Mauchly’s tests, respectively. If the main effect of time was significant, pairwise comparisons were performed using a Bonferonni adjustment. For the self-report inventory survey, paired t-tests were used to assess change from baseline to post-intervention in the practice-based strategies/processes.

Qualitative

Post-intervention in-depth interviews were conducted by phone with lead physicians from each practice by the Principle Investigator and Lead Evaluator, both of whom are experienced qualitative interviewers. Both the interviewer and note taker took comprehensive notes that were used for review and analysis. The qualitative analysis was iterative, separately analyzed by the interviewer and the note taker, and based on the constant comparative method of Grounded Theory (Glaser and Strauss, 2009). Individual interview responses were reviewed; categories were identified, named, and coded as themes. Codes were compared to identify similarities and differences. Similarities between codes revealed categories consisting of major and minor themes. Major and minor themes were discussed, and the analysts compared findings and refereed differences in the analyses. These findings were also triangulated through comparison with two other qualitative data sources: the monthly progress reports submitted by each team and notes from the monthly practice project 60 min conference calls that occurred twice each month to accommodate core improvement team attendance.

Results

Quantitative

Descriptive statistics of information found in the charts based on assessment/screening, anticipatory guidance, resource and tool use within each bundle at each analytic time point are provided in Table 5. The main effect of time was significant for each measure (all P<0.01): the overall average and for each bundle by care process type: maternal screening and assessment, maternal anticipatory guidance, maternal resource and tool use, infant screening and assessment, infant anticipatory guidance, infant resource and tool use, toddler screening and assessment, toddler anticipatory guidance, and toddler resource and tool use (see Table 5). The pairwise comparisons were performed and the pattern of results was fairly consistent across all outcomes. Months three and six had significantly higher implementation compared with baseline but were not significantly different from each other (all significant P’s<0.01). The one exception was for Infant Anticipatory Guidance, where months three and six were not quite significantly different than baseline (P=0.073 and P=0.074, respectively).

Table 5 Means changes from baseline in the use of the maternal, infant, and toddler bundles of tools as assessed from through chart review

All one-way within-subject ANOVA’s were significant at P<0.01.

The practice inventory survey, developed to collect data about the use of bundle-specific tools and processes for maternal depression, crying, discipline, and toilet training, also captures processes and clinical behaviors such as whether practices provide anticipatory guidance, document their practices in medical charts, conduct psychosocial assessment, have specific policies to establish standards of care, and whether every family is screened for psychosocial issues. Table 4 lists the 35 specific items within each of the five measured domains and shows the significance of the changes observed from pre- to post for each domain. Two of the 14 practices did not submit the inventory data but submitted all other data.

Qualitative

In-depth interviews with each lead physician at the conclusion of the project provided insight into the changes, challenges, and unanticipated outcomes that occurred within each practice. Major themes uncovered in the analysis showed that physicians and their teams, across practices, reported that as a result of PS, the practices standardized the use of the bundled tools and processes, systemized anticipatory guidance, assessment/screening for risk, the use of resources and referral, and made changes to charts/EMR to fully integrate the tools and processes into practice for all children aged 0–3.

The qualitative analysis provided insight into how the practices transformed to provide more support for families at risk without putting more burden on the practices. A specific example of change that occurred is that practices noted strengthening linkages to community services. Practicing Safety is designed to allow practices to tailor the activities to their practice cultures and communities. When the tools and processes for community referral for psychosocial risks were introduced, core improvement teams were guided to identify how they would refer parents and children for community services. Practices have varied levels of access to community services depending on the type of practice, their geographic location, and their beliefs as to what services their patients need. While a university practice may have a social worker assigned to the pediatric clinic, a rural independent practice is likely to know all the service agencies’ workers by name. A suburban practice, however, may not know of a single agency or organization to refer their patient families when a risk is evident.

The qualitative data showed that each practice enhanced the level of referral to community services. If the practice had access to a social worker, they improved communication with that social worker. One physician said, ‘As a result of working with PS, we got a direct pager number for the social worker and she comes over to the clinic immediately.’ Another physician noted, ‘We don’t have good community resources for parent education and family support. So, when we identify a problem [mental health issue], we send her back to her [obstetrician] now.’ The practice noted that they had not screened for maternal depression previously and were frustrated by the lack of community services available to their patients. Yet another physician commented, ‘We now have names and contact information for therapists treating maternal depression.’ Eleven of the 14 practices found a social worker or a specific family support agency to refer families at-risk as a result of using the PS tools, and all 14 practices improved their outreach efforts to the community.

The physicians were also asked about the implementation of QI methodology with their core improvement team. They reported the usefulness of the model for improvement to test change beyond the core team by other physicians and staff. Physicians reported that now PDSA cycles are run ‘organically.’ One physician tested changes to their EMR and sought feedback from patients; changes were made based on both patient and staff feedback, and then he expanded testing.

Discussion

In this project, we demonstrated that the PS bundles of tools and processes have multi-level effects on pediatric practice. In our previous studies, we found that the intervention enabled practices to understand their organizational dynamics and how these dynamics effect care to begin to make practice change (Abatemarco et al., 2008; 2012). The current findings extend this previous work by showing that the intervention was effective in promoting the use of all three bundles of tools and processes to assess/screen, provide anticipatory guidance and resources to parents/caregivers, make referrals to community resources, and document follow-up.

Other research has shown the value of primary prevention in the prevention of child maltreatment in one clinic or in one state (Dubowitz et al., 2009; 2012). Our findings show that each of the three bundles of tools – infant, maternal/caregiver, and toddler – could be implemented in many different practice types. The project also included strategies for practice behavior change and for policy adoption. The data suggests bundles of tools and processes were adopted, at different levels, across all practices.

Our previous findings indicated that practice change can be intuitive if teams are reflective (Abatemarco et al., 2008). However, in this study we also found that practice change is enhanced by the use of brief easy to use tools when the tools are bundled and specific to maternal, infant, and toddler issues. In addition, teaching strategies for practice behavior change and empowering physicians and staff to use their data (ie, medical charts) to measure outcomes appeared to have a positive effect.

There are concerns about adding new screening and prevention to a primary care practice already feeling overwhelmed with volume and lack of time. A recent study of adding ‘Bright Future’ developmental and mental health screening demonstrated that the added content added only one extra minute to each well-child visit (Duncan et al., 2015). Most of the added content occurred by restructuring the work of the front desk and the receiving nurse. This actually created more staff satisfaction in their work and increased the productivity of the healthcare provider.

Challenges to implementation of PS were also noted. Specifically, physicians felt a need for ongoing guidance and support from the lead physician; this they deemed a necessity for practice change. Issues related to payment codes and reimbursements for this expanded care were also identified as challenges. There is a need for more multi-lingual, low literacy, and graphic materials. Both administrative and clinical competing priorities were noted, such as EMR implementation and H1N1 incidence requiring additional vaccination efforts which put additional strain on the practice staff. Lead physicians acknowledged concerns over diffusion to all staff and physicians. Two practices that were engaged throughout the intervention did not complete the post-inventory survey. However, because all data points were collected from these practices their data was included in the final analysis. It is worth noting that practice staff turnover is an issue for many practices and at the end of the project, team members for both practices who did not complete post data had left their positions. Otherwise, data collection was sustainable post-intervention and teams used their data to inform their work in this project.

The study has a number of limitations not uncommon in quality improvement research but nevertheless important to take into consideration. Selection bias is a concern, as this study was conducted in practices who are members of a quality improvement network and self-selected to participate, thus results may not generalize to other practices. Additionally surveys were study designed and not validated; and lastly practices chose their own charts to review. Our study was designed to determine if the tools to prevent child maltreatment could easily be implemented in busy pediatric practice environments and this study demonstrate that among self-selected practices there was a positive uptake of tools and processes. As to choosing one’s own medical charts to review, the core improvement teams had differing rates of uptake demonstrated by the chart data, as well as differences within each practice with different bundles of tools. Practices appeared to have chosen charts randomly, as noted when their practice rates were not as high as they would have hoped. Often during the monthly meetings practices teams shared openly with the other practices their barriers and discussed ways to improve uptake. Practices shared with each other how to improve implementation of the tools and practices. We believed this demonstrated the core improvement teams’ curiosity and openness as to their own struggles and barriers to implementation. However, practices did choose their own patient charts so there may be biases in their reporting of outcomes since they were invested in this intervention. Another limitation was that the qualitative interviews were done by the lead investigator and the quality improvement facilitator and the respondents may have tempered their responses as compared with an interview conducted by someone not directly associated with the project.

A number of unanticipated outcomes, although not objectives of the study, emerged as important themes. University-based practices noted that they incorporated the intervention into their nursing and resident education. Another unintended outcome reported was that the implementation of Practice Safety enabled practices to become medical homes. Physicians also noted that participation in PS increased confidence in families and professionals alike; the resources provided made discussions with parents easier and more comfortable. Physician comments included that participating in the project was ‘a great experience,’ ‘a worthwhile learning experience,’ and ‘beneficial and fun to do.’ They also noted that it created a bonding of staff toward the project goal. Physicians noted improved communication and relationships between various professionals within the practice, with other departments (social work, obstetrician/gynecologist, psychology, etc.), and parents/caregivers. Project facilitation, the learning session, and the facilitated monthly calls all supported the work within and among teams. One physician commented at the end of the project, ‘It’s not over; it’s just beginning…’ Although these were not objectives of the study, these examples showed how the intervention may be tailored to specific practice needs and cultures. The findings also indicate that if an intervention is meaningful to the practice and is not perceived to be a consequential burden on the practices, it can be organically and successfully implemented.

This study’s findings set the stage for research to study the effects of the intervention on parents and children. An outcome study is underway to measure the effects of Practicing Safety on well care visit attendance, adherence to the immunization schedule, and standardized developmental screening. Practicing Safety and other such projects should also undergo an economic evaluation to determine the cost of implementing such interventions. Finally, the long-term sustainability of this approach beyond six months is unknown.

The IOM report, ‘Crossing the Quality Chasm’ identified six aims to improving quality and safety. Transforming pediatric care to address these aims while integrating prevention of child maltreatment, toxic stress, and the lifelong effects of adverse childhood experiences is the next program initiative needed to enhance child health. Interventions such as Practicing Safety can enable and motivate pediatric practices to confidently implement brief tools and processes that enable them to provide enhanced anticipatory guidance, assessment/screening, and resources to parents to reduce risks (Felitti et al., 1998). Accessible tools and resources placed within an implementation science framework of quality improvement, such as Practicing Safety, may enable physicians and their staff to reconfigure their practice-based systems to decrease child maltreatment through safe, effective, patient-centered, timely, efficient, and equitable care for all patients.