Evaluation of a Web-based learning system for skills in removing personal protective equipment for highly infectious diseases—A randomized controlled trial

Abstract

Objective:

Which educational method is best for 3-month retention of proper skills in removing personal protective equipment (PPE) in the setting of highly infectious diseases is unclear. We evaluated the effectiveness of a Web-based learning system after 3 months of use.

Setting:

One general hospital in Japan.

Intervention:

We conducted a randomized, nonblinded, parallel-group trial with 35 nurses using the substitution block method. At baseline, both groups received face-to-face training in putting on and removing PPE. The intervention group was given access to the Web-based learning system we developed using Modular Object-Oriented Dynamic Learning Environment (Moodle). After 3 months, we assessed both groups regarding knowledge and skills in removing PPE using a 34-point test, fluorescent markers, and video recordings.

Results:

Overall, 34 participants completed the trial: 16 in the intervention group and 18 in the control group. Postintervention knowledge test scores (1.3 vs −0.8; P = .013; effect size r = .42) and deviations from the required procedure (−5.4 vs 1.9; P = .001; effect size r = .55) were significantly better in the intervention group than in the control group. The number of contaminated sites (−0.5 vs 0.4; P = .128; effect size r = .26) and contaminated participants (−18.7% vs 11.1% decreased in the intervention group, and increased in the control group, although this was not significant (P = .242; effect size ϕ = .47).

Conclusions:

This learning system was an effective educational method in maintaining and improving knowledge of proper PPE removal skills. The number of deviations from the required procedure decreased, and this reduction continued after 3 months.

Clinical trials identifier:

Evaluation of the Web materials for learning PPE removal skills in the setting of highly infectious diseases. University Hospital Medical Information Network, UMIN000042725: https://center6.umin.ac.jp/cgi-open-bin/ctr/ctr_view.cgi?recptno=R000048767

Outbreaks of severe acute respiratory syndrome (SARS) occurred in 2003, outbreaks of Ebola virus disease (EVD) in 2014, and the coronavirus disease 2019 (COVID-19) pandemic in 2020. In response to these highly infectious diseases, various organizations, such as the Centers for Disease Control and Prevention (CDC) and the World Health Organization (WHO), have recommended that healthcare personnel take precautions in addition to those that prevent contact, droplet, and airborne (droplet nuclei) transmission. ^1–4

However, secondary infections and deaths among healthcare personnel caring for patients with highly infectious diseases were still reported because of lack of personal protective equipment (PPE), ways to remove PPE without self-contamination, and inadequate training. ^5–7 The SARS outbreak resulted in 1,707 secondary infections among healthcare personnel; by September 2015, there had been 1,049 secondary infections and 534 deaths among healthcare personnel from EVD. ^8,9 The COVID-19 pandemic is estimated to have claimed the lives of 80,000–180,000 healthcare personnel between January 2020 and May 2021. ¹⁰ The risk of infection among healthcare personnel with COVID-19 may be almost 10-fold higher than that for other groups. ¹¹

Better education is essential to ensure adequate knowledge and safe practices among healthcare personnel and to prevent secondary transmission of highly infectious diseases. ¹² Thus, it is necessary to identify the procedures with the lowest risk of contamination for putting on and removing equipment, the most effective teaching methods, and the most effective methods for mastering and retaining skills in putting on and removing PPE.

Active training in removing PPE and reducing the risk of infection, such as being given spoken instructions by experts and computer simulations, improved compliance with procedures up to 1 week after the intervention compared with traditional training. ^13,14 However, effective educational methods have not been developed for retention of PPE removal knowledge and skills. ¹⁵

In study, we sought to provide healthcare personnel with a Web-based learning system developed for teaching PPE removal skills for highly infectious diseases. We also evaluated its effectiveness on PPE removal knowledge and skills after 3 months of use.

Methods

Study design and settings

We conducted a randomized, parallel-group, controlled trial in a single general hospital in Japan between December 2019 and May 2020. We applied the following eligibility criteria: (1) possession of or access to an Internet-enabled communication device (eg, computer, smartphone, or tablet), (2) no difficulty in changing clothing, (3) no skin damage, (4) no pregnancy, and (5) no redness or rash on the skin during patch testing of the fluorescent marker used in this study.

Sample size and randomization

We calculated a sample size of 52 cases based on a systematic review ¹⁶ using G*Power version 3.1.9.7 software (significance level α = .05, power 1−β = 80%, effect size d = 0.8). We planned to include 95 participants. However, we were able to recruit fewer participants than we anticipated during the recruitment period (July–November 2019), so we included 35 nurses in the study, 17 in the intervention group and 18 in the control group. One participant from the intervention group could not be assigned to the study because of a job transfer. Although the planned sample size was not reached, the sample sizes for several variables had medium-to-large effect sizes, indicating the effectiveness of the intervention. One researcher enrolled participants and randomly assigned nurses to groups using the substitution block method. The nature of this study as a Web-based learning system intervention meant that we were not able to blind the participants or the researcher.

Selection of PPE and procedures for its application and removal

We selected the type of PPE, and the procedures for putting it on and removing it, by collecting and reviewing 44 cases, including CDC and WHO procedures. ¹⁷ We selected the PPE used for patients with EVD because this is a very serious disease and has been reported to cause secondary infections among healthcare personnel. The selection was supervised by a certified infection control nurse.

The PPE included coveralls (Dupont Tyvek Soft Wear III Type; JIS T8115:2015; Dupont, Wilmington, DE), an N95 respirator (3M Aura Particulate Respirator 9211+ N95, 3M, Maplewood, MN), outer aprons, face shield, goggles, double gloves, double boot covers, surgical cap, disposable scrubs, and rubber shoes. The PPE removal procedure included actions and aspects needed to avoid self-contamination (Table 1).

Table 1. Procedures for Removing Personal Protectice Equipment (PPE)

Note. ABHR, alcohol-based hand rub; hot zone, the most infectious area; warm zone, the second most infectious area; cold zone, the third most infectious area.

Web-based learning system

We chose the Modular Object-Oriented Dynamic Learning Environment (Moodle) as the learning management system. ¹⁸ The education module included 2 courses, putting on PPE and PPE removal, and it lasted ∼2 hours. The learning content included videos and PDF files. To predict the effectiveness of the system, we tested it with 7 participants who had experience in using PPE for highly infectious diseases. They used the system for 1 month. We then improved the system, based on their feedback, to reduce data volume and improve usability. To keep learners engaged in the learning process, we asked participants assigned to the intervention group to take a 20-minute test every 2 weeks, and to complete the entire course.

Intervention and study protocol

We started by carrying out a PPE removal skills knowledge test among both control and intervention groups. As a baseline, we then provided both groups with 60 minutes of face-to-face training in putting on and removing PPE. Third, we conducted a second knowledge test and a first assessment of PPE removal skills among both groups. Fourth, we provided the intervention group with exclusive access to the Web-based learning system we had developed. The control group did not have the right to access the Web system, preventing them from using this educational resource. Finally, 3 months later, we conducted a third knowledge test and a second assessment of PPE removal skills for both groups.

For the evaluation of PPE removal skills, we used the following procedure:

1. (1) Participants put on PPE after we read each step in the procedure out loud. We confirmed that the participant did not deviate from the procedure.

2. (2) Before performing each PPE removal procedure, participants applied fluorescent marker (Spectro-pro Plus, Moraine, Tokyo, Japan) to their gloved hands. They also added 2 mL fluorescent marker during steps 1 and 11 after the gloves were changed during the PPE removal procedure, and 0.5 mL fluorescent marker during the other procedures when the gloves had already had fluorescent marker applied.

3. (3) We ensured that the fluorescent marker was applied evenly to the palms and backs of the participants’ gloved hands and that it did not scatter and stick to the body.

4. (4) Participants removed their PPE after we read out each step. We recorded the procedure using 2 video cameras.

5. (5) After PPE removal, we defined contamination as the presence of 1 or more fluorescent markers on the skin or clothing of any of 39 separate parts of the body, and we checked contamination of participants in a dark room (Fig. 1).

   

   Fig. 1. Body parts separated into 39 for contamination measurement.

Outcome measures

We analyzed the following primary outcome measures: (1) knowledge test scores, (2) number of deviations from the standard procedures for removing PPE, (3) number of contaminated sites, and (4) number of contaminated participants. The knowledge test consisted of 34 questions that we developed based on the points to consider when removing PPE and the movements that pose a contamination risk during the process. We gave participants 1 point for each question they answered correctly, for a maximum score of 34 points. We assessed participants by conducting the test 3 times: before face-to-face training, after training, and after the intervention. The number of deviations from the procedure was based on an 80-item checklist developed from the PPE removal procedure. We used video camera recordings to verify the total number of sequencing errors and technical errors. We counted the total number of contaminated sites after PPE removal.

Secondary outcomes were PPE removal duration (in seconds) and usability of the Web-based learning system. We calculated the total duration for PPE removal using the video camera recordings, and we extracted the duration from touching the PPE to be removed to completing the removal process (the PPE being completely removed from the body). The usability of the Web-based learning system was evaluated using The System Usability Scale (SUS). ¹⁹ SUS contains 10 questions, each of which is rated on a 5-point Likert scale from 1 (completely disagree) to 5 (strongly agree). The data were converted into scores on a scale of 0 to 100, and the average SUS score was calculated. A higher numerical score value indicated better usability. In this study, the English version was translated into Japanese using double-back translation.

Data analysis

For the descriptive statistics, we used the Student t test or the Mann–Whitney U test to compare continuous variables (eg, age) and the Fisher exact probability test to compare sex between the groups. For the knowledge test scores, number of deviations from PPE removal procedures, number of contaminated sites, and PPE removal duration, we calculated the mean difference before and after the intervention and compared them using the Student t test or the Mann–Whitney U test. For the number of contaminated participants, we compared the difference before and after the intervention using 2-sample generalization of the McNemar test. We used IBM SPSS Statistics 26 software (IBM, Armonk, NY) for the analysis. We considered a 2-sided P value < .05 to be statistically significant.

Ethical considerations

This study was approved by the research ethics committee at the facility to which the researcher belongs (Juntendo University, approval no. 30-59; 2019-45). We explained the study in writing to the director of the nursing department at the target facility using a research cooperation request letter to obtain their consent, and the research description and consent form were distributed to all participating nurses. We obtained written informed consent from participants before including them in this study. We did not compensate participants for their involvement in this study.

Results

Participant characteristics

In total, 34 participants completed the study: 16 in the intervention group and 18 in the control group. Participants had a mean age of 43.6 years (SD, 4.2) and a mean clinical experience of 21.1 years (SD, 4.9; 20.8 in the control group vs 21.4 in the intervention group; P = .700). All participants were female and right-handed. Participants who had used PPE for highly infectious diseases had done so a median of 2 times (2.0 in the control group vs 1.5 in the intervention group; P = .737). For all experienced participants, it had been >1 year since their last use of PPE. The mean duration of intervention was 109 days (SD, 11.9; 111 in the control group vs 106 in the intervention group; P = .291). We did not detect significant differences between groups in any of the participants’ baseline characteristics. We verified the access logs for the intervention group’s completion of the 2 courses and the completion of the 20-minute test every 2 weeks and found a 100% compliance rate.

Contamination in 68 episodes of PPE removal

Across the 68 episodes of PPE removal that were evaluated (34 before the intervention and 34 after the intervention), contamination during the process was found in 109 sites. Before the intervention, 9 (26%) of 34 front-right shoulder surfaces were contaminated; 6 (18%) of 34 left palm, face, and front-left shoulder surfaces were contaminated; and 5 (15%) of 34 right palms were contaminated. After the intervention, 9 (26%) of 34 front-right shoulders were contaminated; 6 (18%) of 34-front left shoulders, left palm, and right palm were contaminated; and 5 (15%) of 34 back-right palms were contaminated.

Overall, >70% of the participants became contaminated during a procedure: 25 (73.5%) of 34 before intervention and 24 (70.6%) of 34 after the intervention.

Outcome variables

We did not detect a significant difference in knowledge test scores between the 2 groups before and after the face-to-face education (6.9 vs 4.8 points; P = .058) (Table 2). However, the mean difference before and after the intervention was significantly higher in the intervention group than in the control group (1.3 vs −0.8 points; P = .013; effect size r = .42). The number of deviations from the PPE removal procedure was significantly lower in the intervention group than in the control group before and after the intervention (−5.4 vs 1.9; P = .001; effect size r = .55).

Table 2. Outcome Variables Before and After the Intervention During the Randomized Controlled Trial of the Web-Based Learning System

Note. SD, standard deviation; CI, confidence interval.

^a Student t test, M (SD), t, effect size r.

^b Mann–Whitney test, M(SD), z, effect size r.

^c Fisher exact test, n(%), χ², effect size ϕ.

^d 2-sample generalization of the McNemar test, no. (%), χ², effect size ϕ.

We did not detect a significant difference in the number of contaminated sites between the 2 groups before the intervention (1.6 vs 1.6 sites; P = .708). The mean difference before and after the intervention varied between the intervention and control groups, but not significantly (−0.5 vs 0.4 sites; P = .128; effect size r = .26). Overall, 68 (62%) 109 contaminated sites (36 of 68 sites in the intervention group and 32 of 68 sites in the group) observed in the evaluation of PPE removal skills were not considered when removing PPE in guidelines such as those from the CDC, nor were they included in the educational content in this study. We identified 3 procedures and 6 situations related to contamination:

1. (1) Removal of N95 respirator

   1. a. Hand with contaminated glove and the glove itself touched clothing.

   2. b. Hand with contaminated glove touched surgical cap.

2. (2) Removal of surgical cap

   1. a. Hand with glove touched head when removing surgical cap.

3. (3) Removal of gloves

   1. a. With 1 gloved hand, fingers touched the outside of the glove when removing the other glove.

   2. b. When removing 1 glove, the sleeve was undone and the inside the contaminated glove touched the bare hand.

   3. c. After removing both gloves, the sleeve area of the contaminated glove was held and touched the bare hand.

The number of contaminated participants decreased by 18.7% in the intervention group before and after the intervention and increased by 11.1% in the control group, but this was not significant (P = .242; effect size ϕ = .47). PPE removal duration increased in the intervention group and decreased in the control group before and after the intervention, but this was not significant (3 s vs −35 s; P = .148; effect size r = .25). The response rate of the SUS survey administered to the intervention group after the intervention was 87.5% (14 of 16), with a valid response rate of 100%. The average SUS score was 76.3 points (SD, 13.9).

Discussion

After 3 months, the intervention group showed that they had maintained and improved their knowledge more than the control group. The intervention group also showed significantly fewer deviations from the PPE removal procedure than the control group. This result is similar to those of previous studies showing that active training is effective in improving knowledge and decreasing the number of deviations in PPE removal procedures. ^13,14,20,21 However, to the best of our knowledge, this is the first study to confirm through a randomized controlled trial that active training using a Web-based learning system is effective in improving knowledge and reducing the number of deviations from PPE removal procedures after 3 months. We asked participants to take a test every 2 weeks in addition to a series of studies with the system to consolidate their knowledge. We believe that the participants’ regular access to the system and repetitive learning helped them consolidate and reinforce their knowledge, which improved their test scores. We also believe that they recalled their knowledge while removing PPE, leading to fewer deviations from the procedure.

We did not detect significant difference in the PPE removal duration between the intervention and control groups, although the time for the intervention group increased and it decreased in the control group. This result was also similar to those of previous studies. ¹³ Casalino et al ¹³ reported a significant decrease in the number of procedure deviations and an increase in removal duration after 3 sets of active training in PPE application and removal. In this study, repetition of knowledge was not effective in speeding up the movements, but it was effective in maintaining and ensuring the time required for removal. Participants appeared to recall the movements and potential contamination risks to consider. We suggest that the number of deviations from the procedure was also reduced by the participants’ recall of the movements and points to consider.

We detected a difference between the intervention group and the control group in the number of contaminated sites, but it was not significant. In this study, >60% of the contamination occurred in situations that were not included in the face-to-face PPE application and removal training and were not indicated in the CDC literature. A human factors risk analysis of self-contamination in PPE removal suggests that opportunities for incidental contact during the process should be minimized because inappropriate use of PPE can lead to contamination. ²² In the future, it will be necessary to verify whether additional points should be considered when removing PPE, based on the actions that caused contamination in this study. At this point, it may be necessary to revise the training materials to reflect these points. The number of contaminated participants in the intervention group was also lower than in the control group, but the difference was not significant, which may be due to insufficient statistical power. Future studies should expand the target population to further verify these findings.

In this study, more than half of the participants in the intervention group were also contaminated after the intervention. This finding indicates the complexity of PPE removal and agrees with those of previous studies. ^23,24 Our results suggest that a Web-based learning system may be helpful in acquisition and retention of knowledge and in reducing deviations from the procedures. The average SUS score for this system was higher than the average SUS score of 68.1 (SD, 21.6) for Internet-based Web pages and applications, and the score was 75.0 (SD, 13.0) for the survey results for learning management systems using Moodle. ^25,26 Thus, this system is a teaching tool with usability that is at least as good as the systems used in previous studies.

This study had several limitations. First, it was conducted in a single hospital and the sample size was relatively small, so a larger study is needed for generalization. Second, we do not know the effect of the Web-based learning system beyond 3 months. This intervention needs to be evaluated over a longer period. Third, this study used fluorescent marker instead of actual pathogens to simulate a situation where hands were contaminated. This may have overestimated contamination, and contamination when hand disinfection was properly performed is unknown.