A Pilot Study in the Use of the Delphi Method to Document Conference Proceedings: Comparison of the Rate of Consensus Among Attending and Nonattending Participants

Abstract

Objective:

While many medical practitioners value the interactive nature of in-person conferences, results of these interactions are often poorly documented. The objective of this study was to pilot the Delphi method for developing consensus following a national conference and to compare the results between experts who did and did not attend.

Methods:

A 3-round Delphi included experts attending the 2023 Society of Disaster Medicine and Health Preparedness Annual Meeting and experts who were members of the society but did not attend. Conference speakers provided statements related to their presentations. Experts rated the statements on a 1–7 scale for agreement using STAT59 software (STAT59 Services Ltd, Edmonton, Alberta, Canada). Consensus was defined as a standard deviation of ≤ 1.0.

Results:

Seventy-five statements were rated by 27 experts who attended and 10 who did not: 2634 ratings in total. There was no difference in the number of statements reaching consensus in the attending group (26/75) versus that of the nonattending group (27/75) (P = 0.89). However, which statements reached consensus differed between the groups.

Conclusion:

The Delphi method is a viable method to document consensus from a conference. Advantages include the ability to involve large groups of experts, statistical measurement of the degree of consensus, and prioritization of the results.

Following the coronavirus disease (COVID-19) pandemic, in-person attendance by health care professionals has risen for many conferences. Meeting and event numbers reached nearly 90% of pre-pandemic levels at the end of 2022. ¹ According to a 2023 survey of 275 health care providers, the most valuable outcome of attending medical congresses is the opportunity to learn and keep up to date with new developments and the current thinking. ²

However, knowledge gained or presented during conferences is often poorly documented. Conference proceedings, even if published, typically include only the content of the papers and abstracts. Important conversations, conclusions, and issues that are discussed are usually not documented. For instance, groups may discuss and even gain consensus on issues without recording this consensus. Even conferences that include consensus generating techniques—such as focus groups—frequently limit these activities to key conference speakers and organizers without involving conference attendees, in general.

The goal of this study was to pilot the use of the Delphi methodology as a way to document issues discussed during a national conference and translate these discussions into actionable information. The Delphi method is a common and well documented method for achieving consensus when consensus is difficult to obtain. ³ The method is an iterative survey technique, where a group of participants (commonly called experts) is asked to rate statements on a specific scale. Statements are deemed to have reached consensus when experts’ ratings converge toward a common value.

The specific objective of this study was to assess the difference in rate of statement consensus in a group of professionals attending an in-person conference in comparison to the rate of statement consensus among professionals with similar backgrounds who did not attend the conference.

The null hypothesis of no difference between the proportion of statements reaching consensus was tested against the alternative hypothesis of a significant difference.

Methods

Study Design

The overall study design followed the Delphi methodology. Invited speakers from the conference were asked to provide 3 to 5 statements to be used in the first round of the Delphi study. Speakers’ statements were to include content specifically addressed in their own educational sessions. In total, 28 statements were obtained. Examples of statements included:

Patient monitoring devices allowing continuous monitoring of vital signs during mass casualty incidence responses can enhance lifesaving interventions.

Potential biases in data sets used to train artificial intelligence algorithms generate serious ethical concerns.

Prehospital mass casualty incident response training should be delivered to all first responders.

Statements that did not reach consensus were advanced to the next round.

The results of the Delphi ratings were compared between experts who attended a national conference and those who did not.

Study Setting

The in-person conference was the Society of Disaster Medicine and Public Health Preparedness Ten Year Anniversary Conference and Annual Meeting, which took place in Washington DC on December 4–6, 2023. The conference focus was the effects of disasters on vulnerable populations, including climate change and disaster response; conflict medicine and population health; artificial intelligence; decision aids and equipment, and their effect on mass casualty incident response; disaster medicine education, training, and competencies that are culturally and geographically relevant; and mental health. The conference had 15 speakers and 129 attendees.

Participants

All participants of the in-person conference were invited to partake in the Delphi study as experts. These participants formed the “attending” group. All members of the Society of Disaster Medicine and Health Preparedness who did not attend the conference were invited to join the “nonattending” expert group. Participants were contacted through email and enrolled into the study prior to the in-person conference. All participants completed an online informed consent process.

Outcome Measures

The primary outcome measure was the number of statements reaching consensus in the Delphi study. Consensus was defined as a standard deviation of less than or equal to 1.0 on a 7-point rating scale. ⁴

Data Collection

Statements collected from the invited speakers were entered into the online Delphi method software STAT59 (STAT59 Services Ltd, Edmonton, Alberta, Canada). The experts were contacted by email to enroll in the STAT59 platform to answer the survey rounds. In the first round, experts were asked several demographic questions, including current profession, location of work, primary employment, and years of expertise in disaster medicine. Experts were then asked to rate each of the 75 statements on a 7-point linear scale for importance, where 1 represented “Strongly disagree” and 7 represented “Strongly agree.”

Data Analysis

Data analysis was performed using STAT59 and Python, Version 3.6.9 (Python Software Foundation, Wilmington, Delaware, USA). ⁵

Sample Size

No formal sample size calculation was performed. All participants attending the conference and all members of the national society were invited.

Results

The final study group included 27 experts who attended the conference and 10 who did not. Most participants (25/37) were from North America, whereas others were from Europe and Central Asia, East Asia and Pacific, Middle East and North Africa, and sub-Saharan Africa. The most indicated professions included “education and training” (71%), “physician” (43%), “research” (40%), “response/field operation” (20%), and “administration and support” (11%). Employment included “university or research center” (49%), “governmental organization” (23%), “private sector” (20%), and “non-governmental organization” (9%). Among the participants, 40% had greater than 15 years of experience in disaster medicine, 31% had 11 to 15 years of experience, 17% had 6 to 10 years, and 11% had 5 years or less.

The first round of the Delphi study contained 6 demographic statements and 28 rating statements. The second round contained 26 statements. The third round contained 11 statements. In total, each expert rated 75 statements.

In total, there were 2634 ratings on the 7-point linear scale. The overall mean rating on the 7-point scale was 5.4, with a standard deviation of 1.4.

Among the participants who attended the conference, the mean overall rating of the 75 statements on the 7-point scale was 5.4 with a standard deviation of 1.13 (Figure 1). Among the attending participants, 26 statements reached consensus. Among the participants who did not attend the conference, the mean overall rating of the 75 statements was 5.3, with a standard deviation of 1.17 (Figure 2). In this group, 27 statements reached consensus.

Figure 1. Attending experts standard deviation from 75 statements.

Figure 2. Nonattending experts standard deviation from 75 statements.

Using the proportion test, no significant difference was found between the proportion of statements reaching consensus among the attending participants (26/75) when compared to the proportion of 27/75 among the nonattending participants (Z-stat = −0.14; P = 0.89).

Which statements reached consensus differed between the attending and nonattending participants. Eleven statements reached consensus in the attending group only. Ten statements reached consensus in the nonattending group only. Sixteen statements reached consensus in both groups.

At the completion of the study and after analyzing the results in aggregate by combining the attending and nonattending groups and calculating the standard deviation for each of the statements, 35 statements reached consensus: 12 in round one, 15 in round two, and 8 in round three. The final list of statements reaching and not reaching consensus will be published separately.

Discussion

This study demonstrated no significant difference in the number of statements reaching consensus in the Delphi study when comparing experts who had attended the national conference versus those who did not. While the difference in standard deviation between the attending and nonattending group was small, Figure 1 and Figure 2 suggest that there was less dispersion in the SD among the attending group. Nonetheless, which statements reached consensus differed considerably between the 2 groups.

At present, there is no published universal best practice for achieving consensus. Waggoner et al. reviewed common methods of consensus gathering in 2016 and included nominal group process, consensus development panels, and the Delphi method. ⁶ Although they failed to find documented best practices, the manuscript ended with the following statement: “We hope that future studies will be conducted using more stringent standards, especially as it pertains to reporting of design, methods, and results.” ⁶

This pilot demonstrates that the Delphi method can be used as a more rigorous method to document conclusions from an in-person conference. The Delphi method offers several advantages over more informal methods of conference proceedings. First, the Delphi method can be used with any size of expert group, whereas focus groups and expert panels often become difficult to manage when more than 10–15 participants are involved. Second, the Delphi method can be used to combine the opinions of those present at the conference with those who could not attend, leading to a more inclusive and diverse expert panel. Third, while both focus groups and Delphi studies aim to provide a set of actionable statements, only the Delphi method provides an objective measure of the degree of consensus and also prioritizes these statements. Finally, the Delphi method is also an excellent way to document statements that are considered but do not reach consensus; these statements can be invaluable for anticipating future trends and highlighting areas for further investigation.

While this pilot study demonstrates that the Delphi method can be implemented as a robust approach to documenting conference proceedings and indicates that consensus is similar in both attendees and nonattendees, further rigorous studies of the Delphi method in conference proceedings are indicated.

Strength and Limitations

To our knowledge, this is the first published study to pilot the use of the Delphi method to document conference proceedings and compare attending versus nonattending participants.

Limitations of this study include the small sample size of only 35 experts and application of the method to only a single conference. However, the size of the expert group was large by the standard for consensus which typically suggests panels of 6 to 11 members. ⁶

Furthermore, while the Delphi method is commonly employed to gain consensus, the statistical analysis of these studies is controversial. In this study, we chose to use the 7-point linear rating scale for agreement, and to define consensus as a standard deviation of less than or equal to 1 as this has been previously shown to be a robust and reliable method to analyze Delphi survey data. ⁴ A different rating scale or different definition of consensus may have led to different results.

Conclusion

The Delphi method represents a viable and implementable technique to prioritize and document consensus for conference proceedings. The technique allows conference attendees and nonattendees to assess statements developed at the conference. The proportion of statements obtaining consensus after the first round appears similar between attendees and nonattendees. However, which statements reached consensus were markedly different between the 2 groups. Advantages over traditional conference proceedings include the ability to involve larger groups of experts, statistical measurement of the degree of consensus, and prioritization of the results.