Inviting a statistician to join an evidence-based journal club

Abstract

AIMS AND METHOD

In response to frequent cancellations, we recently changed the format of our journal club. We invited a statistician to attend and each meeting focused specifically on critical appraisal and statistical analysis. Active learning principles were used where every participant critically appraised an aspect of the paper presented at each meeting. The aim of this paper was to evaluate the impact of the new format on attendance and satisfaction levels. Attendance and satisfaction levels were measured in the 3 months before and the 3 months after the new format was introduced.

RESULTS

Average attendance increased from 10 to 15, with two psychologists and two trainees who did not work in the hospital starting to attend regularly. Satisfaction levels also increased significantly with respect to academic interest, enjoyment, usefulness and overall satisfaction.

CONCLUSION

Combining statistical expertise with principles of active learning has led to increased satisfaction levels among participants in our journal club.

We were concerned about our journal club. Traditionally, based as it was in the department of psychiatry in a large teaching hospital, the weekly meetings had been successful in terms of longevity and attendance. In contrast, over the previous 6 months we had experienced last minute cancellations, presentations that were poorly prepared and low levels of attendance. All this despite the fact that we fulfilled many of the criteria recommended by Sidorov (1995): a small training programme, available food, mandatory attendance and an active research programme. At a focus group held to identify the causes of the recent decline in enthusiasm, trainees attributed the problem to the introduction of the new clinical appraisal paper in the Membership exam. The traditional journal club format, where an article was presented and then discussed, was seen as no longer relevant to trainees' needs. When an attempt was made to introduce a more evidence-based format as described by Gilbody (1996) pre-Membership trainees did not feel confident in presenting. Moreover, more senior trainees who had taken the Membership exam before the introduction of the new paper also lacked confidence in chairing the meetings. Taking into account these concerns about the journal club, the focus group generated several suggestions to develop the journal club. We have now implemented these suggestions and the aim of this paper is to evaluate the impact of the new format on attendance and satisfaction levels.

Method

New format

Trainees expressed a wish for the journal club to be made more relevant to their needs, particularly with regard to statistics and the critical appraisal paper of the Membership examination, and the following changes were therefore made:

1. (a) A statistician who has a background in psychology was asked to attend the meetings on a weekly basis.

2. (b) A list of papers that illustrates all the statistical terms and methods covered in the Membership curriculum was identified.

3. (c) The nominated presenter reads the paper in advance and decides on a list of questions similar to those asked in the critical appraisal paper.

4. (d) At the meeting, those attending are divided into small groups, given the paper to read and allocated several questions. Each group has 20 minutes to decide on answers to the questions, and they then present their conclusions to the large group.

5. (e) The presenter chairs the discussion that ensues and the statistician is used as a resource to help with areas of difficulty.

6. (f) At the end of the meeting, the presenter gives a brief résumé of the statistical questions that were mentioned in the paper and distributes a short (less than a page) definition of any terms considered relevant.

Assessment of impact of new format

We studied attendance and satisfaction in the last 3 months of the traditional journal club and the first 3 months of the new format using the following measures:

1. (a) the number of meetings cancelled

2. (b) the average number of people attending the meeting

3. (c) a 4-point Likert scale (1=no interest/satisfaction to 4=highly interesting/very satisfied) to examine clinical interest, academic interest, enjoyment and overall satisfaction.

T-tests were used to look for significant differences between the mean scores in each of the categories before and after the new format was introduced. Finally, participants were asked for any additional comments.

Results

Prior to the intervention, the core group attending the journal club consisted of 10 doctors: three consultants, three post-Membership trainees involved in teaching and research and four pre-Membership trainees. Two members of the psychology department had been invited to the old format journal club but after one meeting had decided that it was not relevant to their work. In a 3-month period, only four of 14 meetings were actually held, the rest being cancelled owing to trainees not being prepared or failing to attend. Table 1 shows the mean satisfaction levels.

Table 1. Mean satisfaction levels before and after introduction of a new format for journal clubs

					95% Cl
	Before or after intervention	Mean	d.f.	Significance (2 tailed)	Lower	Upper	n
Clinical interest	Before	2.6	21	0.142	-1.22	-0.19	9
	After	3.1	15				14
Academic interest	Before	2.4	22	0.011	-1.9	-0.3	10
	After	3.5	12				14
Enjoyable	Before	2.4	22	0.037	-1.6	-6.11	10
	After	3.3	11				14
Usefulness	Before	2.4	22	0.005	-2.2	-0.5	10
	After	3.8	11				14
Overall satisfaction	Before	2.2	22	0.008	-2.7	-0.4	10
	After	3.5	12				14

Three months after the intervention, the number of people attending the meetings had increased to 15. One new lecturer started working in the department. The two psychologists now attend on a regular basis. Two pre-Membership trainees working in another hospital heard about our new format and felt that the additional statistical input and training in critical appraisal techniques was sufficiently helpful for their exam preparation to travel to the department. All 14 of the scheduled meetings were held. As can be seen in Table 1, satisfaction levels improved in all areas, with statistically significant increases in academic interest, enjoyment, usefulness and overall satisfaction. Comments offered by participants included acknowledgement of a greater sense of ownership of the journal club by trainees. Several people noted that where the meeting would have been cancelled in the past if no consultants had been able to attend, now there was enough confidence to carry on regardless. Trainees also said that they found it easier to ask questions of someone who was outside the medical hierarchy, particularly when these questions might reveal their lack of knowledge in statistics. Finally, there were several comments about the benefits of regular scheduled contact with a statistician, when informal advice could be sought regarding on-going research projects.

Discussion

This study examines the impact on our journal club of a new format that placed increased emphasis on critical appraisal and used adult learning principles. We found a significant increase in satisfaction levels and improved attendance rates.

Some limitations to this study are obvious. The length of follow-up is short, merely 3 months. It is likely that any change that is perceived as emanating from the group will be associated with a sense of dynamism and enthusiasm, which is then reflected in higher morale. It is interesting that in the literature to date, although many different formats for journal clubs have been studied, every intervention has been reported as having positive effects on attendance and satisfaction.

A second limitation is that we have no objective measurements of improvements in statistical knowledge or ability to critically appraise a piece of research. Previous studies in this field have found inconsistent evidence of objective improvement in critical appraisal skills even where trainees have reported satisfaction (Seelig et al, 1991; Sandifer et al, 1996; Norman & Shannon, 1998; Fu et al, 1999). Most of these studies were based in the US or Canada where, as far as we are aware, trainees are not formally assessed in critical appraisal as part of their qualification as a specialist. It is possible that trainees in the UK and Ireland may be more realistic in their evaluation of their own abilities given that they will be examined in critical appraisal skills. Therefore, their expression of satisfaction, and particularly the attendance at the journal club of trainees who are not obliged to attend, may be given more weight. The advantages of our new format are several. First, identifying specific learning objectives (namely statistical knowledge and critical appraisal ability) that are seen as relevant to participants' needs has created an improved sense of ownership. Both Sidorov (1995) and Linzer et al (1987) suggested that a balance needed to be struck between allowing trainees to feel a sense of self-determination and involving faculty in teaching. In a similar vein, Rowlands and Geddes (1993) pointed out the importance of active involvement in a journal club to the morale of junior doctors. We feel our format encourages trainees to take responsibility for the journal club while ensuring that expertise is readily available.

Second, dividing participants into small groups and requiring them to devise answers and to debate these answers with other participants leads to active learner involvement. Like Elnicki et al (1999), we were concerned about the risk of embarrassing individuals who might feel self-conscious about their lack of knowledge in an area. In general we find that this is not an issue once there are at least three participants in each group. Moreover, the involvement of all those present reduces the workload for the presenter: in general, participants have found that the preparation time is less than for the traditional format.

Finally, the presence of statisticians at each meeting has created an environment where consultants, psychologists and trainees feel freer to question each other in a scientific manner. Knowing that a statistician is there to answer and elaborate on difficult points encourages all those present to focus on the statistical strengths and weaknesses of a study.

We are not aware of other journal clubs that have sought regular statistical support. Other groups seeking outside expertise have mainly looked to librarians for assistance in conducting adequate literature searches (Scherrer, 1999). Clearly the determining factor here is the learning objectives set for the meetings, but we feel that other groups might, like ourselves, identify statistical analysis as a particularly challenging area.

Conclusion

Reviews of journal clubs (Jouriles et al, 1996; Alguire, 1998) suggest that there is no one ideal format: rather, different groups need to tailor a journal club according to their needs. In our department, trainees and researchers expressed an interest in meetings that focused primarily on critical appraisal. Combining expert statistical advice with active learning has produced a format associated with improved participant satisfaction. To maintain enthusiasm we hope to develop this format further, allowing participants to move from pure critical appraisal to a broader evidence-based medicine approach, formulating a clinical question and searching for the most relevant literature.