Recall errors in a weekly survey of diarrhoea in Guatemala: determining the optimal length of recall

Summary

We measured the recall error, optimal recall length and factors associated with diarrhoea in a weekly survey. Data was taken from a year-long randomized controlled trial in which characteristics of diarrhoeal episodes were recorded weekly. We labelled the recall period as days 1–6; day 1 being the day before the visit. Recall error was the percentage difference between the number of episodes reported to begin on a particular day and the mean for days 1 and 2. Generalized estimating equations were used to determine associations. Recall error was 37% on day 3 and 51% on day 5. The error was less in younger children (by 10%), severe episodes (by 29%) and when blood was present in the stool (by 18%). Diarrhoea was underreported when the recall period extended beyond 2 days. Surveys that use longer recall periods risk underestimating diarrhoea incidence and selectively capturing more severe episodes.

INTRODUCTION

Diarrhoea is a leading cause of death in children in underdeveloped nations. It is estimated that diarrhoea causes 21% of all deaths in children aged <5 years [1], killing 1·87 million children annually [2]. Even though the mortality from this illness has progressively decreased over time, the morbidity has remained high [1]. The morbidity of diarrhoea is commonly measured using longitudinal surveys. Previous workers have noted methodological limitations to measuring diarrhoea using longitudinal surveys including recall errors [3–5].

When data from weekly diarrhoeal surveys collected during intervention studies were analysed later to determine the proportion of cases that were underreported, an error of up to 44% was found in Bangladesh [6], 45% in India [7], and 50% in The Gambia [8]. In each study the reporting error was directly related to length of the recall. The prevalence of daily diarrhoeal episodes decreased as the length of the recall period increased in Bolivia when Boerma et al. assessed the national demographic health survey [9]. Melo et al. found a recall accuracy of only 30% when an end-of-the-month survey was compared with information gathered by visits every other day in Brazil [10]. In rural Northeast Brazil, McAuliffe et al. found a 28% deficit in the diarrhoeal incidence when weekly surveillance was compared with daily records in the same population during the same time period [11].

In countries where limited resources need to be prioritized, the accuracy of data on disease burden is important. Longitudinal household surveys are used by epidemiologists, health programme managers and policy makers. This information cannot be imported from other countries. Reporting errors vary across countries [9]; cultural and other local factors play a role [4]. The error due to recall, thus needs to be explored in different settings before national policies, comparisons and global estimates are made. The optimal recall period also needs to be determined in order to minimize expenditure while maximizing accuracy.

The objectives of our study were to determine the accuracy and optimal recall period for mothers reporting diarrhoeal episodes for their children aged <5 years in a year-long weekly survey conducted in the Guatemalan highlands. We also aimed to identify the factors that influenced this recall.

METHODS

We extracted data from a previously conducted randomized controlled trial which studied the effect of household drinking water treatment with a flocculant-disinfectant on diarrhoeal episodes. Reller et al. [12] conducted this study in 12 villages of the state of San Juan Sacatepéquez, Guatemala from August 2001 to September 2002. Residents of these villages usually live in small huts with dirt floors. Typically ⩾5 people sleep in the same room. The study had five arms comparing different water-treatment techniques: (a), standard habits and practices; (b), flocculant-disinfectant alone (used with traditional vessels); (c), flocculant-disinfectant with special vessel; (d), bleach with traditional vessel; (e), bleach with special vessel. These groups were randomized and assigned by household and were balanced within each village. A total of 492 households agreed to participate in this trial. After a baseline survey, field-workers conducted weekly visits for about a year to record occurrence of diarrhoea in household members of all ages.

In the weekly visits the trained field-workers used standardized questionnaires to record the presence of diarrhoea in any member of the household during the last 7 days. Diarrhoea was defined by the respondent who was usually the mother. Additional information recorded from the weekly visits was the date of onset and termination of the diarrhoeal episode, presence of blood in stool and the maximum number of stools in a 24-h period. When the field team encountered someone with diarrhoea they supplied packets of oral rehydration salts and encouraged them to seek care at a community health post. For more severe cases, when urgent medical attention was required, the field team arranged for an immediate visit by a physician or transport to the local hospital.

Since the focus of the current analysis was to assess the recall error made by mothers when reporting diarrhoeal episodes in their young children, we combined the data from all five arms of the trial and limited it to include only the person-weeks in which the age of the person was <60 months (5 years) at the time of the interview. Children attaining age >5 years and new births were accordingly excluded or included, respectively. Of the 492 households, 17 did not have a child aged <5 years during the entire study period. The remaining 475 households were included in the analysis. We used the information gathered during the weekly visits and the baseline survey to identify reporting errors and assess the factors responsible for these. We considered the possibility of variable accuracy in recall between the different arms of this unblinded study and tested this during our analysis.

We made the same assumptions as Alam et al. that an episode of diarrhoea is equally likely to begin on any day of the week and therefore the number of episodes on each day should be similar. We used the same method to calculate recall error [6]. The recall period was labelled as recall day 0, 1, 2, 3, 4, 5 and 6, where 0 was the day of the interview. Since the interviews were usually conducted during the morning, the number of episodes reported on day 0 was not considered complete and so was not included for further analysis. We defined severe diarrhoea as a diarrhoea episode with >5 stools in a 24-h period.

We calculated the mean of the number of times in which diarrhoeal episodes were reported to begin on recall days 1 and 2, the first 2 days preceding the day of the interview, and used this as the reference value. Previous studies have considered the first 48 h of recall to be the most accurate [13]. Our initial analysis identified a sudden fall in reporting after these two days. This pattern was similar regardless of the day of the week in which the interview was taken. Since recall of recent events is better than recall of distant events [14] we assumed that the shorter recall period was accurate and the longer recall period less accurate, and thus the difference between the two to be an erroneous underreport. We labelled the difference between the reference value and the number of episodes of diarrhoea reported on the previous days as the reporting error and calculated it as a percent value using the following formula:

where ND is the number of times in which diarrhoeal episodes were reported to have begun for a given recall day beyond 48 h. The overall 6-day reporting error was calculated by the following formula:

where ND6 is the summation of the number of days in which diarrhoeal episodes were reported on recall days 1, 2, 3, 4, 5 and 6.

We tested for factors that influence the recall error by determining which factors made the diarrhoeal event more memorable and thus more likely to be reported on days 3–6 than only on days 1 and 2. The comparison groups were thus: (i) those diarrhoeal episodes that were reported on recall days 1 or 2 and (ii) the diarrhoeal episodes that were reported on days 3–6. We used generalized estimating equations (GEE) to calculate bivariate odds ratios and 95% confidence intervals of characteristics associated with recall of >2 days. This approach accounted for the repeated measurements within households. We also used GEE to construct a multivariate model to control for confounding [15]. Variables with a P value of <0·1 on bivariate analysis were included in the multivariate GEE model. We encountered only four such variables and none were eliminated from the multivariate model. We used an exchangeable correlation matrix to adjust for repeated measures and household clustering.

For a better depiction of the difference in magnitude, we calculated the overall recall errors for the significant variables and presented them as percent values. We achieved this by first selecting the variables with significant associations upon multivariate analysis (P<0·05). We then applied the same formula mentioned above for ‘overall 6-day reporting error’ to each of these variables and their categories. For example, for severe diarrhoea the reference value was the mean of the number of days in which severe diarrhoea was reported on days 1 and 2, multiplied by 6. The ND6 was the total number of days in which a severe diarrhoeal episode was reported on recall days 1, 2, 3, 4, 5 and 6. Similarly for mild (or ‘not severe’) diarrhoea, the reference value and the ND6 were with regard to the number of days on which mild diarrhoea was reported.

The original study was reviewed and approved by an institutional review board at CDC and the Ethics Committee Review Board at the Universidad del Balle de Guatemala.

RESULTS

The demographic details of the study population have been presented elsewhere [12]. Our analysis covered 475 households and included 1033 children. We analysed 43 992 person-weeks of observation, 22 994 of those were from male and 20 998 were from female children. The mean age during the study was 27 months (s.d.=16). The study population experienced 3972 person-days of diarrhoea, an average of 3·8 days of diarrhoea per child, over an average of 298 days of observation per child.

There were 824 (22%) episodes of diarrhoea that were reported to begin 1 day before the interview and 822 (22%) reported to begin 2 days before. After this we observed a sudden drop in reporting and only 516 (14%) episodes were reported to begin 3 days before the visit. This figure fell further as the recall period increased (Fig. 1). The recall error increased from 37% on day 3 to 49% on day 4, 51% on day 5 and 48% on day 6. The overall 6-day recall error was 31%.

Fig. 1. Reporting of diarrhoea episodes by length of recall.

We compared episodes that were recalled more than 2 days before the interview with those that were recalled in the first 2 days. Episodes associated with short recall periods were in older children, those without blood in the stool and those that were not severe (Table 1). In the multivariate analyses compared to diarrhoeal episodes recalled for longer periods, diarrhoeal episodes that were recalled within 48 h were 1·2 times more likely to be of children aged >2 years, 1·5 times more likely to not have blood in the stool and two times more likely to be non-severe (Table 2). The 6-day recall error for children aged <2 years was 28% compared to 38% for older children (Table 2). The underreporting for severe diarrhoea was only 11%, compared to 40% for non-severe diarrhoea. When blood was present in the stool the recall error was 13% compared to the 31% error when no blood was reported to be present.

Table 1. Factors associated with a short recall period of diarrhoea in bivariate analysis* of a weekly survey in Guatemala

OR, Odds ratio; CI, confidence interval.

* Analyses were performed using generalized estimating equations to account for clustering.

† Diarrhoea reported on days 1 and 2 and not on days 3, 4, 5 and 6.

‡ Severe diarrhoea was defined as >5 stools in a 24-h period.

Table 2. Factors associated with a short recall period of diarrhoea in multivariate analysis* of a weekly survey in Guatemala

OR, Odds ratio; CI, confidence interval.

* Analyses performed using generalized estimating equations.

† Severe diarrhoea was defined as >5 stools in a 24-h period.

We did not find a difference in recall errors between the intervention and control groups or by maternal education, gender of the child or the presence of >5 household members.

DISCUSSION

The similar number of reports for recall day 1 (824) and day 2 (823) followed by the sudden fall for day 3 and subsequent decrease in the previous days strongly suggest that diarrhoea was underreported beyond a 2-day recall period. While it is possible that the difference may be due to overreporting of diarrhoea on days 1 and 2 there was no reason for the mothers to do so. Medical benefits provided were minimal and if this were the case then reporting on the day of the visit, day 0, should have been much higher than what we found. Previous studies conducted in Bangladesh and India have also observed a similar rapid fall in the number of reported episodes after 2 days of recall [6, 7].

Alam et al. studied data from an interventional study in Teknaf, Bangladesh which used weekly diarrhoea surveillance [6]. They reported a sudden 26% fall in the number of reported diarrhoeal events after 2 days of recall. Similarly Ramakrishnan et al. using data from a community trial studied the accuracy of a weekly diarrhoea survey in South India [7]. They demonstrated a sudden fall of 15% in the reporting of diarrhoeal events on the third day of recall. Both studies found a progressive decrease in the numbers of reported diarrhoeal events as the length of recall increased. Boerma et al. in their comparison of 19 demographic health surveys highlighted a fall in the daily prevalence of reported diarrhoeal episodes in Bolivia [9]. Here the prevalence fell from about 150/1000 children on recall days 1 and 2 to around 110/1000 on day 3 to about 80/1000 on day 4. Two-week recall periods have previously been considered to be the best in terms of balancing the recall error and the cost of the study [3] and are still being used in many health surveys. The inevitable recall errors arising from such surveys render the data inaccurate and their use questionable [16].

The overall 6-day recall error for children aged <2 years was 28% as opposed to 38% for older children. One reason for this may be, as Melo et al. also observe, that in rural areas and in large family settings it is usually the eldest child that takes care of the toddlers while the mother attends to the infants and younger children [10]. The mother, whom we rely on for information, is thus more likely to give more accurate accounts in the cases of younger children. Another reason could be that mothers pay more attention to diarrhoeal diseases when the child is young and are less worried about it as the child grows up.

Diarrhoea is such a common event in such impoverished communities that it makes sense for the mother or guardian to overlook it, especially milder episodes. In our study the 6-day recall error associated with severe diarrhoea was only 11% while that for mild diarrhoea was 40%. Alam et al. also demonstrated an inverse relationship between the severity of diarrhoea and recall error [6] in which ⩾7 loose movements in a 24-h period had a 22% reporting error, 5–6 loose movements had a 32% error while 3–4 in a day had a 44% recall error.

A criticism of point-of-use water-treatment and other household interventions is that the subjective reporting of diarrhoea may bias unblinded studies, because responders belonging to the intervention groups may report fewer diarrhoeal episodes to please the interviewers [17]. A recent meta-analysis suggests that when subjective outcome measures (such as self-reported gastrointestinal symptoms) are used in randomized controlled trials the results typically overestimate effects [18]. In this analysis we did not find any difference in recall errors between the intervention and control groups. This suggests similar patterns of recall in the unblinded point-of-use water-treatment intervention and control groups.

Our study presents data from one area of Guatemala, and different recall patterns are possible in other places. However, the findings are consistent with results from other low-income settings. Another limitation is that we could only gather information on the episodes of diarrhoea that were reported. Therefore, we could not directly determine the factors that made episodes forgettable. We instead determined the factors that made the diarrhoeal events more memorable. It is possible that we were unable to identify or assess other influencing factors. Moreover, we could not determine the effect of ‘duration of illness’ on recall ability (whether shorter or longer episodes were more likely to be recalled) as the interviews were conducted every week. A longer period of time between the interviews would be needed to test for this; however, we did observe that the recall of shorter diarrhoeal episodes (⩽2 days' duration), as expected, showed the same pattern and rate of reporting error as that of all episodes (data not shown).

Furthermore, the study did not define diarrhoea and used a maternal definition for it instead. Our results therefore may not be strictly comparable to the ‘⩾3 stools’ definition that is used commonly in other studies. However, many studies use alternate definitions, a number of which have used maternal definitions [1]. The analysis of our study is based on the assumption that diarrhoea is equally likely to begin on any day of the week. This assumption is reasonable for the rural settings of this study, but may not hold for settings in which eating habits change during specific days of each week. Our analysis showed a similar pattern of reporting on the Monday–Friday periods that the data were collected.

We have highlighted the inaccuracy due to recall errors in longitudinal diarrhoea surveys. Diarrhoea was underreported in this population if the recall period extended >2 days. Milder episodes and those of older children are less likely to be remembered for longer periods by mothers and thus may remain undetected by researchers. Even when the goal is longitudinal surveillance less frequent visits by field-workers that measure diarrhoea prevalence during a more accurate shorter recall period is an efficient means of collecting valid assessments [19]. Surveys that use a recall period of >48 h risk substantially underestimating diarrhoea incidence.