Nutritional studies often rely on self-reported dietary intakes because of the feasibility of this approach especially in large-scale national surveys( Reference Nielsen and Adair 1 ). Inevitably, self-reported dietary intakes involve misreporting (i.e. under- and over-reporting) and implausible intakes( Reference Black, Goldberg and Jebb 2 ). The prevalence of under-reporting varies between 10 and 50 % in different studies depending on the cut-off point used for identifying misreporters( Reference Schoeller 3 – Reference Schoeller, Bandini and Dietz 5 ). Misreporting of energy and nutrients can be both variable and substantial; hence, it poses a challenge for epidemiologists trying to find a clear relationship between dietary intakes and health outcomes( Reference Freedman, Schatzkin and Midthune 6 ). Some studies have confirmed that there is a tendency towards omission of food items that are socially undesirable (i.e. high in fat, added sugars and alcohol), also referred to as selective misreporting( Reference Freedman, Schatzkin and Midthune 6 – Reference Huang, Roberts and Howarth 8 ). In addition, energy intake (EI) reporting is influenced by subjects’ characteristics( Reference Nielsen and Adair 1 , Reference Lioret, Touvier and Balin 7 , Reference Murakami, Miyake and Sasaki 9 , Reference Black and Cole 10 ), for example, the magnitude of under-reporting increases with increasing BMI, which may falsely lead into the conclusion that overweight and obese individuals consume less energy compared with their normal-weight counterparts (differential misreporting)( Reference Schoeller 3 , Reference Lafay, Basdevant and Charles 11 ). As a result, misreporting is a particular problem for studies investigating the association of diet with obesity as it may render the relationship ambiguous or attenuated, diminishing the usefulness of nutrition data in informing public health policy (bias towards the null)( Reference Mendez, Popkin and Buckland 12 ).
The misreporting phenomenon is still largely overlooked in obesity research. Several procedures have been proposed for identifying implausible dietary recalls( Reference Goldberg, Black and Jebb 13 , Reference McCrory, McCrory and Hajduk 14 ); even though methods of handling physiologically implausible recalls are less well-studied. Thus far, only few studies (none on adolescents) have investigated methods of handling implausible recalls( Reference Nielsen and Adair 1 , Reference Huang, Roberts and Howarth 8 , Reference Mendez, Popkin and Buckland 12 , Reference Bornhorst, Huybrechts and Hebestreit 15 ), and none have compared all available methods among different age groups, especially in a large-scale national survey. The purpose of this study is therefore to systematically compare the utility of seven different statistical approaches for handling inaccurate reports of dietary intakes among a nationally representative sample of Canadian adolescents (12–17 years) and adults (≥18 years), when examining the relationship between dietary intakes and obesity. The following methods were used for correcting the misreporting: (1) adjusting for variables related to misreporting, (2) excluding misreported recalls, (3) adjusting for the reporting groups (under-reporter, plausible reporter and over-reporter), (4) adjusting for the propensity score( Reference Bornhorst, Huybrechts and Hebestreit 15 , Reference Rosenbaum and Rubin 16 ), (5) adjusting for both reporting groups and propensity score, (6) stratifying the analyses by reporting groups and (7) stratifying the analyses by reporting groups and adjusting for the propensity score( Reference Bornhorst, Huybrechts and Hebestreit 15 , Reference Rosenbaum and Rubin 16 ). Propensity score is a statistical technique aimed at reducing bias by equating groups based on variables associated with misreporting( Reference Bornhorst, Huybrechts and Hebestreit 15 , Reference Rosenbaum and Rubin 16 ). Adjustment for propensity score in nutrition surveys has only been used among children (proxy reports) and was found to be a useful tool for counteracting attenuation of risk estimates caused by misreporting( Reference Bornhorst, Huybrechts and Hebestreit 15 ).
Additionally, we examined the validity of participants’ self-estimated intake amount (less than, the same or more than the usual amount) collected as part of the 24-h recall procedure, through comparison with the calculated cut-off points for identifying misreporters. Recommendations on how best to counteract attenuation of risk estimates caused by misreporting bias in obesity research are also given.
Methods
Study population
Data for this study were collected under the authority of the Statistics Act of Canada, and all data analyses were conducted at the Statistics Canada’s Research Data Center. The Canadian Community Health Survey (CCHS) cycle 2.2 (2004–2005) provides the most complete nutritional data on Canadian dietary intakes and is the only available national nutrition data in >30 years( 17 ). CCHS 2.2 is a complex multistage nationally representative survey including cross-sectional nutritional and health data from 35 107 Canadians of all ages, representing >98 % of the Canadian population from ten provinces( Reference Béland, Dale and Dufour 18 ). Details on the CCHS 2.2 study design, sample and procedures have been published previously( 17 ). For the purpose of this study, we excluded all pregnant (n 175) and lactating (n 92) women, those under 12 years of age (n 8335) and participants with invalid self-reported dietary recalls (as defined by Statistics Canada) (n 39). Data from all respondents with complete data on physical activity and measured weight and height were included, resulting in a final sample of 16 187 subjects. Only participants ≥12 years were included in this study as only this group had self-reported dietary recalls (as opposed to proxy reports for children). In order to evaluate the association of misreporting with lifestyle and socio-economic characteristics, missing values for these variables were additionally removed, leaving a total of 15 722 individuals for regression analyses. None of the socio-demographic or lifestyle characteristics of individuals included in the final analyses was significantly different from those of participants who were excluded because of missing variables (data not shown).
Data collection
Detailed dietary intake data were collected through the 24-h recall method using a five-step modified version of the US Department of Agriculture (USDA) Automated Multiple Pass Method (AMPM)( 17 , 19 , Reference Moshfegh, Borrud and Perloff 20 ). Respondents were asked to recall all foods and beverages consumed in the previous 24 h (midnight to midnight), and energy and nutrient compositions of reported foods were derived from the Health Canada’s Canadian Nutrient File( 21 ). As part of the 24-h recall procedure, participants also responded to a question whether they ate less than the usual amount, the same as usual or more than the usual amount during the recall day( 17 ). We additionally used these data to test whether the self-reported usual intake (subjective measure of misreporting) was valid as compared with the cut-off points calculated to identify misreporters (objective measure of misreporting).
Trained interviewers measured height and weight according to standard protocols, and BMI in adults was used as a measure of body fatness using the standard cut-off points for overweight (≥25–29·99 kg/m2) and obesity (≥30 kg/m2). For adolescents, categories of Cole et al.’s( Reference Cole, Bellizzi and Flegal 22 ) were used. Data on socio-demographic characteristics, lifestyle behaviours and health history were collected using interviewer-administered questionnaires( 17 ). Anthropometric measurements and data collection interviews were carried out in person and at participants’ homes( 17 ).
EI (kJ/d (kcal/d)) (in 100s of kcal), fibre density (g/4184 kJ (g/1000 kcal)), percentage of EI from solid fats and added sugars (SoFAS), percentage of EI from fruits and vegetables, and dietary energy density (kJ/g (kcal/g)) were used as exposure measures in this study as these have been repeatedly associated with overweight and obesity risk( 23 – Reference Howarth, Saltzman and Roberts 25 ). Fibre density was derived by calculating grams of NSP fibre intake (g) consumed per 4184 kJ (1000 kcal) EI. Fruits and vegetables were defined on the basis of the WHO Global Strategy on Diet, Physical Activity and Health, and included all fruits and vegetables reported by participants excluding potatoes, nuts and juices( 26 ). SoFAS were defined by the USDA Dietary Guidelines for Americans as high-energy low nutrient-dense food items that need to be limited( 24 ). Dietary energy density was calculated using two definitions: (i) dividing the total energy from foods/beverages (kJ (kcal)) by total weight of foods/beverages (g) or (ii) as above, using foods alone (excluding all drinks)( Reference Ledikwe, Blanck and Kettel Khan 27 – Reference Johnson, Mander and Jones 29 ).
All analyses were performed in terms of EI and not the absolute amounts to reduce extraneous variability and to control for confounding( Reference Livingstone, Robson and Black 30 , Reference Willett and Stampfer 31 ). The effect of EI was evaluated as a 100-unit offset from the mean, whereas for all other dietary variables a 1-unit change was applied. Descriptive analyses were stratified by sex and age categories, as defined in the Institute of Medicine (IOM) Dietary Reference Intakes ( 32 ).
Identification of implausible reporters
Each respondent was categorised as either under-reporter, plausible reporter or over-reporter on the basis of the comparison of their total estimated energy requirement (EER) with their reported EI. The EER was calculated using the IOM factorial equations that were developed from meta-analyses of studies using double-labelled water as the criterion measure of EER( 33 ). These equations use participants’ age, sex, BMI, weight, height and physical activity level (PAL) (sedentary, low active, moderately active, highly active) to estimate their EER( 33 ). As CCHS 2.2 only includes energy expenditure in terms of metabolic equivalents (MET) (kJ/kg per d (kcal/kg per d)), the IOM method was used to convert MET (intensity of an activity compared with the RMR) to the PAL (ratio of total energy expenditure:basal energy expenditure), which was then used in equations to predict EER( 33 ).
Among several methods developed for detecting implausible recalls, the method of McCrory et al. is currently the most detailed procedure by which EI is directly compared with EER using cut-off points for their agreements based on error propagation calculations( Reference Huang, Roberts and Howarth 8 , Reference McCrory, McCrory and Hajduk 14 , 34 ). This is important as other commonly used procedures to identify misreporters (e.g. Goldberg et al.( Reference Goldberg, Black and Jebb 13 )) are prone to several potential errors, especially in assigning appropriate PAL to individuals( Reference McCrory, McCrory and Hajduk 14 ). In this study, we applied McCrory et al.’s intervals for four different levels of physical activity, to data from adolescents and adults, using the level of physical activity each participant reported ( Reference Huang, Roberts and Howarth 8 , 34 , Reference Garriguet 35 ). As the EI distribution was skewed, we constructed the CI in the log scale and exponentiated the cut-off points, in line with previous studies( Reference Huang, Roberts and Howarth 8 , Reference Garriguet 35 ). On the basis of our data set, individuals whose EI was <70 % of their EER were classified as under-reporters, and those whose EI was >142 % of their EER were classified as over-reporters (±1 sd). Equations used for this calculations have been published elsewhere( Reference Huang, Roberts and Howarth 8 , Reference McCrory, McCrory and Hajduk 14 , 34 , Reference Garriguet 35 ). We additionally classified individuals on the basis of the ±2 sd cut-off points using the 50 and 198 % as the cut-off points for the EI:EER ratio.
Statistical analyses
All statistical analyses were performed using SAS software (version 9.3; SAS Institute Inc.). To account for the complex multistage sampling frame of the CCHS 2.2, variance estimation was performed using the bootstrap balanced repeated replication technique( Reference Rao, Wu and Yue 36 , Reference Yeo, Mantel and Liu 37 ) and the sample survey weights calculated by Statistics Canada. To maintain a nationally representative sample, a specific weight calculated by Statistics Canada was used in all analyses, which was based on respondent classes with similar socio-demographic characteristics. A two-tailed P value<0·05 was used to define statistical significance. Group comparison with Tukey’s post hoc adjustment was used to evaluate the characteristics of participants classified as under-reporters, plausible reporters and over-reporters (PROC SURVEYREG).
Calculation of the propensity score
Step-wise elimination in the logistic regression was applied to identify lifestyle and socio-demographic factors significantly associated with under-reporting among adolescents and adults. We started fitting a model containing all determinants of under-reporting mentioned in previous studies, and the backward selection procedure was applied to screen out non-relevant factors. The following variables were significant in the final model for adolescents: age, sex, physical activity, alcohol intake in the past 12 months, highest household education, self-reported health, smoking status, province of residence and income. For adults, variables that remained significant in the final model included the following: age, sex, physical activity, having a chronic disease, province of residence, highest household education, self-reported health and smoking status. BMI was not used in the construction of the propensity score as it was the main outcome in the present research( Reference Bornhorst, Huybrechts and Hebestreit 15 ). The conditional probability of being classified as an under-reporter given the above-mentioned variables was calculated for adolescents and adults using two separate multiple logistic regression models, as follows( Reference Bornhorst, Huybrechts and Hebestreit 15 ):
Propensity score=estimated probability (under-reporter│covariates).
Statistical models for handling misreporting
To compare the utility of different procedures for handling misreporting, the association of overweight/obesity was assessed in relation to a number of key food items identified by the WHO as the main determinants of obesity( 23 ). Multinomial logistic regression-generalised logit model (PROC SURVEYLOGISTIC) was conducted using a classification variable indicating overweight and obesity as outcomes of interest and six dietary variables as exposure measures: EI (kJ/d (kcal/d)), fibre density (g/4184 kJ (g/1000 kcal)), percentage of energy from SoFAS, percentage of energy from fruits and vegetables, total energy density (kJ/g (kcal/g)) and food-only energy density (kJ/g (kcal/g)). The following eight models were then examined and compared. The first model (basic model) was only adjusted for individuals’ age and sex (model I). The second model was the same as the basic model but also adjusted for all confounding variables used in calculation of the propensity score (model II). Model III, however, was identical to the basic model with the recalls identified as under-reporter or over-reporter using the method of McCrory et al.( Reference McCrory, McCrory and Hajduk 14 ) being removed (model III). Other models were the same as the basic model but additionally adjusted for the reporting group (model IV), propensity score (model V) and for both the reporting group and propensity score at the same time (model VI). Further analyses were conducted stratifying the analyses by the reporting group (model VII), and stratifying the analyses by the reporting group and adjusting for the propensity score simultaneously (model VIII).
Results
Part A: prevalence and determinants of misreporting
On the basis of the ±1 sd cut-off point, 40·47 % of Canadian adolescents and 42·3 % of adults were categorised as misreporters, whereas the corresponding percentages using the ±2 sd cut-off point were 12·18 and 13·6 %, respectively (online Supplementary Fig. S1). Throughout this study the more stringent cut-off point (±1 sd) was used for screening out implausible recalls.
Generally, the weighted mean ratio of EI:EER was significantly lower in overweight and obese individuals compared with their normal-weight counterparts (P<0·0001) (Table 1). In addition, the ratio of EI:EER decreased by age among both sexes and was lower for females compared with males; however, after approximately 50 years of age, men consistently showed lower total EI:EER values (P<0·024) (online Supplementary Fig. S2). In Table 1, disparity values between the reported EI and the recommended EER were also calculated in order to estimate the amount of EI being misreported among different age, sex and BMI categories. Disparity values were calculated by subtracting the IOM EER( 33 ) from the EI reported in the CCHS 2.2. Negative disparity values represent the magnitude of energy under-reporting, whereas positive values show over-reporting. Among normal-weight males, disparity values were positive and significantly higher in adolescents compared with adults (P=0·013). Disparity values among overweight and obese individuals were consistently negative, with the highest value being −3715 (se 460) kJ (−888 (se 110) kcal) among obese males aged 12–17 years, followed by −2769 (se 435) kJ (−662 (se 104) kcal) among obese females of the same age group (an under-reporting of approximately 25 % of EER).
Table 1 Weighted mean ratio of energy intake (EI):estimated energy requirement (EER) and disparity values (EI – EER) for Canadian adolescents (12–17 years) and adults (≥18 years) by BMI categories (n 16 187)Footnote * (Mean values with their standard errors)
F, females; M, males.
* Estimates are weighted means and bootstrapped variances (balanced repeated replication technique); EI was from the Canadian Community Health Survey 2.2 24- h dietary recalls and the EER was from the Institute of Medicine( 33 ) equations; for adolescents 12–17 years of age, Cole et al.’s( Reference Cole, Bellizzi and Flegal 22 ) categories were used to define obesity.
† Negative values indicate total kJ/d (kcal/d) of under-reporting.
‡ Significantly different between adolescents and adults in normal-weight individuals in each sex (P<0·05).
§ Significantly different between normal weight and overweight in each age and sex group (P<0·001).
║ Significantly different between normal weight and obese in each age and sex group (P<0·001).
¶ Significantly different between overweight and obese in each age and sex group (P<0·05).
** Significantly different between adolescents and adults in obese individuals in each sex (P<0·05).
†† Significantly different between males and females in normal-weight adults (P<0·01).
‡‡ Significantly different between males and females in obese adults (P<0·01).
Table 2 presents the descriptive analyses of several covariates stratified by reporting group (under-reporters, plausible reporters and over-reporters) among adults (≥18 years). Under-reporters were more likely to be older (P=0·0013) and to have higher BMI (P<0·0001), diabetes (51·13 v. 30·33 %; P<0·0001), hypertension (36·64 v. 30·40 %; P=0·0003), heart disease (36·08 v. 31·11 %; P=0·035) and at least one chronic disease (34·53 v. 28·92 %; P=0·0031). In addition, the percentage of under-reporters was higher among residents of Prairie provinces (Manitoba and Saskatchewan) and Ontario, those with secondary education or less and daily smokers (P<0·006). Similar results were observed among adolescents (12–17 years) (online Supplementary Table S1).
Table 2 Descriptive weighted analysis of covariates (row percentages) stratified by the reporting group (differential misreporting) among Canadian adults (≥18 years) (n 11 748)Footnote * (Mean values with their standard errors)
NFLD, Newfoundland; PEI, Prince Edward Island; NS, Nova Scotia; NB, New Brunswick; QC, Quebec; ON, Ontario; MB, Manitoba; SK, Saskatchewan; AB, Alberta; BC, British Columbia.
* Estimates are weighted means and bootstrapped variances (balanced repeated replication technique).
† Under-reporters: individuals whose energy intake (EI) was <70 % of their estimated energy requirement (EER).
‡ Plausible reporters: individuals whose EI was between 70 and 142 % of their EER.
§ Over-reporters: individuals whose EI was >142 % of their EER.
The weighted mean values of dietary determinants of obesity by reporting status are reported for different age and sex categories to examine evidence of potential ‘selective misreporting’ (Table 3). Adult under-reporters reported substantially lower mean EI (5999 (se 75) kJ/d (1434 (se 18) kcal/d) in males and 4497 (se 58) kJ/d (1075 (se 14) kcal/d) in females) compared with the plausible (10 924 (se 83) kJ/d (2611 (se 20) kcal/d) in males and 8229 (se 83) kJ/d (1967 (se 20) kcal/d) in females) and over-reporters 18 756 (se 297) kJ/d (4483 (se 71) kcal/d) in males and 13 669 (se 322) kJ/d (3267 (se 77) kcal/d) in females) (P trend<0·0001). Similarly, percentage of energy from SoFAS, total energy density and food-only energy density were significantly higher in over-reporter males and females compared with under- and plausible reporters (P trend<0·0001). In contrast, weighted mean fibre density and percentage of energy from fruits and vegetables were higher in under-reporters compared with plausible and over-reporters (P trend<0·0064). Similar selective misreporting of dietary variables was also observed among adolescents, although the magnitude was not as large as in adults, probably because of the lower rate of misreporting in younger individuals.
Table 3 Descriptive weighted analysis of dietary determinants of obesity as set by the WHO stratified by the reporting group (selective misreporting) among Canadian adults (≥18 years) and adolescents (12–17 years) (n 16 187)Footnote * (Mean values with their standard errors)
EI, energy intake; SoFAS, solid fats and added sugars.
* Estimates are weighted means and bootstrapped variances (balanced repeated replication technique).
† Under-reporters: individuals whose EI was <70 % of their estimated energy requirement (EER).
‡ Plausible reporters: individuals whose EI was between 70 and 142 % of their EER.
§ Over-reporters: individuals whose EI was >142 % of their EER.
║ SoFAS were defined by the US Department of Agriculture & US Department of Health and Human Services( 24 ) as high-energy low nutrient-dense food items that need to be limited.
¶ Fruits and vegetables were defined based on the World Health Organization( 26 ) excluding potatoes, nuts and juices.
** Total energy density was calculated dividing the total EI from foods and beverages (kJ (kcal)) by total food and beverages weight (g)( Reference Ledikwe, Blanck and Kettel Khan 27 – Reference Johnson, Mander and Jones 29 ).
†† Food-only energy density was calculated dividing the total EI from foods (kJ (kcal)) by total weight of foods (g)( Reference Perez-Escamilla, Obbagy and Altman 28 , Reference Johnson, Mander and Jones 29 ).
Part B: comparison of different methods to handle misreporting
Table 4 shows the OR and 95 % CI obtained from six different regression models for the association between overweight and obesity as outcomes and several dietary variables as exposures in adults. In the basic model (model I) adjusted for age and sex, a significant negative association was seen between EI and overweight (OR 0·988; 95 % CI 0·979, 0·998; P=0·0135) and obesity (OR 0·989; 95 % CI 0·977, 0·999; P=0·0553). A similar inverse association was observed between total energy density and overweight (OR 0·670; 95 % CI 0·487, 0·923; P=0·0142) and obesity (not significant). More specifically, only the association between fibre density and food-based energy density with obesity was significant and in the expected direction in model I (P<0·0019). Adjusting for covariates (model II) revealed very similar OR for all dietary exposures so that the direction and significance of none of the variables changed. The strongest relationship between overweight, obesity and dietary exposures was seen after excluding misreporters from the analyses (model III). In this model, significantly positive associations between EI and overweight (OR 1·045; 95 % CI 1·021, 1·070) and obesity (OR 1·139; 95 % CI 1·108, 1·171) were observed (P<0·0001), as well as direct positive associations between percentage of energy from SoFAS, total energy density and food-only energy density with obesity risk (P<0·0028). Furthermore, the negative association between fibre density and percentage of energy from fruits and vegetables with obesity was changed to be strong and significant (P<0·0205). Including all respondents and adjusting for the reporting group (model IV) revealed similar results that were slightly less pronounced compared with the model excluding misreporters (model III). After adjusting for the propensity score (model V), all associations changed to be similar to the model II (adjusted for the covariates) and be no longer in the expected direction. Finally, adjustment for both the propensity score and the adjusting group (model VI) did not improve results compared with adjusting for the reporting group alone. Additional inclusion of dietary variables into the propensity score calculation did not improve the results (data not shown). The same results were confirmed among adolescents where excluding misreporters (model III) yielded the strongest association between most dietary exposures and overweight and obesity (online Supplementary Table S2).
Table 4 Association between overweight and obesity risk with dietary determinants of obesity as set by the WHO among Canadian adults (≥18 years)Footnote * (Odds ratios and 95 % confidence intervals)
SoFAS, solid fat and added sugar; EI, energy intake; %E, percentage of EI.
* Estimates are weighted and variances are bootstrapped using the balance repeated replication technique. For the propensity score, 0·01 unit offset from mean was chosen due to its small scale and for the EI a 100-unit offset from mean was considered. All other continuous variables were assessed based on 1-unit offset from the mean. Under-reporters: individuals whose EI was <70 % of their estimated energy requirement (EER); plausible reporters: individuals whose EI was between 70 and 142 % of their EER; over-reporters: individuals whose EI was >142 % of their EER.
† Model I: weighted multinomial logistic regression adjusted for age and sex.
‡ Model II: model I additionally adjusted for physical activity, having a chronic disease, province of residence, highest household education, self-reported health and smoking status.
§ Model III: basic model but excluding under-reporters and over-reporters.
‖ Model IV: basic model adjusted for the reporting groups (under-reporters, plausible reporters, over-reporters).
¶ Model V: basic model adjusted for propensity score.
** Model VI: basic model adjusted for both propensity score and the reporting group.
In adults, when the basic model was stratified by the reporting group, only EI was significantly associated with obesity in all three groups (under-reporters, plausible reporters and over-reporters) (model VII) (Table 5). Additional adjustment for the propensity score did not improve statistical models (model VIII), except for a slight improvement in EI associations with obesity. Generally, the association of most dietary variables with overweight and obesity was significant and in the expected direction among plausible reporters, even though the strongest association between most dietary variables and overweight and obesity was observed among under- or over-reporters (Table 5 and online Supplementary Table S3). Graphical representations of the relationship between EI and BMI among under-reporters, plausible reporters and over-reporters are presented in Fig. 1(a)–(d).
Fig. 1 The relationship between energy intake (EI) and BMI among under-reporters (
, UR), plausible reporters (
, PR) and over-reporters (
, OR) Canadian adults (≥18 years) and adolescents (12–17 years). (a) Adult males; (b) adult females; (c) adolescent males; (d) adolescent females. Estimates are weighted and variances are bootstrapped using the Balance Repeated Replication technique. ±1 sd cut-off point for plausible reporting: 0·7≤EI/estimated energy requirement≤1·42. To convert kcal to kJ, multiply by 4·184.
Table 5 Association between overweight and obesity with dietary determinants of obesity as set by the WHO in different models stratified by the reporting group among Canadian adults (≥18 years)Footnote * (Odds ratios and 95 % confidence intervals)
EI, energy intake; %E, percentage of EI; SoFAS, solid fats and added sugars.
* Estimates are weighted and variances are bootstrapped using the balance repeated replication technique. Under-reporters: individuals whose EI was <70 % of their estimated energy requirement (EER); plausible reporters: individuals whose EI was between 70 and 142 % of their EER; over-reporters: individuals whose EI was >142 % of their EER.
† Basic model (adjusted for age and sex) stratified by under-reporting, plausible reporting and over-reporting.
‡ Basic model (adjusted for age and sex) additionally adjusted for the propensity score and stratified by under-reporting, plausible reporting and over-reporting.
Part C: agreement of subjective and objective measures of recall validity
To test the validity of participants’ self-reported usual intakes, we additionally compared the self-reported usual intake amounts with the ±1 sd cut-off point for the agreement between EI and EER. As presented in online Supplementary Fig. S3(a) and (b), 58·95 (se 1·04) % of adults and 60·06 (se 1·55) % of adolescents who said they consumed ‘their usual amount’ were actually plausible reporters and 29·99 (se 1·34) % and 20·23 (se 1·22) % of these individuals under-reported their intakes. Among those who reported consuming ‘less than the usual’ amount of food, only 42·58 (se 2·13) % of adults and 21·72 (se 2·05) % of adolescents were under-reporters and 49·75 (se 2·10) % and 58·62 (se 3·17) % reported accurately. In addition, of those who reported that they consumed ‘more than the usual’ amount only 16·98 (se 2·87) % of adults and 31·11 (se 4·22) % of adolescents actually over-reported their intakes.
Discussion
To our knowledge, this is the first study using a large-scale national survey data on adolescents and adults (self-reports) to compare seven different statistical approaches to counteract attenuation of risk estimates caused by misreporting. Consistent with previous studies on differential misreporting by weight and disease status( Reference Mendez, Popkin and Buckland 12 , Reference Bornhorst, Huybrechts and Hebestreit 15 , Reference Archer, Hand and Blair 38 , Reference Broyles, Harris and Taren 39 ), under-reporters were more likely to be obese and have higher rates of chronic diseases compared with the plausible and over-reporters. In addition, our results showed strong evidence of selective misreporting in line with others( Reference Subar, Kipnis and Troiano 4 , Reference Heitmann, Lissner and Osler 40 , Reference Voss, Kroke and Klipstein-Grobusch 41 ), where under-reporters reported significantly higher intakes of healthy foods – such as fibre, fruits and vegetables – and lower intakes of energy and energy-dense foods. Given the high prevalence of such systematic differential and selective misreporting, statistical models that neglected misreporting of EI rendered the association of nearly all dietary exposures with obesity as insignificant or even reversed, even though the variables studied have been strongly suggested by the WHO as major determinants of overweight and obesity( 23 ). In addition, the nature of the relationship between dietary variables and obesity was different among different reporting groups (under-reporters, plausible reporters and over-reporters). Exclusion of misreporters, adjusting for the reporting groups and stratification resulted in risk estimates that were more consistent with the established hypotheses regarding the role of dietary variables in obesity( 23 , Reference Tohill, Seymour and Serdula 42 , Reference Rolls, Drewnowski and Ledikwe 43 ). Particularly, adjusting for the reporting group yielded more consistent results, even when compared with those from plausible reporters in stratified analyses, and it provided the maximum sample size while maintaining biological plausibility.
In line with a previous study( Reference McCrory, McCrory and Hajduk 14 ), findings of this research showed a significant disagreement between objective and subjective measures of intake validity, which suggests that, although individual’s self-defined ‘usual amount’ may be within the normal range of day-to-day intake variation, this does not necessarily translates into the ‘habitual’ amount needed to maintain the current body weight( Reference McCrory, McCrory and Hajduk 14 ). In addition, this inconsistency confirms that individual’s self-assessment of intake amounts cannot be used for identification of inaccurate recalls in nutritional surveys( Reference McCrory, McCrory and Hajduk 14 ).
Thus far, only a few studies have investigated methods of dealing with implausible recalls( Reference Nielsen and Adair 1 , Reference Huang, Roberts and Howarth 8 , Reference Mendez, Popkin and Buckland 12 , Reference Bornhorst, Huybrechts and Hebestreit 15 ). Huang et al.( Reference Huang, Roberts and Howarth 8 ) in 2005 evaluated this issue and concluded that lack of exclusion of misreporters from the analyses results in non-significant, weak and misleading diet–obesity relationships. However, these authors did not consider the loss of statistical power that occurs as a result of excluding such large number of participants from the analyses and the fact that results would no longer be generalisable to the entire population, because misreporters have unique characteristics that are not shared by the plausible reporters (i.e. differential misreporting)( Reference Huang, Roberts and Howarth 8 , Reference Broyles, Harris and Taren 39 , Reference Bornhorst, Huybrechts and Ahrens 44 ), as also clearly demonstrated in our study. In addition, extreme observations and outliers usually contain valuable information about the outcome of interest, and their exclusion may introduce an unknown bias( Reference Livingstone and Black 45 ). Even though excluding misreporters may strengthen the diet–disease relationships, as was also seen in the present study, it is not an appropriate methodology and may lead to a selection bias( Reference Mendez, Popkin and Buckland 12 , Reference Bornhorst, Huybrechts and Hebestreit 15 , Reference Livingstone and Black 45 , Reference Poslusna, Ruprich and de Vries 46 ). Generally, results from the study by Huang et al.( Reference Huang, Roberts and Howarth 8 ) should be interpreted with caution as all individuals were assumed to be low active for EER calculations because of the lack of data on PAL, which could potentially lead to misclassification of additional subjects to the under-reporting group.
Another study on alternative methods of dealing with inaccurate recalls was based on the National Health and Nutrition Examination Survey (NHANES) 1999–2002 data and concluded that stratification by the intake level is more representative of population nutrient intakes compared with data elimination or exclusions( Reference Nielsen and Adair 1 ). These authors observed a significant association between EI and BMI only among plausible reporters and not the total sample( Reference Nielsen and Adair 1 ). This also supports our findings for the total group where no significant association was observed (model I). Nevertheless, the association of EI with BMI in our study was significant for nearly all reporting groups (under-reporters, plausible reporters and over-reporters), which is in line with a previous study( Reference Bornhorst, Huybrechts and Hebestreit 15 ). Generally, the limitations of stratification (models VII and VIII in our study) are similar to those of data exclusion, as it results in reduced sample size and loss of statistical power, especially among the over-reporter group (smaller n)( Reference Bornhorst, Huybrechts and Hebestreit 15 ). An important limitation of the study by Nielsen & Adair( Reference Nielsen and Adair 1 ) is the use of a modified Goldberg et al.( Reference Goldberg, Black and Jebb 13 ) method for identifying misreporters, which assumes a certain habitual PAL for individuals without accounting for the error in assigning this PAL. In our study, however, EI was directly compared with EER using cut-off points for their agreements based on error propagation calculations( Reference Huang, Roberts and Howarth 8 , 34 ). This is important as previous studies have noted a very low precision for assigning PAL to individuals, which may also explain the lack of sensitivity of the Goldberg et al.( Reference Goldberg, Black and Jebb 13 ) cut-off point for identifying inaccurate dietary reports( Reference Black 47 ). Another limitation of the Goldberg et al.( Reference Goldberg, Black and Jebb 13 ) method is that only extremely inaccurate recalls (±2 sd) are identified, even though misreporting can occur to varying degrees.
In 2011, Mendez et al.( Reference Mendez, Popkin and Buckland 12 ) concluded that adjusting for the reporting status through inclusion of a dummy variable for reporting group resulted in stronger associations between diet and obesity and yielded results similar to when misreporters were excluded from the analyses. Our findings corroborate these conclusions; adjustment for the reporting group maintained the statistical power and shifted the association of dietary exposures with overweight and obesity to the expected direction among Canadian adolescents and adults. Although the study by Mendez et al.( Reference Mendez, Popkin and Buckland 12 ) was the first to systematically compare the effect of ‘adjusting for the reporting group’ with ‘excluding misreporters’, it suffers from the same limitation as other previous studies in the field, which is assumption of a habitual PAL for all participants without consideration of error in assigning this PAL.
The most recent study that explored different methods of handling misreporting was conducted on children aged 2–9 years (proxy reports) and was the first to calculate and apply a propensity score for handling inaccurate recalls( Reference Bornhorst, Huybrechts and Hebestreit 15 ). These authors concluded that mutual adjustment for the reporting group and a propensity score is a useful tool for obtaining unbiased risk estimates in obesity research on children( Reference Bornhorst, Huybrechts and Hebestreit 15 ). However, their findings may have been influenced by the proxy-reported nature of diet recalls and lack of consideration of children’s PAL in calculation of EER, for identifying misreporters and for developing the propensity score. Our study is the first on adolescents and adults to develop and apply the propensity score as a means of counteracting misreporting bias. We found that among adolescents and adults adjusting for the propensity score had no benefit over adjusting for the reporting group for improving the association between dietary exposures and obesity. This discrepancy may reflect higher differential and selective misreporting among adolescents and adults compared with children, which may not be simply accounted for by inclusion of a propensity score or other calibration methods, which assume a linear (non-differential) measurement error with a constant variance( Reference Spiegelman, McDermott and Rosner 48 – Reference Kaaks, Riboli and van Staveren 50 ).
Future studies could test the applicability of constructing calibration scores based on biomarker data in large-scale national surveys where dietary measures are also available for the same subjects. Although exclusion of misreporters strengthened the diet–obesity relationships in this study, it is not an appropriate strategy because of the introduction of an unknown bias by exclusion of about 40 % of the population (misreporters) who are systematically different from the plausible reporters (different lifestyle and higher obesity and chronic disease risk). In the absence of biomarker measurements in the Canadian national nutrition survey, our results suggest that simple adjustment for the reporting group is superior to other statistical techniques for handling the misreporting bias, retaining adequate power among adolescents and adults.
Strengths and limitations
This is the largest known study to compare seven different statistical approaches to address the misreporting bias among adolescents and adults in a nationally representative sample, and it provides important knowledge on the critical role of handling misreporting in obesity research. Developing specific cut-off limits for defining misreporting on the basis of participants’ PAL and the algorithm-based method of McCrory et al.( Reference McCrory, McCrory and Hajduk 14 ) was one of the strengths of this research, compared with studies( Reference Archer, Hand and Blair 38 ) that mistakenly apply the first cut-off points used by Goldberg et al.( Reference Goldberg, Black and Jebb 13 ) in 1991 to identify misreporting( Reference Black 51 ). This is problematic as cut-off points should be derived on the basis of characteristics of the population being studied to avoid subject misclassification. Inclusion of various covariates, use of a large nationally representative sample and measured anthropometry are some of the other strengths of this study. In addition, the likelihood of misreporting due to missing items or eating occasions was minimised in this research as dietary data were collected using the USDA AMPM; therefore, some of our results may not be applicable to surveys with less comprehensive methods of dietary data collection.
One limitation of this study is day-to-day variation (random non-differential error) associated with dietary recalls, which may have weakened the associations between dietary intakes and obesity. In addition, causal inference is limited owing to the cross-sectional nature of this research( Reference Nielsen and Adair 1 ).
Conclusions and implications
The present study clearly demonstrated widespread prevalence of selective and differential misreporting across all age and sex groups in the Canadian national nutrition survey, which can undermine the validity of existing national dietary assessments, diet–disease relationships and public health policies that are developed based on these data, unless appropriate statistical methods are used to deal with such misreporting. Unlike some groups that concluded that national surveys have extremely limited ability for estimating EI and explaining the obesity epidemic( Reference Archer, Hand and Blair 38 ), we suggest that rigorous methods to control for the misreporting bias are needed and should be applied to any such analysis.
In this study, ‘adjusting for the reporting group’ maintained the statistical power and shifted the association of dietary exposures with obesity to the expected direction. These results can help advance knowledge about the relationship between dietary variables and obesity and demonstrate to obesity researchers and nutrition policy makers the importance of adjusting for recall plausibility in obesity research. Future studies that assess the sensitivity and specificity of different statistical techniques for correcting the misreporting bias against reference biomarkers of dietary intakes will further advance our abilities to handle misreporting in epidemiological and national cross-sectional studies.
Acknowledgements
The authors are grateful to staff at Statistics Canada’s Research Data Centre, especially Carmina Ng, for their technical support and Didier Garriguet at Statistics Canada for his professional comments.
This research was supported by funds to the Canadian Research Data Centre Network from the Social Science and Humanities Research Council, the Canadian Institute for Health Research (CIHR), the Canadian Foundation for Innovation and Statistics Canada.
M. J. is supported by the Canadian Institute of Health Research (CIHR) Vanier Canada Graduate Scholarship, and the CIHR/Cancer Care Ontario (CCO) Population Intervention for Chronic Disease Prevention (PICDP): a Pan-Canadian Fellowship. M. L. is the Earle W. McHenry professor and is supported by the chair-endowed unrestricted research funds, University of Toronto. Funders had no role in the design, analysis or writing of this article.
Contributions of each author are described as follows: M. J. conceptualised and designed the study; M. J. conducted the statistical analyses and prepared the data tables; W. Y. L, M. J. and M. R. L provided scientific input into the statistical analyses and manuscript; M. J. led drafting of the manuscript; and all authors approved the final version to be published. All of the authors contributed intellectually to the final manuscript.
None of authors declared personal or financial conflicts of interest.
Supplementary material
For supplementary material/s referred to in this article, please visit http://dx.doi.org/doi:10.1017/S0007114515004237
