Missing data in FFQs: making assumptions about item non-response

Abstract

Objective

FFQs are a popular method of capturing dietary information in epidemiological studies and may be used to derive dietary exposures such as nutrient intake or overall dietary patterns and diet quality. As FFQs can involve large numbers of questions, participants may fail to respond to all questions, leaving researchers to decide how to deal with missing data when deriving intake measures. The aim of the present commentary is to discuss the current practice for dealing with item non-response in FFQs and to propose a research agenda for reporting and handling missing data in FFQs.

Results

Single imputation techniques, such as zero imputation (assuming no consumption of the item) or mean imputation, are commonly used to deal with item non-response in FFQs. However, single imputation methods make strong assumptions about the missing data mechanism and do not reflect the uncertainty created by the missing data. This can lead to incorrect inference about associations between diet and health outcomes. Although the use of multiple imputation methods in epidemiology has increased, these have seldom been used in the field of nutritional epidemiology to address missing data in FFQs. We discuss methods for dealing with item non-response in FFQs, highlighting the assumptions made under each approach.

Conclusions

Researchers analysing FFQs should ensure that missing data are handled appropriately and clearly report how missing data were treated in analyses. Simulation studies are required to enable systematic evaluation of the utility of various methods for handling item non-response in FFQs under different assumptions about the missing data mechanism.

Accurate assessment of dietary intake is challenging, given the vast range of foods available for consumption and the difficulty of recalling past food and beverage consumption. FFQs are a popular method of capturing dietary information in epidemiological studies as they are relatively inexpensive and simple to conduct. FFQs estimate usual consumption frequencies of foods, which are commonly used to derive usual intakes of nutrients, dietary patterns and diet quality, and are particularly useful for ranking participants according to intake^{( 1 , 2 )}. However, FFQs can be burdensome for study participants to complete as detailed assessment may require inclusion of many food items, with up to 350 questions used in some cases^{( 3 )}. As a consequence, participants may fail to complete all questions (termed ‘item non-response’), leaving researchers to decide how to best deal with these missing data. Although many studies have examined the validation and design of FFQs^{( 4 )}, few have considered how various methods used to deal with missing data in FFQs can influence study findings. Potential approaches for dealing with missing data in nutritional epidemiology have been described elsewhere as part of a discussion of wider data challenges in this field^{( 5 )}. However, particular issues when dealing with missing FFQ data require further elaboration.

Given recognition of the importance of overall dietary patterns, as opposed to individual foods or nutrients, in influencing health and disease^{( 6 , 7 )}, FFQs are increasingly being used to measure diet quality through the calculation of composite scores. These scores are commonly derived by adding sub-scores reflecting consumption of foods or nutrients consistent with current dietary guidelines or recommendations aimed at reducing chronic disease, resulting in a single value representing the healthfulness of the total diet. Missing data in FFQs are problematic in calculations of diet quality scores or nutrient intake, as item non-response means that these dietary exposures cannot be calculated.

The purpose of the present commentary is to review and critique strategies for dealing with item non-response in FFQs. We conclude by proposing a research agenda that can help to determine the most appropriate methods for dealing with item non-response in FFQs.

Methods for dealing with item non-response in FFQs

Reported methods for dealing with item non-response in FFQs have varied, with many studies failing to report the approach adopted^{( 8 , 9 )}. Most commonly, investigators have used complete case analysis (described below) or assumed that missing responses indicate no consumption of that particular food item^{( 10 – 12 )}. Although imputing zero for missing responses is acceptable in some situations (e.g. quantity of alcohol consumed when participants previously indicated that they do not drink alcohol), it is possible for FFQ items to be missing without a known rationale. Therefore zero imputation may not be an appropriate strategy to adopt for all missing FFQ items. In other instances investigators have imputed the median, mode or mean value from the sample for the missing food items^{( 13 )}. Less often, more complex statistical approaches such as multiple imputation have been used^{( 14 )}. The various approaches that have been used to deal with item non-response in FFQs are described below, along with their various assumptions.

Complete case analysis

Complete case analysis is often used in epidemiological studies as this is the standard approach that statistical software packages adopt when dealing with missing data. In complete case analysis, only participants with complete data for all study variables to be considered in the analysis are included. This means that the dietary exposure (e.g. nutrient intake or diet quality score) would not be derived for a participant who had missing data on one of the FFQ items used to compute the exposure; instead, this participant’s data would be completely discarded from the analysis. This method can lead to biased findings if the missing data are not missing completely at random (MCAR). Data can be assumed to be MCAR when there are no systematic differences between the missing data and the observed data; that is, the probability of data being missing is unrelated to observed values of other variables or to the unobserved values themselves (e.g. FFQ data could be assumed to be MCAR if a participant accidentally skipped over a question). In addition to potential biases, complete case analyses can lead to a large reduction in sample size if missing data are common, resulting in reduced statistical precision and power. Furthermore, it is inefficient to exclude participants from the analysis who provided a substantial amount of complete data for many FFQ items but failed to respond to a small number of items.

Single imputation

Single imputation approaches are appealing as these are relatively simple to conduct. This type of approach replaces each missing value with only one value in the imputation process, producing a filled-in data set that allows standard analytical methods to be used. Single imputation approaches include zero imputation (where missing values are replaced with zero to represent no consumption of that food item) or mean, mode and median imputation, which replace missing values with the sample mean, mode or median, respectively. More complex single imputation approaches include k nearest neighbours imputation^{( 9 )}. This approach identifies k participants with similar responses on other variables to that of the participant with the missing data on the FFQ item, all of whom have complete data for the FFQ item to be imputed. The average value for that particular FFQ item among those k participants is taken as the FFQ item value for the participant with the missing response.

Single imputation methods have various shortcomings, with disadvantages including the very strong assumptions made. For example, assuming that the missing data equate to no consumption when using zero imputation has the potential to result in misleading findings if it is participants who do not want to report consumption of particular items (e.g. energy-dense, nutrient-poor foods) who have missing data on these items. Furthermore, single imputation methods do not reflect the uncertainty created by the missing data as they assume that the imputed values are real observations. This leads to underestimation of variances and poor coverage of confidence intervals^{( 15 )}, which can result in incorrect inferences about associations between exposures and outcomes. Thus, these approaches for dealing with missing data are not recommended, with journals and medical research guidelines increasingly turning towards multiple imputation as a preferred method for handling missing data in epidemiological studies^{( 16 , 17 )}.

Multiple imputation

Multiple imputation, like single imputation, produces filled-in data sets that enable standard analytical methods to be used, allowing partially observed information from all participants to be included in analyses. Each missing value is replaced with multiple imputed values that are generally drawn from a statistical model (known as an ‘imputation model’). This produces multiple filled-in data sets, each of which is then analysed separately using standard methods and the multiple results are then combined to give an overall estimate^{( 18 )}. A benefit of multiple imputation over single imputation is that it appropriately accounts for the uncertainty in the missing values, ensuring that the variance is not underestimated. It can also correct for bias, for example, if predictors of non-response are included in the imputation process. However, although multiple imputation approaches have been highlighted as potentially useful approaches for dealing with item non-response in FFQs^{( 9 )}, these have not often been used within nutritional epidemiology, although some nutritional cohorts, such as the Adventist Health Study 2 (AHS-2) cohort, have begun to routinely adopt this method^{( 19 , 20 )}. This is perhaps due to the complexity of these techniques and the lack of guidance about when these methods are suitable to use.

Multiple imputation involves the specification of an imputation model which should generally include all covariates to be considered in the underlying analytical model (of dietary exposure and health outcome), in addition to other covariates which may be associated with the variables that have missing data or that are predictive of non-response. These additional covariates, known as ‘auxiliary variables’, have been shown to increase efficiency and reduce bias^{( 21 )}. A particular challenge when using dietary exposures in epidemiological analyses is deciding which variables to include in the imputation model. Considering nutrient intake, it would seem more appropriate to impute individual food items as each individual food will potentially contribute to the score of multiple nutrients. Similarly, it would be ideal to retain all of the complete information available for individual FFQ items on which the diet quality score is based, rather than simply the complete score, in the imputation model. This is because using the complete score can result in inefficiencies as this will fail to consider complete FFQ items within the score, meaning that participants with missing data for only one FFQ item would have a missing score. However, multiple imputation can be computationally intensive, particularly when many categorical variables are included or when there are substantial missing data. Imputation models may fail to converge when they involve large numbers of FFQ items^{( 22 )}. One solution, although less efficient, could be to impute sub-scale scores in order to retain as much information as possible within a more parsimonious model, as discussed elsewhere^{( 23 )}. Although options for conducting multiple imputation are now available in standard statistical software, such as Stata and SAS, appropriate ways in which multiple imputation can best be used to address item non-response in FFQ data, and particularly when deriving dietary exposures, have received little attention in the literature to date.

A further challenge with multiple imputation is that most software implementations of multiple imputation assume that the missing data are missing at random (MAR). This means that the probability that data are missing does not depend on unobserved data (i.e. the missing values themselves), but may depend on observed data (i.e. the non-missing data)^{( 24 )}. An example of MAR in the context of diet assessment is that males may be more likely to have missing fruit and vegetable intake. If sex is observed, then within each sex category there will be no systematic differences between observed and missing data. However, if data are missing not at random (MNAR), meaning the probability that an FFQ response is missing depends on unobserved data (i.e. the missing values themselves)^{( 24 )}, then multiple imputation can give misleading results (as will most complete case analyses and single imputation methods). Given known issues affecting assessment of dietary intake, such as social desirability bias^{( 25 , 26 )}, it may be the case that FFQ data are MNAR if participants who have missing data on ‘unhealthy’ FFQ items (e.g. energy-dense, nutrient-poor foods) are those with high levels of consumption. Unfortunately, it is not possible to determine from the data if the missingness is MAR or MNAR. Therefore, as highlighted by Sterne et al.^{( 24 )}, researchers must consider the possible reasons why the missing data could have occurred in order to decide if MNAR could be a problem in their study. Strategies have been proposed to recover missing data when the MAR assumption may not be reasonable, such as guided multiple imputation where a random sub-sample of participants with missing data is contacted to recover the missing data. This information is then used in the imputation model^{( 27 )}. However, it is not always possible to re-contact a sub-sample of study participants to obtain this information. Therefore, it is important to consider potential strategies for dealing with missing FFQ data in this context as well as to perform sensitivity analyses to examine the robustness of results to departures from the assumptions made about the missing data mechanism. Commonly researchers employ a combination of these methods. For example, principles of complete case analysis may be applied in the first instance, with participants missing a large number of items excluded from the analysis. This has often been recommended as a form of global assessment for ensuring questionnaire fidelity and to allow exclusion of poorly completed questionnaires that would impact on validity of the dietary assessment^{( 1 )}. Subsequently, missing data may be subjected to imputation, with zero imputation a common approach^{( 9 )}.

Prior investigations of methods for dealing with item non-response in FFQs

Very few studies have compared approaches for dealing with missing data in FFQs. Of those that have, these investigations have typically compared differences in nutrient intakes when analyses include only individuals with no missing FFQ responses with analyses that include individuals whose initially missing responses have been subsequently filled in through resurveying them up to 2 years later^{( 28 )}. Findings from these studies tend to suggest that the common approach of assuming that non-response equates to not eating that particular food can introduce bias^{( 28 – 30 )}. One study examined a sub-sample of participants followed up 3 months after the initial FFQ to recover missing data in order to determine the influence of assuming that non-response equated to no consumption. Nutrient intake was derived using an eighty-four-item FFQ and results showed that average energy intake, among other dietary measures, increased following resurvey^{( 29 )}. A second study followed up a sub-sample of participants approximately 1 year after the initial FFQ and used a guided multiple imputation approach to derive complete data for those with missing observations for the eighty foods considered. This study showed that typically the resurveyed values of those who had missing data for commonly consumed foods tended not to be zero so the potential for bias may be greater if focusing on these foods^{( 30 )}. Both studies highlighted that their findings were influenced by the number of items that had missing responses, with one noting that the greater the number of missing food items, the greater the likelihood that they truly reflected no consumption of that item^{( 30 )}; while the other noted that the greater the number of items missing, the higher the increase in average energy intake^{( 29 )}. A third study involving a sixty-item FFQ noted that 37 % of the items that were initially missing were subsequently reported to be consumed at least monthly and so assuming no consumption would not be accurate for these participants^{( 28 )}. However, the authors concluded that the assumption of zero intake was reasonable if examining nutrient intake expressed as a percentage of energy. One drawback to these comparison studies is that they assume that resurveying participants at a later time point captures the true consumption at the earlier time point. This assumption may not be valid in practice given the well-known difficulties associated with accurately recalling dietary intake^{( 31 )}. It is therefore difficult to use these studies as a basis to infer the likely impact of assuming item non-response equates to zero consumption of the food item in question, although re-contacting participants may at least provide an idea of the direction of potential biases (i.e. whether participants with missing data were more likely to have consumed those foods).

To date, Parr et al.^{( 9 )} have provided the most comprehensive comparison of methods for dealing with item non-response in FFQs. In addition to recovering missing data by resurveying participants with missing data 3 months after they were originally surveyed, they compared various single imputation approaches for dealing with missing FFQ data, including zero imputation, mode imputation, median imputation and k nearest neighbours imputation. The authors were particularly interested in considering the implications of zero imputation as their review of the literature suggested that this was the most common method used when dealing with item non-response. In the context of the Norwegian Women and Cancer study, Parr et al. concluded that assuming no consumption of a food item for which no response was provided may lead to underestimation and misclassification of the dietary intake (measured as the calculated intakes of food groups and a range of nutrient intakes, rather than dietary patterns or quality) of study participants as the resurvey values suggested that it was not accurate to assume zero consumption of missing FFQ items. However, a major limitation of the study, acknowledged by the authors, is that they were unable to conclude which imputation method provided the most accurate estimate of consumption since they did not know the true values for the missing data. Thus, that study is unable to provide guidance on the optimal approach to address item non-response. Although the authors noted that an evaluation of imputation methods required a simulation study with a complete data set as reference, to our knowledge, no study of this type has been conducted. Furthermore, the authors did not examine the influence of different missing data assumptions on exposure and outcome associations involving FFQ measures. Therefore, the effect of these assumptions in epidemiological studies in practice is unclear.

Summary and research agenda

There are many approaches for dealing with item non-response in FFQs and all make quite different assumptions about how the missing data occurred. When deciding how to deal with missing FFQ responses, it is important to consider the potential reasons for the missing data. However, although existing studies that have examined missing FFQ data have provided useful information about potential biases when assuming missing data equates to no consumption of that item, these have not been able to clearly demonstrate the mechanisms behind the missing data, such as social desirability bias leading to a desire to avoid reporting the consumption of unhealthy foods, inability to recall consumption, questionnaire fatigue or simply randomly missing questions when completing lengthy questionnaires. Furthermore, to our knowledge, no studies have fully explored the best means of addressing item non-response in FFQs, the implications of assumptions related to missing data mechanisms and how these assumptions might impact associations between diet and health, and, in particular, in relation to the use of diet quality scores and, nutrient intake. Thus, there is a clear gap in the nutrition literature on appropriate approaches for dealing with missing FFQ data. Given the possible implications of poor handling of missing data (e.g. biased study findings), we propose this as an important research agenda; a significant issue that should be considered alongside measurement error issues frequently discussed in this field^{( 32 , 33 )}. While researchers can adopt different missing data methods to deal with item non-response, further research is required to determine the sensitivity of research findings to the various approaches and determine the implications of the different approaches. To do this, a rigorous evaluation of missing data methods involving simulation studies is required, as these allow systematic evaluation of the performance of these methods under different assumptions about the missing data mechanism.

In addition, considering the potential implications of the method adopted to address missing data on study findings, it is important that researchers working with FFQs are transparent in the reporting and handling of the missing FFQ responses. There are clear guidelines for reporting missing data and the details of multiple imputation analyses^{( 24 )}. Journals, such as the New England Journal of Medicine ^{( 16 )}, are increasingly recognising the need to handle missing data using principled methods and this is reflected in reporting guidelines such as the Consolidated Standards of Reporting Trials (CONSORT)^{( 34 )} and Strengthening the Reporting of Observational studies in Epidemiology (STROBE)^{( 35 )} statements. Despite this, missing data do not appear to be appropriately handled or adequately described in the field of nutrition. Moving forward, we propose a need for better reporting of how missing FFQ data were treated in analyses, consistent with these guidelines. Researchers should report both the amount of missing data in their studies and how the missing data were dealt with in analyses so that the assumptions are clear. Furthermore, they should provide information on sensitivity analyses conducted to assess the sensitivity of research findings to the assumptions being made. Specific guidelines relating to nutritional epidemiology (STROBE-nut) are currently under development^{( 36 )} and may assist in strengthening the reporting of studies, including the handling of missing data. With advancements in missing data methods and the availability of these methods in statistical software packages routinely used in nutritional epidemiology, such as Stata and SAS, it is now possible for researchers to make use of techniques such as multiple imputation to deal with missing FFQ data.

Conclusions

Missing data may be unavoidable when using long, self-administered questionnaires which can prove burdensome for participants to complete. The present commentary provided an overview of common techniques used to deal with item non-response in FFQ and the assumptions underlying each one. Researchers should take care to use robust techniques to address missing data in FFQ and to clearly and fully report the details of these analyses. Further research is required to systematically evaluate the performance of approaches for dealing with missing data in FFQ.