Validity and reproducibility of a FFQ for assessing dietary intake among residents of northeast China: northeast cohort study of China

Abstract

The study was to evaluate the reproducibility and validity of the FFQ for residents of northeast China. A total of 131 participants completed two FFQ (FFQ1 and FFQ2) within a 3-month period, 125 participants completed 8-d weighed diet records (WDR) and 112 participants completed blood biomarker testing. Reproducibility was measured by comparing nutrient and food intake between FFQ1 and FFQ2. The validity of the FFQ was assessed by WDR and the triad method. The Spearman correlation coefficients (SCC) and intraclass correlation coefficients (ICC) for reproducibility ranged from 0·41 to 0·69 (median = 0·53) and from 0·18 to 0·68 (median = 0·53) for energy and nutrients and from 0·37 to 0·73 (median = 0·59) and from 0·33 to 0·86 (median = 0·60) for food groups, respectively. The classifications of same or adjacent quartiles ranged from 73·64 to 93·80 % for both FFQ. The crude SCC between the FFQ and WDR ranged from 0·27 to 0·55 (median = 0·46) for the energy and nutrients and from 0·26 to 0·70 (median = 0·52) for food groups, and classifications of the same or adjacent quartiles ranged from 65·32 to 86·29 %. The triad method indicated that validation coefficients for the FFQ were above 0·3 for most nutrients, which indicated a moderate or high level of validity. The FFQ that was developed for residents of northeast China for the Northeast Cohort Study of China is reliable and valid for assessing the intake of most foods and nutrients.

As one of the main determinants of chronic diseases, diet plays a key role in preventing diet-related diseases⁽¹⁾. However, the accurate measurement of dietary intake is still an ongoing challenge in diet-related disease studies. In large-scale epidemiological studies, a FFQ is the most commonly used tool for determining habitual dietary intake because it is easy to manage and apply, has a low cost and is less burdensome for participants⁽²⁾. The FFQ can rank individuals according to the intake of nutrients and foods, and it is further used to explore associations between diets and diseases⁽³⁾. The reproducibility and validity of the FFQ should be evaluated before its application because dietary habits can vary greatly according to the ethnicity and social and cultural backgrounds of the target population⁽⁴⁾.

The reproducibility of the FFQ reflects the consistency of the same subject at different time points⁽⁵⁾. Traditionally, validation studies of FFQ have been based on comparisons with more accurate measurement methods. The error of the reference method is required to be independent of the FFQ⁽⁶⁾. The weighed diet record (WDR) has been regarded as the best reference method for assessing the validity of the FFQ⁽⁷⁾. Additionally, biomarkers are considered objective, and their measurement errors are independent of self-report methods, such as the FFQ and WDR, showing that biomarkers are useful for further verifying the validity of the FFQ⁽⁸⁾.

Many studies of the validity and reproducibility of FFQ have been previously conducted in different areas of China, such as Shanghai⁽⁹⁾, Chengdu⁽¹⁰⁾, Guangdong⁽¹¹⁾, Nanjing⁽¹²⁾ and Chaoshan^(5,13) . China has a large population and vast territory. Different regions have different eating habits. Therefore, the validity and reproducibility of the FFQ need to be evaluated among populations in other areas of China.

For the ongoing Northeast Cohort Study of China (NEC-Biobank), we developed a new FFQ to estimate nutrient and food intake among the population of northeast China⁽¹⁴⁾. The NEC-Biobank was a large, prospective, dynamic cohort study that focused on major chronic diseases and related risk factors in various populations. The FFQ is culture specific. Thus, its reproducibility and validity need to be verified before its application. The present study evaluated the reproducibility of the FFQ based on a test–retest method and assessed its validity based on WDR and biomarkers as reference methods.

Methods

Study population

The participants in this study were randomly recruited from the NEC-Biobank. Participants who met the following criteria were selected for the study: residents who lived in northeast China for more than 5 years, 18–80 years old and had no serious diseases (e.g. cancer and cerebral thrombosis). The participant recruitment flow diagram is shown in Fig. 1(a). A total of 150 participants were recruited for the study between October 2018 and September 2019. Of 150 participants, forty-two participants were recruited in winter (October 2018–December 2018), thirty-six participants were recruited in spring (January 2019–March 2019), forty participants were recruited in summer (April 2019–June 2019) and thirty-two participants were recruited in autumn (July 2019–September 2019).

Fig. 1. The study design of FFQ reproducibility and validation study. (a) Flow diagram of sample selection; (b) sequence of validation study measurements.

The exclusion criteria for the reproducibility study were as follows: participants who did not complete both the first FFQ (FFQ1) and second FFQ (FFQ2) (n = 10), participants whose diet changed during study participation (n = 1) and participants who had extreme total energy intake values (> 14644 kJ (3500 kcal) or < 2510·4 kJ (600 kcal) for females, > 17572·8 kJ (4200 kcal) or < 3347·2 kJ (800 kcal) for males; n = 8)⁽⁷⁾. After applying the inclusion and exclusion criteria, a total of 131 participants were included in the reproducibility study. For the validation study, four participants did not complete WDR, and two participants had extreme total energy intake values. A total of 125 participants were included in the validation study. Additionally, 112 participants completed the biomarker test and were included in the ‘Triad Method’ study. Finally, fifty participants completed four 4-d WDRs and the third FFQ (FFQ3).

This study was conducted according to the guidelines laid down in the Declaration of Helsinki, and all procedures involving human subjects were approved by the Ethics Committee of Shengjing Hospital of China Medical University (Shenyang, China) (ethics number: 2017PS190K). Written informed consent was obtained from all subjects.

Study design

The sequence of the study measurements is shown in Fig. 1(b). The participants completed the FFQ1 when they were recruited for the study. The same participants completed FFQ2 and FFQ3 approximately 3 and 12 months later, respectively. During the study period, four 4-d WDRs (WDR1, WDR2, WDR3 and WDR4) were collected from each participant at intervals of 3 months. WDR1 was obtained 1 week after administering FFQ1, and WDR4 had to be completed before FFQ3. Blood samples were collected and tested on the day FFQ1 was completed.

Sample size calculation

According to previous studies, the range of correlation coefficients between FFQ and reference methods was 0·4–0·6⁽¹⁵⁾. The sample size was estimated according to the following formula: $n = (Z_{\alpha } + Z_{\beta})^2\sigma^{2}/d^{2}$ , with Fisher’s Z transformation of correlation coefficients, where σ ² = 1 for the Z-scale⁽⁷⁾, α = 0·05 and (1−β) = 0·80. After calculation, the number of required participants was 110⁽⁷⁾.

FFQ

The FFQ in this study was a modified version of the FFQ used in the Tianjin Chronic Low-grade Systemic Inflammation and Health (TCLSIH) cohort study that contained 100 food items. The TCLSIH cohort study was a large prospective dynamic cohort study conducted in Tianjin (a city also located in northern China)^(16,17) . We added ten food items according to the dietary characteristics of people in northeast China. The FFQ contained 110 food items. Dietary intake was assessed using the interview-administered FFQ, in which participants reported their usual consumption frequency over the past year, according to seven categories: never, 2–3 times/month, 1 time/week, 2–3 times/week, 4–6 times/week, 1 time/day and ≥ 2 times/day. For seasonal foods, such as fruits and Chinese sauerkraut, the participants were asked to choose the season and frequency of consumption during the season.

In addition to dietary information, we collected basic information from the questionnaire, such as general demographics and lifestyles. The height and weight were obtained from physical examinations, and BMI was calculated by dividing weight (kg) by height squared (m²).

Weighed diet records as reference

We used four 4-d WDRs (WDR1, WDR2, WDR3 and WDR4) as the reference method. Each WDR consisted of dietary records for four non-consecutive days, including three work days and one weekend. For daily dietary information, the participants were asked to record data and place of dietary intake, food name, ingredient composition and measured weight of the consumed food and beverage (weight before and after meals). For mixed dishes, participants were asked to weigh the dishes they consumed and calculate each ingredient composition according to the percentage of each food ingredient.

The participants received detailed instructions about how to weigh and record the amounts of foods and beverages that were consumed during meals. We also provided participants with analytical food scales with 0·1 g precision. We asked participants to check their diet records after each completion.

Biomarker selection

Although serum levels of Mg are regulated by the homoeostatic regulation mechanism⁽⁷⁾, the correlation between serum Mg concentration and daily dietary Mg intake was reported be 0·28⁽¹⁸⁾, which suggests that serum Mg can be used to measure dietary intake in clinical practice. In addition, serum concentrations of Fe, vitamin E, vitamin C, riboflavin, thiamine and fatty acids (such as PUFA, MUFA and SFA) are related to dietary intake^(19–23), which suggests that concentrations of these nutrients are useful biomarkers for intake⁽⁷⁾. Thus, in this study, we selected serum Mg, Fe, vitamin E, vitamin C, riboflavin, thiamine and fatty acids to assess the validity of the FFQ.

Biochemical measurements

After fasting overnight, a venous blood sample was drawn and collected in EDTA tubes. Before processing, serum was transferred to two 1 ml tubes. One portion of the serum was given directly to the Laboratory of Medicine of Shengjing Hospital of China Medical University (Shenyang, China) to measure serum ions and serum vitamins. Serum ions (Ca, P, K, Mg and Fe) were analysed by a BH5300S Atomic Absorption Spectrometer (BOHUI). Serum vitamins (vitamin A, thiamine, riboflavin, niacin, vitamin C and vitamin E) were quantified using a mass spectrometer.

The remaining serum samples (> 300 μl) were frozen at −80°C before testing for fatty acids. The serum profile of fatty acids was tested by Applied Protein Technology. After thawing, chloroform and methanol solution were added to 200 μl samples. The supernatant was added to 2 ml of 1 % sulphuric acid-methanol at 80°C for 30 min. After methyl esterification, the fatty acid methyl esters were extracted in n-hexane and separated using an Agilent DB-WAX capillary column GC system. The samples were then evaluated using an Agilent 7890A/5975C mass spectrometer (Agilent). MSD ChemStation software was used to calculate the content of the serum profile of fatty acids in the samples.

Food and nutrient calculations

Nutrient datas were from Chinese Food Composition Tables (sixth edition)⁽²⁴⁾. The correct match between the reported food intakes to the Chinese Food Composition Tables was determined by two researchers with a background in nutrition. Double data entry was done by two different researchers, and the two data sets were then cross-checked for accuracy by a third researcher.

Nutrient intake was determined by multiplying the amount of each food consumed (in g) and the nutrient content/100 g of food. The estimated nutrient intake per day was summed to obtain the total daily nutrient intake. For nutrient intake estimation from FFQ, we first calculated the amount of each food in the food group based on data from the 8-d WDRs and defined the median value of each food as the portion size of each food. We then converted the frequency of food intake in the FFQ to the number of times it was consumed per day and calculated the average daily food intake by multiplying the portion size and average consumption by the frequency of intake. Afterwards, a custom Excel-based macro was used to calculate energy content and nutrient composition from each food multiplied by each food intake and extrapolated for day. All the calculations were processed by self-programmed Excel Macros.

Statistical analysis

In the study, we collected information from FFQ at the same time as the complication of WDR (WDR1 v. FFQ1, WDR2 v. FFQ2 and WDR4 v. FFQ3). The number of people lost to follow-up was large due to poor compliance during follow-up, which may also have affected the sample size. Therefore, FFQ1 and FFQ2 (3-month interval) were selected to assess the reproducibility of the FFQ. In addition, the FFQ was validated by referring to the 8-d WDRs (WDR1 and WDR2). To eliminate the effect of seasonality on the results and make the results more convincing, we conducted sensitivity analyses.

We calculated the median and interquartile range for nutrients and food groups based on the FFQ and WDR. Differences between methods were tested by the Wilcoxon signed-rank test.

Reproducibility of FFQ

To determine the reproducibility of the FFQ, Spearman correlation coefficients (SCC) and intraclass correlation coefficients (ICC) were calculated with crude and energy-adjusted values for the main food items and groups/nutrients. The main food items and groups/nutrients were classified into quartiles, and cross-classification was performed into exact, exact + adjacent and extreme quartiles. Weighted κ statistics (κ _w) were calculated to assess the agreement of both methods. κ _w > 0·60 indicated good agreement, κ _w = 0·41–0·60 indicated moderate agreement, κ _w = 0·21–0·40 indicated fair agreement and κ _w ≤ 0·20 indicated poor agreement⁽²⁵⁾. Energy-adjusted intake was calculated by adding the mean nutrient intake to the residual that was derived from the regression analysis⁽²⁶⁾.

We conducted subgroup analyses according to the characteristics of the participants (sex, age and education) and seasonality. In addition, we performed a sensitivity analysis of participants who completed the FFQ1 and FFQ3 (at a time interval of 12 months) to assess the reproducibility of the FFQ using the SCC and ICC.

Validity of FFQ

The average of the two FFQ (FFQ1 and FFQ2) was used in all analyses because the FFQ that was completed before the food records prevented the participants’ recording from being altered by their greater awareness. However, comparing the food records with the first FFQ (FFQ1) may underestimate validity because the FFQ asks about past intake⁽⁷⁾. Therefore, the use of the mean FFQ value before and after the WDR allowed the minimal and maximal estimation of true validity⁽⁷⁾. The following statistical methods were used for the FFQ validation study.

First, the relative validity of the FFQ was obtained by calculating crude, energy-adjusted and de-attenuated SCC. Energy-adjusted intake was calculated by adding the mean nutrient intake to the residual that was derived from the regression analysis⁽²⁶⁾. De-attenuated correlation coefficients⁽²⁷⁾ were calculated to remove within-person variability using the following formula: ${r_t} = {r_0}\sqrt {1 + r/n} \;$ , where r _t is the true correlation, r ₀ is the observed correlation, r is the ratio of within- and between-person variances and n is the number of WDR (n = 8).

Second, cross-classification was used to estimate the consistency of the FFQ and WDR. All of the participants were divided into quartiles according to their intake of crude and energy-adjusted main food items and groups/nutrients. We then calculated the percentage of agreement between the two methods and analysed cross-classification into exact, exact + adjacent and extreme quartiles. The weighted κ statistic was calculated to assess agreement between both methods.

Third, Bland–Altman analyses were used to assess agreement between the FFQ and WDR for the main food items and groups/nutrients. This analysis was used to evaluate agreement between two different measurements to determine the precision of one method relative to a reference method⁽²⁸⁾. For the scatter plots, the 95 % limit of agreement (LOA) was calculated as the mean difference ± 1·96 standard deviation (SD).

Fourth, validation coefficients (VC) were calculated among the three studied variables (FFQ, WDR and biomarkers) from the nutrient consumption correlation coefficients between the estimated dietary and serum level methods, as proposed by the triad method⁽²⁹⁾. VC were divided into three categories: high (> 0·6), moderate (0·2–0·6) and low (< 0·2)⁽²⁹⁾. Bootstrap sampling was used to determine the 95 % CI for VC⁽³⁰⁾. A total of 1000 bootstrap samples (n = 112) of the same size were obtained by random sampling and replaced as research objects. We assessed the VC of FFQ1 because FFQ1 and blood were collected at the same time.

We conducted subgroup analyses according to the characteristics of the participants (sex, age and education) and seasonality. In addition, we performed a sensitivity analysis restricted to participants who collected 16-d WDRs to assess the validity of the FFQ using SCC.

Results

Studied population

The general characteristics of the study population are presented in Table 1. For the reproducibility study, the mean age of the participants was 31·50 years, thirty-two participants (25·8 %) were men and 105 participants (80·2 %) had an education level above college. The mean BMI of the participants was 22·66 kg/m², and forty-six (35·1 %) participants were obese or overweight (> 24 kg/m²). For the validation study, the mean age of the participants was 31·05 years, thirty-six participants (28·8 %) were men and 101 participants (80·8 %) had an education level above college. The mean BMI was 22·58 kg/m², and forty-two (33·6 %) participants were obese or overweight (> 24 kg/m²). The proportion of frequent supplement users in this population ranged from 0·8 % to 4·8 % (online Supplementary Table S1).

Table 1. The characteristics of participants in the reproducibility and validation studies

(numbers and percentages; mean values and standard deviations)

BMI = weight (kg)/height (m)².

Reproducibility of the FFQ

Reproducibility of nutrient intake

As shown in Table 2, we found that thirteen out of thirty-five selected nutrients were significantly different between FFQ1 and FFQ2. In addition, we found that SCC for energy and all nutrients were above 0·40. The crude and energy-adjusted SCC between FFQ1 and FFQ2 for reproducibility ranged from 0·51 in PUFA to 0·74 in fibre (median = 0·63) and from 0·41 in thiamine to 0·69 in fibre (median = 0·53), respectively. The crude and energy-adjusted ICC ranged from 0·17 in retinol to 0·67 in fibre and insoluble fibre (median = 0·62) and from 0·18 in retinol to 0·68 in Mn (median = 0·53), respectively. The crude values were greater than 0·50 for all nutrients except vitamin A (ICC = 0·23), retinol (ICC = 0·17), DHA (ICC = 0·46) and β-carotene (ICC = 0·46). The weighted κ values ranged from 0·26 for retinol to 0·53 for folate (median = 0·38), indicating fair or moderate agreement.

Table 2. Reproducibility of FFQ for energy and nutrients intake

(median values and percentiles)

FFQ1, first FFQ administration; FFQ2, second FFQ administration; 25th–75th, 25th–75th percentile; SCC, Spearman correlation coefficient; ICC, intraclass correlation coefficient; ALA, α-linolenic acid; κ _w, weighted κ.

* P < 0·05.

† P values were derived from the Wilcoxon signed-rank test.

‡ Energy-adjusted intakes by the residual method.

When the nutrient intake was categorised into quartiles (online Supplementary Table S2), the agreement rates for the same or adjacent quartile classifications ranged from 76·15 % (MUFA) to 91·60 % (α-carotene). After energy adjustment, the range was from 73·64 % (protein) to 90·84 % (insoluble fibre). The crude and energy-adjusted nutrients into opposite quartiles ranged from 0 % (energy, insoluble fibre, SFA, DHA, DPA, vitamin A, α-carotene, retinol, vitamin E, Fe, Se, Mn) to 3·85 % (PUFA) and from 0 % (Cu) to 7·75 % (niacin), respectively.

We also conducted subgroup analyses according to sex, age and education level (online Supplementary Table S3) and found that 76·5 % (26/34), 52·9 % (18/34) and 38·2 % (13/34) of nutrients for energy-adjusted correlations were higher in males than in females, in older participants than in younger participants and in well-educated participants than in less educated participants, respectively.

In addition, we conducted subgroup analyses according to season (online Supplementary Table S4). The energy-adjusted correlation between FFQ1 and FFQ2 was from 0·20 to 0·77 (median = 0·64) for spring, from 0·33 to 0·78 (median = 0·50) for summer, from 0·20 to 0·77 (median = 0·64) for autumn and from 0·33 to 0·78 (median = 0·47) for winter.

As shown in online Supplementary Table S4, we conducted sensitivity analysis to assess the reproducibility of the FFQ (FFQ1 and FFQ3) at 12-month intervals (n = 50). The crude and energy-adjusted SCC ranged from 0·52 to 0·79 (median = 0·68) and from 0·38 to 0·71 (median = 0·54), respectively. The ranges of crude and energy-adjusted ICC were from 0·19 to 0·76 (median = 0·635) and from 0·15 to 0·66 (median = 0·40), respectively.

Reproducibility of main food items and groups intake

We calculated the intake of food items and groups and assessed the reproducibility of the FFQ. The assessed food items and groups are presented in online Supplementary Table S5. The reproducibility of the FFQ for the main food items and group intake is presented in Table 3. SCCs between FFQ1 and FFQ2 for the main food items and group intake were above 0·50 except for red meat (SCC = 0·44). The crude and energy-adjusted SCCs between FFQ1 and FFQ2 ranged from 0·44 in red meat to 0·85 in coffee (median = 0·67) and from 0·37 in nuts to 0·73 in poultry (median = 0·59), respectively. The ICCs for the main food items and group intake were above 0·50 except for red meat (ICC = 0·36), offal (ICC = 0·31), tubers (ICC = 0·31) and alcohol (ICC = 0·37). The crude and energy-adjusted ICC ranged from 0·31 in offal to 0·86 in tea and coffee (median = 0·61) and from 0·33 in offal to 0·86 in tea (median = 0·60). Weighted κ values ranged from 0·24 in nuts to 0·56 in poultry (median = 0·43).

Table 3. Reproducibility of FFQ for main food item and food groups intake

(median values and percentiles)

FFQ1, first FFQ administration; FFQ2, second FFQ administration; 25th–75th, 25th–75th percentile; SCC, Spearman correlation coefficient; ICC, intraclass correlation coefficient; κ _w, weighted κ.

* P < 0·05.

† P values were derived from the Wilcoxon signed-rank test.

‡ Energy-adjusted intakes by the residual method.

As shown in online Supplementary Table S6, the percentage of participants who were classified into the same or adjacent and extreme quartile by the FFQ and WDR ranged from 79·85 % to 92·25 % (median = 85·38 %) and from 0 % to 11·63 % (median = 1·55 %), respectively. After energy adjustment, the same or adjacent and extreme quartiles ranged from 75·97 % to 93·80 % (median = 85·27 %) and from 0 % to 8·53 % (median = 3·08 %), respectively.

As shown in online Supplementary Table S7, we conducted subgroup analyses of the main food items and group intake according to sex, age and education level. The energy-adjusted correlation coefficients were higher among males than females for 63·2 % (12/19) of the main food items and groups, among the low-age group than the old-age group for 63·2 % (12/19) of the main food items and groups, and among participants with a high education level than a low level for 52·6 % (10/19) of the main food items and groups.

We conducted subgroup analyses of the main food items and group intake according to season (online Supplementary Table S8) and found that the energy-adjusted SCC ranged from 0·32 to 0·80 (median = 0·63) for spring, from 0·26 to 0·81 (median =0·53) for summer, from 0·34 to 0·80 (median = 0·63) for autumn and from 0·40 to 0·81 (median = 0·54) for winter. The sensitivity analysis (online Supplementary Table S8) showed that the ranges of crude and energy-adjusted SCC for FFQ reproducibility with 12-month intervals were 0·34–0·76 (median = 0·66) and 0·17–0·69 (median = 0·53), respectively. The crude and energy-adjusted ICC values ranged from 0·23 to 0·80 (median = 0·56) and from 0·18 to 0·81 (median = 0·48), respectively.

Validity of the FFQ

Validity of nutrient intake

The results of the validity of the FFQ for energy and nutrient intake are presented in Table 4. We found that nineteen out of the thirty-five selected nutrients were significantly different between FFQ and WDR. Although 32·4 % (11/34) of the energy-adjusted SCC (0·27–0·56) were lower than the crude coefficients (0·26–0·57), de-attenuation slightly improved the SCC for most variables. The energy-adjusted and de-attenuated SCC ranged from 0·28 in α-carotene to 0·56 in P (median = 0·39), and the values of all the nutrients were above 0·30 except for cholesterol (SCC = 0·28), α-carotene (SCC = 0·28) and Ca (SCC = 0·29). Weighted κ values ranged from 0·14 to 0·37 (median = 0·28), and all values were above 0·20 except for fibre (κ _w = 0·15), insoluble fibre (κ _w = 0·14), cholesterol (κ _w = 0·17) and Se (κ _w = 0·18).

Table 4. Relative validity of the FFQ for energy and nutrients intake

(median values and percentiles)

FFQ, mean of FFQ1 and FFQ2; WDR, weighed diet record; 25th–75th, 25th–75th percentile; SCC, Spearman correlation coefficient; ALA, α-linolenic acid; κ _w, weighted κ.

* P < 0·05.

† P values were derived from Wilcoxon signed-rank test.

‡ Energy-adjusted intakes by the residual method.

For the classification into quartiles (online Supplementary Table S9), the proportion of participants who were classified into the same or adjacent and extreme quartiles by the FFQ and WDR ranged from 68·42 % for insoluble fibre to 83·06 % for vitamin E (median =77·42 %) and from 0 % for P to 9·77 % for insoluble fibre (median = 4·27 %), respectively.

The results of the Bland–Altman analyses are shown in online Supplementary Table S10. Less than 9 % of the participants were outside the 95 % LOA for crude and energy-adjusted nutrients. The percentage of participants with values outside the 95 % LOA for crude and energy-adjusted nutrients ranged from 1·61 % (cholesterol) to 7·26 % (Fe) and from 2·42 % (protein) to 8·73 % (Se), respectively. To illustrate the LOA between the two methods, Bland–Altman scatter plots were generated for daily energy, protein (energy-adjusted), fat (energy-adjusted) and carbohydrate (energy-adjusted) intake (Fig. 2).

Fig. 2. Bland–Altman plots assessing the agreement between the FFQ and the 8-d weighed dietary record (WDR) in estimating the intakes of: (a) energy, (b) carbohydrate (energy-adjusted), (c) fat (energy-adjusted) and (d) protein (energy-adjusted) among residents of northeast China. The mean intake of the two methods ((FFQ + WDR)/2). The difference in intake between the two methods (FFQ–WDR).

We conducted subgroup analyses according to sex, age and education level (online Supplementary Table S11). For energy-adjusted SCC of thirty-four nutrients, we found that males had higher values than females for eighteen nutrients, younger participants had higher values than older participants for twenty-three nutrients and participants with higher education levels had higher correlation coefficients than lower education for fourteen nutrients.

As shown in online Supplementary Table S12, the ranges of energy-adjusted SCC were 0·13–0·54 (median = 0·30) for spring, 0·15–0·69 (median = 0·37) for summer, 0·09–0·71 (median = 0·33) for autumn and 0·14–0·62 (median = 0·38) for winter. For sensitivity analysis, we found that the crude and energy-adjusted SCCs between FFQ (mean of FFQ1 and FFQ3) and WDR (mean of 16-d WDR) ranged from 0·13 to 0·62 (median = 0·335) and from 0·13 to 0·51 (median = 0·13), respectively.

Validity of main food item and group intake

As shown in Table 5, the crude and energy-adjusted SCC between the FFQ and WDR ranged from 0·26 in tubers to 0·70 in tea (median = 0·52) and from 0·10 in offal to 0·65 in legumes and soya products (median = 0·46), respectively. De-attenuation slightly improved SCC for most variables. The de-attenuation values ranged from 0·26 to 0·71 (median =0·52), and all the values were above 0·30 except for tubers (SCC = 0·26). Agreement between the FFQ and WDR was fair or moderate (κ _w > 0·20) for the main food items and group intake except for fresh fruit (κ _w = 0·15), offal (κ _w = 0·19) and tubers (κ _w = 0·18).

Table 5. Relative validity of the FFQ for main food items and food groups intake

(median values and percentiles)

FFQ, mean of FFQ1 and FFQ2; WDR, 8-d weighed diet records; 25th–75th, 25th–75th percentile; SCC, Spearman correlation coefficient; κ _w, weighted κ.

* P < 0·05.

† P values were derived from Wilcoxon signed-rank test.

‡ Energy-adjusted intakes by the residual method.

As shown in online Supplementary Table S13, after energy adjustment, the ranges of agreement rates for the same or adjacent and extreme quartile classifications were 65·32 %–86·29 % (median = 78·23 %) and 0 %–12·90 % (median = 12·90 %), respectively, when the main food items and groups were derived from the FFQ and WDR. As shown in online Supplementary Table S14, the Bland–Altman analyses indicated that a few participants (< 9 %) fell outside the 95 % LOA. The percentage of subjects with values outside the 95 % LOA for crude and energy-adjusted main food items and groups ranged from 3·20 % (eggs) to 8·87 % (sugar drink) and from 2·40 % (eggs) to 8·06 % (sugar drink), respectively.

The validity coefficients between the FFQ and WDR, grouped by sex, age and education level, are shown in online Supplementary Table S15. We found that the energy-adjusted SCCs of the main food groups were higher for 31·6 % (6 of 19) of males, 42·1 % (11 of 19) of older participants and 57·9 % (10 of 19) of participants with a high level of education.

As shown in online Supplementary Table S16, we conducted subgroup analysis according to season and found that the energy-adjusted correlations ranged from 0·03 to 0·74 (median = 0·37) for spring, from 0·06 to 0·74 (median = 0·40) for summer, from 0·14 to 0·53 (median = 0·31) for autumn and from 0·15 to 0·69 (median = 0·36) for winter. For sensitivity analysis, we found that the crude and energy-adjusted SCC between FFQ (mean of FFQ1 and FFQ4) and WDR (mean of 16-d WDR) ranged from 0·18 to 0·77 (median = 0·47) and from 0·12 to 0·66 (median = 0·41), respectively.

Validation study by the triad method

As shown in online Supplementary Table S17, SCC between nutrients determined by the FFQ and serum biomarkers ranged from −0·10 (vitamin E) to 0·20 (Mg). The correlation coefficients were poor and non-significant except for Mg (SCC = 0·20), SFA (SCC = 0·14), MUFA (SCC = 0·13) and PUFA (SCC = 0·11).

As shown in Table 6, the VC of the FFQ ranged from 0·14 (thiamine) to 0·70 (SFA). The VCs of the FFQ were considered moderate (≥ 0·30) for all nutrients except for thiamine (VC = 0·14). The VCs of the FFQ were good (≥ 0·60) for Mg and SFA. Moreover, we found that the VCs of some nutrients that were tested could not be estimated because of negative sample correlation coefficients between the three measurements.

Table 6. Validity coefficients of the FFQ1, the WDR and the biomarker estimated by the method of triads

(coefficients and 95 % confidence intervals)

FFQ1, first FFQ administration; WDR, 8-d weighed diet records; VC_QT, validity coefficients of FFQ; VC_RT, validity coefficients of biomarker; VC_MT, validity coefficients of weighed diet record.

* Values > 1 were all set to 1·00.

† The correlation coefficient between the FFQ and the biomarker was the lower limit and the validity coefficient calculated by the method of triads was the upper limit.

Discussion

The present study investigated the reproducibility and validity of the FFQ for the Northeast Cohort Study of China, which was designed to capture the usual intake of nutrients and major foods among residents of northeast China. The reproducibility and validity of the FFQ were assessed at both the nutrient and food group levels. The results showed that the ICCs and SCCs between the two FFQ (FFQ1 and FFQ2) were above 0·5 for all nutrients and food groups except for vitamin A, retinol, DHA, β-carotene, offal, tubers and alcohol. In addition, the correlation coefficients between the FFQ and WDR were roughly between 0·3 and 0·7 for the nutrients and food groups, with the exception of cholesterol, Ca, α-carotene, offal and tubers. In addition, the VCs of the FFQ were above 0·3 for most nutrients, indicating moderate or high levels. Generally, the reproducibility and validity of the FFQ were acceptable.

Reproducibility study

In the present study, the differences between dietary intake from two FFQs were found to be significant for many nutrients and food groups, which was similar to the other studies^(12,31,32) . The reproducibility of FFQ has tended to depend on correlation analysis of nutrients and/or foods measured, with SCC and ICC roughly above 0·5, which indicated that the FFQ presented acceptable reproducibility to assess the dietary intake of participants⁽³³⁾. However, we found that the ICCs of vitamin A, retinol, DHA and β-carotene were relatively low. The reason is that the food sources of some nutrients are seasonal foods (e.g. DHA comes from fish and seafood; β-carotene comes from vegetables) or less common foods that change with a high frequency (e.g. retinol and vitamin A come from offal)⁽³⁴⁾. The intraindividual differences in the intake of these food groups may account for the low ICC. For the main food groups, the highest correlation coefficients were found for most food groups more frequently consumed (e.g. cereals, milks and dairy products and poultry) and habitual consumption of food or drink (e.g. tea, coffee and sugar drink). The low ICC value of alcohol may be due to that alcohol intake levels are relatively low in our study. Alcohol consumption is primarily driven by socially oriented motivations (e.g. attending a social event with friends who have alcohol), which may indicate differences in alcohol intake between the two FFQ sets at different times.

Compared with other studies in other areas of China^{(11,12,31,32,35,36)} , in which correlation coefficients generally ranged from 0·25 to 0·86 for nutrients and from 0·23 to 0·65 for main food groups, the correlation coefficients in the present study were similar or slightly higher. The time between administering the FFQ1 and FFQ2 were 6 months⁽³¹⁾, 9 months⁽³⁶⁾ and 12 months^{(11,12,32,35)} in these previous studies, whereas the interval between FFQ1 and FFQ2 in the present study was 3 months. One possible explanation for the higher reproducibility correlation in the present study was the time interval between FFQ1 and FFQ2⁽⁷⁾. The time interval is an important factor that influences the reproducibility of the FFQ⁽³³⁾. If the interval between two measurements is short, participants may remember and repeat their answers from the first questionnaire, which makes the relevance higher and the repeatability of the FFQ overestimated. Conversely, if the interval is long, the participants’ eating habits could change, resulting in lower FFQ reproducibility and consequently underestimation of the reproducibility coefficient. A previous study mentioned that to avoid changes in diet due to food seasonality and to analyse long-term trends in diet, an interval of 3 months may be suitable for reproducibility studies⁽³⁷⁾. However, reproducibility coefficients in the present study were lower than the coefficients (0·77–0·94 for food groups, 0·81–0·90 for nutrients) that were reported in the study of adult doctors and nurses who resided in the Chaoshan area of China, with an interval of 3 months between the two FFQs⁽⁵⁾. The main reason for the disparate results was that the respondents in the previous study were educated doctors and nurses, and the reproducibility of the FFQ increased with years of education⁽³⁸⁾.

Additionally, the results of the cross-classification analyses were similar to those of other reproducibility studies^(9,39–41) . Weighted κ values ranged from 0·21 to 0·60, indicating fair or moderate agreement, which is consistent with previous studies^(9,41) . The results showed that the FFQ had relatively acceptable reproducibility to assess dietary intake.

In addition, we performed a sensitivity analysis of participants who completed FFQ1 and FFQ3 (at a time interval of 12 months) to assess the reproducibility of FFQ. The main results of the study were similar to the sensitivity analyses, which indicated that seasonality has little effect on the reproducibility of the FFQ. One reason may be that our FFQ design takes into account seasonal food intake. With the development of the economy, our country’s diet has become more diversified, and ‘off-season’ food can be found at home for most of the year, which may be another reason⁽³¹⁾.

Validity study

Weighed dietary record

The differences between FFQ and WDR appeared to be significant for absolute intakes of most nutrients. Although similar results were reported by others^(12,30,31) , correlation coefficients have been widely used in FFQ validation studies. The correlations were found to be above 0·3 for most nutrients in the present study, which indicated that the FFQ was acceptable for assessing dietary intake⁽⁴²⁾. However, the validity coefficients of the FFQ for cholesterol, Ca, α-carotene, offal and tubers were relatively low; thus, it is not accurate in determining the magnitude of intake for these nutrients and food groups. The FFQ required participants to report their average diet consumed over the past year, while the WDR measured food eaten over a 4-d period; therefore, a few infrequently eaten foods have low validity (e.g. offal). In addition, a single food item in the FFQ includes different varieties of a particular food (e.g. red meat), which resulted in participants not being able to accurately assess FFQ dietary intake.

Compared with results from other studies in China^{(10,12,31,36)} , validity coefficients of the FFQ in the present study were similar or slightly lower. The reason for this difference could be that 24-h recall was used as the reference method to verify the validity of the FFQ in these previous studies^{(10,12,31,36)} , whereas the present study used WDR as the reference method, and its bias was independent of the FFQ⁽⁶⁾. The FFQ and 24-h recall have similar sources of error, such as their reliance on memory and perception of portion sizes. However, WDR and the FFQ have different sources of error^(43,44) , likely leading to fewer correlated errors with the FFQ, which may indicate the relatively low validity of our FFQ.

Additionally, compared with previous validation studies that were conducted in other countries, such as New Zealand (0·24–0·74 for nutrients)⁽²⁰⁾, Japan (0·08–0·94 for nutrients; 0·10–0·98 for food groups) and the USA (0·36–0·77 for nutrients)⁽⁴⁵⁾, the validity correlation coefficients in the present study were slightly lower. Since China countries have totally different dietary habits from Western countries, it is difficult to evaluate portion size accuracy due to diet complexities, which may affect the accuracy of food portion size intake estimations in dietary assessments. Compared with other studies in the Chaoshan area of China⁽¹³⁾, the VCs (0·31–0·53 for nutrients and 0·12–0·58 for food groups) were lower than those in the present study. Compared with the 3-d WDR used as the reference method for assessing the validity of the FFQ in Chaoshan, the reference method used in this study – the 4-d WDR – contained more food groups, possibly resulting in a relatively high validity coefficient.

Energy adjustment did not improve the correlations for nutrients and food items in our study. Although energy adjustment increases correlation coefficients when the variability of nutrient consumption is related to energy intake, the correlation coefficients decrease when the variability of nutrient consumption depends on systematic errors of overestimation and underestimation⁽⁷⁾. The lower correlations in the present study may be explained by an increase in correlated measurement error as a consequence of controlling for total energy intake. The weak associations for some food groups may in part be due to the within-subject variance in the 8-d WDR. The calculation of de-attenuated correlation coefficients to correct for intraindividual variability has been used in many studies. Accordingly, de-attenuated SCC are slightly higher.

In addition, joint classification and the Bland–Altman method were applied in the present study to assess agreement. The classification of participants by nutrient intake level is often useful for comparing disease risk across categories of intake in epidemiological studies. For all nutrients and food groups, the majority of participants (65·3 %–86·3 %) were correctly classified into the same or adjacent quartile when comparing the FFQ with the WDR, which is consistent with previous studies^(11,46–48) . In addition, acceptable agreement (κ _w = 0·20–0·39) was obtained for most nutrients and food groups, which was consistent with other studies^(48,49) . Our results showed that the FFQ is acceptable for classifying participants’ nutrient intake, which may be useful for studying diet–disease associations. The Bland–Altman method was used to obtain further information about the relationship between the FFQ and the results obtained via the WDR. The results we observed are similar to those shown in the studies conducted by other studies^(41,50–52) , where a small number of individuals fell outside the recommended limits, confirming an acceptable level of agreement between both methods.

To avoid possible seasonal and weekday variation, we collected dietary information four times at 3-month intervals, and WDR had to cover weekdays and weekends. The FFQ were administered at the same time as the complication of WDR. The number of people lost to follow-up was large due to poor compliance during follow-up, which may also have affected the sample size. Therefore, 8-d WDR (WDR1 and WDR2) were used to assess the validity of the FFQ. Sensitivity analyses were performed on participants who collected 16-d WDR (four 4-d WDR at a 3-month interval) to assess the validity of FFQ, taking into account seasonality that might influence differences in reported intake. The results from the sensitivity analysis were similar to the main results, which indicated that our results were robust.

Biomarkers

We also used biomarkers as a reference to verify the validity of the FFQ. Such biomarkers (measurement bias) provide an objective measure of intake, in which errors are largely independent of errors that are associated with the FFQ (recall bias)^(5,27) . In the present study, SCC (–0·04 to 0·14) between serum biomarkers and the FFQ were weak and not statistically significant for most nutrients and were lower than those between WDR and biomarkers. First, the WDR presents short-term dietary intake, whereas the FFQ reflects eating habits over the past year. The low correlation between the levels of serum water-soluble vitamins (thiamine, riboflavin and vitamin C) may be due to the short storage time in the blood⁽⁵³⁾. Second, the correlations for water soluble vitamins and fatty acids were lower than the results of correlation analysis detected in adipose tissue⁽⁵⁴⁾. The different selection of biological samples may have contributed to the low correlations. Third, some biomarkers are easily affected by bioavailability and metabolism, leading to the instability of biomarkers, which may induce a low correlation of Fe⁽⁷⁾.

We used the triad method to assess the validity of the FFQ, considering the poor correlation coefficients between the FFQ and blood biomarkers. In this method, biomarkers should be used as an additional measurement rather than diet surveys⁽⁵⁵⁾. In the present study, the VC of the FFQ was considered moderate (≥ 0·3) for most nutrients, indicating that the FFQ was valid for estimating the intake of these nutrients in this study. Several studies reported the use of the triad method to validate the FFQ to assess fatty acid consumption^(30,56–59) , and their results were consistent with the present findings. However, the triad method also has certain limitations. For example, the triad method could be accompanied by the occurrence of Heywood cases⁽³⁰⁾, in which validity coefficients for some nutrients cannot be calculated because of negative correlations between the three measurements (e.g. vitamin C and vitamin E in the present study).

One strength of our study was the use of two reference methods (WDR and biomarkers) with different error sources to validate the FFQ. Another strength was that reproducibility and validity of the FFQ were based on comparing levels of intake of nutrients (macronutrients and micronutrients) and food groups, which makes our results reliable, effective and more convincing.

The present study also has some limitations. First, the FFQ and food records in the present study were administered during only one season for each individual, which does not take into account seasonal changes in diet. However, both the subgroup analysis and sensitivity analysis suggested that the general result was robust. Second, blood collection for biomarker analysis was only performed once and did not reflect long-term dietary intake. However, this is common in validation studies due to the high cost of the determinations. Third, Heywood case events were found for some of the assessed nutrients, indicating that the diet method had related errors. The Heywood cases are mainly due to random sampling fluctuations⁽²⁹⁾. Increasing the sample size may reduce the amplitude of random sampling fluctuation to some extent⁽⁶⁰⁾. Fourth, we did not take into account the intake of dietary supplements during the nutrient intake calculation, which may underestimate nutrition intake, particularly for frequent supplement users. However, the effect of dietary supplements on nutrient intake is likely to be small in the population because the use of dietary supplements is rare.

Conclusion

Our dietary assessment is both reproducible and valid, with acceptable correlations for most nutrients and food groups. The reproducibility coefficients for the FFQ were acceptable for all nutrients and food groups except for vitamin A, retinol, DHA, β-carotene, offal, tubers and alcohol. In addition, the relative validity showed that the ranking ability was acceptable for estimated intake except for vitamin A, retinol, DHA, β-carotene, offal, tubers and alcohol. This result indicated that the FFQ was developed for residents in northeast China and is reasonably reliable and valid for assessing the intake of most foods and nutrients. However, caution should be taken when interpreting estimation from the FFQ.