Complex interventions, particularly in the case of policy and community interventions, tend to generate multiple outcomes that can be difficult to combine when evaluating the intervention. The aim of the present project was to develop a realistic timeline for evaluation of nutrition-related community interventions. The model used in the present study was based on the anticipated impact of the Schools (Health Promotion and Nutrition) (Scotland) Act 2007(1) in primary schools. The Act was implemented in primary schools in August 2008 and aimed to improve the nutritional standards of food and drinks provided in schools and increase health promotion in schools.
Concept mapping is a method that has been used for developing evaluation frameworks for interventions and has been described as a technique for ‘simplifying complexity’(Reference Trochim2). It involves consultation with stakeholders and comprises three stages: brainstorming the anticipated outcomes; sorting outcome statements into higher-order themes; rating the outcome statements. A list of outcome statements was generated during a workshop held with academic researchers and policy makers. Participants were asked to complete the statement ‘I think the introduction of the Schools Act (2007) in primary schools will …’. Additional statements were generated from interviews carried out with key stakeholders from local authorities and the National Health Service (NHS) and focus groups run with parents of children in primary schools. A total of 177 statements were generated, which after reviewing and removing duplications left eighty-five statements for the sorting and rating. The statements were rated, using Likert scales, on their importance (from 1, not at all to 5, extremely important), the likelihood of them occurring (from 1, not at all to 5, extremely likely) and the earliest time to measurable impact (from 0, immediate to 6, ≥6 years). Participants from academic research, national government, government agency, local authority, NHS and community organisations completed sorting and rating the statements on an online website (Concept Systems software; Concept Systems Inc., Ithaca, NY, USA).
The statements were sorted into higher-order themes by 111 participants and 102 participants completed all three sets of ratings. Multidimensional scaling and hierarchical cluster analysis were used to combine the individually-sorted statements into meaningful concepts. Initially, fifteen clusters were identified with a stress value of 0.25 and this low stress value indicates a good fit of the model. The clusters comprised statements with similar themes (e.g. health issues, diet of children, attitudes, knowledge and food preferences, financial implications). The rating scores highlighted differences between the importance and the likelihood of the outcomes as well as the anticipated earliest time to measurable impact. For example, the ‘health’ cluster, which included issues of obesity, diabetes, dental health and blood pressure was rated highly for importance (4.29) but lower on likelihood of occurring (3.08) and scored the highest on the time to measureable impact (4.91). This result compared with the ‘knowledge & awareness of healthy foods’ cluster, which rated highest on likelihood of occurring (3.83) and importance (4.50) but had a much shorter time to measurable impact (2.69). In addition, potential unintended consequences were identified (e.g. increased food waste, increase appeal of banned food and drinks).
From these data it was possible to develop a realistic timeline of anticipated outcomes that provides an important framework for evaluation. An advantage of a timeline is that it identifies early indictors for the longer-term impact and likely success of an intervention, as well as providing realistic timescales for when health impacts should be expected. This timeline could be used as a basis for evaluation of other similar community-based nutrition interventions.
