Methods of defining best practice for population health approaches with obesity prevention as an example

Abstract

Childhood obesity has reached a crisis stage and has become a population health issue. The few traditional systematic reviews that have been done to identify best practice provide little direction for action. The concept of evidence-based practice has been adopted in health care, and in medicine in particular, to determine best practice. Evidence-based medicine has its origins in the scientific method and for many researchers this concept means strict adherence to standards determining internal validity in order to justify a practice as evidence based. Practitioners addressing population health face challenges in identifying criteria for determining evidence, in part because of the nature of population health with its goal of shifting the health of whole populations. As well, the type of evidence provided by more traditional critical appraisal schema is limiting. Expanded approaches in finding and defining evidence have been proposed that use: expert panels; broad and inclusive search and selection strategies; appraisal criteria that incorporate context and generalizability. A recent synthesis of 147 programmes addressing childhood overweight and obesity provides a concrete example of using a broader approach to identify evidence for best practice (Flynn et al.2006). Incorporating evaluation and population health frameworks as criterion components in addition to traditional methodological rigour criteria, this synthesis has identified programmes that provide contextual information that can be used to populate what Swinburn et al. (2005) have described as the ‘promise table’. Using this approach a range in ‘certainty of effectiveness’ and a range in ‘potential for population impact’ are integrated to identify promising strategies. The exercise can provide direction for agencies and practitioners in taking action to address obesity.

Abbreviation:

RCT

double-blind randomized placebo-controlled trial

The obesity challenge

It has been recognized for several years that obesity has reached epidemic proportions (World Health Organization, 2000; Ebbeling et al. 2002; Lobstein et al. 2004) and that prevention and treatment of childhood obesity is key to preventing the continuing increase in rates (Swinburn et al. 2005). Obesity has been described as a public health crisis because it is recognized that if it is not addressed successfully healthcare systems will be overwhelmed. No country has yet developed and implemented a coherent action plan to prevent further weight gain in the population to manage its current obesity epidemic (Swinburn et al. 2005). Obesity has moved beyond the simple problem of advising individuals to eat a healthy diet and be active. Obesity has become a major public health problem and population health approaches are required to turn around the epidemic rise in fatness levels that is now affecting whole populations. Although there are just two key elements involved in obesity prevention, healthy eating and active living, achieving a shift in patterns of whole populations is very complex. This complexity is demonstrated by the continuous rise in obesity prevalence despite the myriad of programmes that have been implemented to combat obesity. Whole-population shifts require multiple integrated strategies that are multidisciplinary and multisector, ranging from targeting individuals to groups to sectors (Health Canada, 2001). A range of strategies that address issues at the level of the individual, create supportive environments and build healthy public policy are all required. However, there is a paucity of what could be called strong evidence, from a traditional sense, on which to base best practice for population health approaches. Nonetheless, to do nothing is not an option. There is movement within public health, health systems and health-policy literature to reconsider the way in which evidence is derived. The development of a common understanding of the terms ‘best practice’ and ‘best evidence’ is crucial for moving forward on population health in general and on obesity in particular. Agreement on how evidence is defined is fundamental to reaching the collective consensus required. Some researchers suggest that the term ‘best evidence’ should be replaced with ‘best-available evidence’ (Swinburn et al. 2005). Others identify ‘better practice’ rather than ‘best practice’ because this term reflects the continual evolution in context and knowledge that is generated from research.

Best practice and evidence

‘Best practice’ is a term that is widely used in business and health care. It implies that there are approaches or practices to prevent, treat or cure that are the ‘best’ or at least ‘better’ to use than others. In the earliest history of medicine and health care ‘best practices’ were approaches that an individual or group of individuals used because in their ‘experience’ such approaches worked best to treat, cure or care for an individual with a health problem. Their experience was their evidence for practice.

Evidence-based practice is often used synonymously for ‘best practice’, meaning there is evidence to support a particular practice or intervention. Evidence generally has meant knowledge gained about a particular subject from using scientific methods. Scientific methodology has a long history; the first controlled experiments were conducted by Francis Bacon in the late 16th century, with the first description of the approach to scientific methods credited to Descartes in the early 17th century (Wikipedia, 2006). Controlled experimentation is the centre piece of the scientific method. Despite its >500-year-old history, evidence-based medicine, which is the use of scientific methods for decision making, is a relatively recent phenomenon.

Campbell & Stanley (1963) were the first to suggest that there is hierarchy of study designs, identifying that some approaches are more consistent with scientific methods and thus provide more robust evidence than others (Rychetnik et al. 2002). That premise was extended by the Canadian Task Force on the Periodic Health Examination (1979) with the idea of levels of evidence based on study design. It was further refined by the Centre for Reviews and Dissemination (Khan et al. 2001), the Centre for Evidence Based Medicine (Phillips et al. 2001) and others (Briss et al. 2000) and is relied upon for the development of clinical practice guidelines that are considered best practices. Levels I and II as shown in Table 1, considered to be the highest form or level of evidence, are versions of the randomized controlled trial with the first and most stringent trials having all personnel involved in the study blinded to knowledge of the group to which an individual has been randomized. Progressing down the hierarchy, lack of randomization lowers the level of evidence while having no control group on which to base comparisons is only a step above using opinion. Thus, through the establishment of a hierarchy and judging of evidence, the double-blind randomized placebo-controlled trial (RCT) is considered the most reliable source of evidence. The rationale for RCT as the most important form of evidence rests on the internal validity of the design (Des Jarlais et al. 2004). Internal validity in a study means that as much as possible bias and confounding have been taken care of along with other design considerations of adequate sample size and accurate and reliable measures. Randomization controls for known and unknown factors that may influence results, while blinding ensures that vested interests, such as those of investigators, subjects or stakeholders, do not influence study results. Internal validity ensures there is a high probability (usually 95%) or confidence that significant outcomes at the end of the study are the result of the intervention and not chance.

Table 1. Hierarchy of evidence (adapted from University of York NHS Centre for Reviews and Dissemination 2001; Khan et al. 2001)

RCT, double-blind randomized placebo-controlled trial.

Hierarchies of evidence place systematic reviews of RCT at the pinnacle of defining evidence. However, RCT may not be available for review because of cost, practicality, feasibility or appropriateness for health promotion interventions. If RCT are considered the only valid design there is a risk that evidence could actually be distorted by relying on a research design that does not match the research question or evaluation objectives (Green & Tones, 1999). Nonetheless, consistent results from multiple RCT, identified using a systematic review process as described by The Cochrane Collaboration (The Cochrane Collaboration, 2005) and Oxman (Oxman et al. 1991, 1994; Oxman, 1994), is considered the gold standard on which to base recommendations to change clinical medical practice. Systematic reviews of RCT consist of a carefully-constructed process of locating and critically appraising studies to answer very specific questions of very specific populations, usually about single interventions. In other words, they have the potential to answer very narrow questions very well.

While RCT have proven valuable for interventions focusing on individuals, for many areas of health practice, and in particular for health service delivery, there are few RCT and fewer systematic reviews of RCT to guide practice. Population health approaches in particular are not easily amenable to RCT designs for many reasons.

Limitations of double-blind randomized placebo-controlled trial designs for identifying evidence for population health

Interventions in public health may target individuals, groups, whole communities or populations (Green & Tones, 1999). The distinctions between clinical care and population health are many, starting with the objectives. The goal of population health is to shift the health of whole populations or communities. Evidence-based medicine approaches are designed for individuals (Green & Tones, 1999; Heller & Page, 2001) and generally short-term outcomes are considered as end points. Interventions to shift the health of whole populations take time, and thus for some public health practices surveillance data may provide better indicators of the success of multiple interventions than RCT (Heller & Page, 2001). In fact, some researchers would argue that it is unethical to include control groups who do not receive an intervention because of the length of time required to study the impact of interventions and the potential harm reduction that would be lost to control groups while waiting for results to become available.

According to Victora et al. (2004) RCT are useful when the causal chain between intervention and outcome is short and simple and results are easily extrapolated. On the other hand, public health interventions are usually large-scale, long-term and complex (Victora et al. 2004; Swinburn et al. 2005) and may not be easily packaged in a form that is amenable to random allocation (Green & Tones, 1999). In addition, the costs of mounting such large-scale studies may be prohibitive. Generally, for an RCT to be successful a single or limited set of variables is manipulated, which can be too limiting for examining effects on a full population.

The focus of population health and public health is often on the determinants of health and their interactions (McKinlay, 1992). Short- and medium-term outcomes as well as long-term outcomes are of interest. Thus, process and mediating factors (which may be on causal or contributing pathways) can be as important as long-term outcomes. Mediating factors of interest in public health are usually theory driven and often focused on human behaviours or environmental influences that are not easily amenable to experimental conditions. The appropriate methodology for gathering evidence on mediating factors might include qualitative approaches alone or in combination with more quantitative research designs (McKinlay, 1992; Green & Tones, 1999). The situation in clinical medicine, in which ‘bench’ research identifies molecular and other biological pathways for disease development and potential cures, is in direct contrast to this approach.

In public health target populations are free-living in environments that are influenced by a myriad of complex factors (many of which are inter-related). Such populations are generally healthy and need to be enticed to ‘comply’ with interventions to improve health. It is difficult for RCT designs to change with the changing focus of a population and the popularity of activities that might serve as intervention strategies.

As population health approaches take place in the real world, control is often not possible to the extent required to address issues of confounding and bias (Green & Tones, 1999; Swinburn et al. 2005). Efficacy is determined from experimentation in a controlled environment with little variation in procedure, i.e. effectiveness under ideal conditions (Green & Tones, 1999; Victora et al. 2004). The integrity of the setting, intervention and measurement are all maintained (Green & Tones, 1999; Victora et al. 2004). For public health all the interventions are taking place in ‘real world’ conditions (Green & Tones, 1999; Victora et al. 2004) and thus the results reflect the effectiveness, rather than the efficacy, of interventions.

As a result of the ‘real world’ nature of public health interventions context and process are crucial to understanding success. Success of an intervention depends not only on outcomes but also on the contextual factors that make up the environment in which a study takes place. In population health success is not just about outcomes, every stage of programme development is important, e.g. its popularity and the participant satisfaction with it.

External validity, or generalizability, appears to be a secondary concern in traditional approaches to critical appraisal that rely on RCT design (Mays et al. 2005). In contrast, assessment of external validity is a critical component of population health approaches used in public health, as it takes into consideration the contextual factors that provide information allowing for judgments on applicability beyond the study population. Relative to this issue, tracking of integrity of interventions (i.e. faithfulness to design and implementation over entire period of intervention) is important and not always reported in research reports and publications (Green & Tones, 1999; Flynn et al. 2006).

In fact, the Transparent Reporting of Evaluation with Non Randomized Designs Statement published in 2004 (Des Jarlais et al. 2004) seeks to improve the reporting quality of non-randomized evaluations of interventions. The criteria in this statement include some of the traditional standards in the Consolidated Standards of Reporting Trials Statement (Begg et al. 1996), such as a description of the planned study population, inclusion and exclusion criteria and flow of participants through each study phase (Des Jarlais et al. 2004). However, also included are factors such as barriers to implementing the intervention, how well the intervention stayed true to its original intent and compliance rates. These aspects, which can be critical for synthesis research endeavouring to develop best practice guidelines for population health approaches used in public health, are often missing in reports and publications.

Synthesis research

There are some researchers who consider the complexity of population health and health promotion interventions to be not amenable to traditional approaches of systematic review (Green & Tones, 1999; Victora et al. 2004; Flynn et al. 2006), and there are others who might argue that levels of evidence are important but not sufficient or necessarily required in determining best practice (Swinburn et al. 2005). There is concern by still others that the pendulum will swing too far in reverse and clinical trials will not be deemed useful in informing public health practice (Rychetnik et al. 2002). In fact, cluster randomized trials provide opportunities for randomization (Rychetnik et al. 2002) and advanced statistical modelling has been successful in addressing the bias and confounding that plagues less-stringent designs (Heller & Page, 2001). All these viewpoints are part of a growing body of literature and evolving consensus on the need for developing new approaches to defining and interpreting evidence for public health practice (Green & Tones, 1999; Heller & Page, 2001; Victora et al. 2004; Swinburn et al. 2005). A few published reports describe expanded approaches for conducting systematic reviews to identify best practice for population health approaches, which include the community guide to preventive services (Briss et al. 2000) and a review of methods used to synthesize evidence for effectiveness in health promotion (Jackson et al. 2001). In addition, those researchers concerned with making systematic reviews more relevant to policy makers have proposed even broader approaches to the process of synthesizing evidence (Atwood et al. 1997; Lomas, et al. 2005; Mays et al. 2005; Pawson et al. 2005), highlighting disparities in how researchers and policy makers define and describe evidence.

Reducing obesity in children: a synthesis

A recently-published synthesis research study, which has moved beyond the traditional approach of systematic reviews, provides a starting point on which to begin making judgments on best practices in population health (Flynn et al. 2006). This synthesis meets the criteria that are now being proposed to make systematic reviews and synthesis research useful to programme planners and policy makers. As a result of the number of programmes included and the broad approach used to critically appraise the research and evaluation designs, this synthesis can be used to begin to identify programmes that hold promise as examples of better practice in population health.

The methodology used in this synthesis included the following key innovations:

1. (1) using an international panel of experts and a concept model on population health (Hamilton & Bhatti, 1996) to guide the process;

2. (2) not limiting the review to English language reports;

3. (3) including critical appraisal criteria in addition to the usual assessment of scientific rigour; the aim being to identify as much information as possible on best practice relevant to interventions on population health.

The search strategy was wide and comprehensive, and covered eighteen library databases (>13 000 hits), >250 internet sites, a hand search of the International Journal of Obesity, reference lists from articles identified in the search and key informant surveys. Unlike other systematic reviews on best practice a purposely broad set of inclusion criteria was used. For example, at least one reported outcome (e.g. indices of overweight) or process indicator (e.g. number invited to participate v. number completing the programme) and few exclusion criteria (e.g. children with health problems) were used. While non-English language reports are commonly excluded, sixty-four non-English-language articles and reports were identified in this synthesis. This approach resulted in a high number of reports being eventually selected for critical appraisal (500 studies or reports of >900 identified had at least one process or outcome indicator; see Table 2, which shows the steps in the synthesis review process.

Table 2. Steps in synthesis review (adapted with permission from Flynn et al. 2006)

Another key difference between this synthesis research and other systematic reviews of obesity intervention or prevention is that critical appraisal was not limited to assessment of scientific or methodological rigour in quantitative studies. Each programme also received a critical appraisal for qualitative methodological rigour (if applicable), programme development and evaluation, relevancy of population health principles and use of approaches to support immigrant health (it should be noted that immigrant health was a sub-objective of the review but will not be discussed in the context of the present paper). In order to synthesize findings and reduce the number of reports and articles to a more-manageable number, a criterion-based scoring system was developed and only those scoring in the top one-third of the scoring range in any of the four appraisal approaches used were included for the synthesis of findings. This process gave most programmes four opportunities to be included in the final synthesis based on each appraisal level. This selection of only top-scoring programmes reduced the number to a more-manageable 147 programmes (based on 158 reports) on which to base the synthesis.

Thus, the use of multiple critical-appraisal perspectives not only increases the number of programmes included in the review but it also adds a much broader perspective to the review of specific programmes. In the context of the Flynn et al. (2006) synthesis review the addition of the appraisal of programme development resulted in the inclusion of more studies compared with previously-published systematic reviews focusing on childhood obesity and related factors, e.g. 147 compared with forty-two in a review by Jelalian & Saelens (1999). The use of the additional appraisal criteria also, however, facilitated consideration of other pertinent factors in the assessment of better practice that are of interest in population health. For example, the appraisal of population health principles was based on the population health promotion model (Hamilton & Bhatti, 1996) that served as the theoretical framework guiding this work. This assessment used scoring criteria that were based on the use of multidimensional strategies and the level of upstream investment. In fact, only one programme (Macaulay et al. 1997; Jimenez et al. 2003) was found to score highly on principles of population health (forty-eight programmes used some multidimensional approaches and forty included some ‘midstream’ approaches, which were primarily found in school-based settings). This dearth of population health approaches resulted despite the broad search strategy and inclusion criteria used (see Table 2 and Section 3.3 of Flynn et al. 2006). Clearly, this synthesis supports the contention of other researchers (Kumanyika, 2001; Swinburn & Egger, 2002) that there is a lack of published and unpublished research and evaluation of population health approaches to address childhood obesity (Flynn et al. 2006).

A final key component of the synthesis was the development of a ‘features table’, which outlined the main characteristics for each of the top-ranking 147 programmes according to their settings (school, home, community, clinic etc.; Flynn et al. 2005). These characteristics facilitated an understanding of populations addressed, key programme intervention features and outcomes. The remainder of the synthesis process included the identification of gaps in populations studied and programme foci and, in a more limited number of programmes for which data was adequate, a summary of best practices in programme development and programme effectiveness (Flynn et al. 2005).

The components of this synthesis research, and in particular the ‘features table’, provides the information that is needed to begin to populate the promise table described by Swinburn et al. (2005; see Table 3) that can be used by clinicians, health promotion specialists, community developers and policy makers to categorize the level of promise for potential interventions to address childhood obesity in their jurisdictions.

Table 3. Promise table (with permission, Swinburn et al. 2005)

Promising programmes

As part of the Prevention Group of the International Obesity Task Force, Swinburn et al. (2005) describe Hawe & Shiell's (1995) portfolio approach to health promotion, which allows for selection of programmes or strategies based on best-available evidence while not excluding untried but promising interventions. This approach is designed to encourage a balance of high- and low-risk investments (programmes) to maximize returns (outcomes) much in the same way as individuals or corporations would choose financial investments. It is accomplished by examining intervention or programme effectiveness and the potential for population impact using a matrix to classify strategies from least to most promising (Swinburn et al. 2005).

The basic premise is that judgements on the risks of investments are made based on two components ‘certainty of effectiveness’ and ‘potential for population impact’ (see Table 4). ‘Certainty of effectiveness’ can be evaluated by identifying a level of certainty (high, medium and low) based on an assessment of quality of evidence in the traditional sense by judging internal validity. Internal validity is based on the design and conduct of the study or evaluation, i.e. its methodological or scientific rigour. In addition, programme outcomes are also examined on the basis of programme effectiveness, i.e. did it work and were there negative side effects?

Table 4. Assessment components for promise table rankings*

* For details of promise table, see Table 3.

Three factors can be considered in examining the ‘potential for population impact’ component in order to identify the level (high, medium and low) of the ‘potential for population impact’:

1. (1) programme logic, where the theoretical basis of the programme is considered, including whether it is logical and consistent with current knowledge of physiology, behaviour etc.;

2. (2) reach, which takes into account range and breadth of participation, including characteristics of the study population;

3. (3) uptake, which includes utility (such as evidence and extent of stakeholder involvement) and feasibility (such as practicality and costs etc. including rates of participation).

This latter component broadens consideration criteria to incorporate contextual and policy relevance, implementation and sustainability factors that require review when considering adoption of programmes to specific settings (Swinburn et al. 2005).

The judgments made for ‘certainty of effectiveness’ and ‘potential for population impact’ can transfer to the promise-table matrix to determine relative promise. Thus, programmes of high ‘certainty of effectiveness’ but low ‘potential for population impact’ (e.g. a clinical programme to treat obesity at a summer camp) would rate as promising. On the other hand, programmes of low ‘certainty of effectiveness’ and high ‘potential for population impact’ (e.g. the Pathways schools intervention programme from the USA that had positive qualitative results on reach, uptake and programme logic (Davis et al. 1999; Gittelsohn et al. 1999) but lack of hard positive outcomes (Caballero et al. 2003)) could also be rated as promising.

The final component of the promise-table exercise is consideration of adoption within a specific context, which requires investment from stakeholders that understand the specific conditions that will impact feasibility and sustainability in a particular setting. This process underscores the ‘real world’ nature of population health work; the need for understanding programme implications from a local perspective and the complex analysis required to identify useful evidence for practice. Two programme examples previously reviewed in the Flynn et al. (2006) synthesis will be described to demonstrate use of the promise table; for example, if the focus of interest were on public health interventions to reduce chronic disease risk factors in children related to obesity.

Two examples for using the promise table

The features table from the web-site document (Tables 8.1–8.6 of Flynn et al. 2005) contains the content that is needed to identify ‘certainty of effectiveness’ and ‘potential for population impact’. The 147 programmes that were reviewed are all included in the features table arranged by the setting in which they took place and provide important contextual information on the population characteristics as well as description of the interventions and study or evaluation outcomes (for examples, see Table 5).

Table 5. Programme details for examples of studies (from Flynn et al. 2005)

MD, doctor; RD, registered dietitian; RN, registered nurse; ↑, increase; ↓, decrease; ↔, no change; ↑↓, P<0·05.

Example 1

The Special Turku Coronary Risk Factor Intervention Programme (Niinikoski et al. 1996; Rasanen et al. 2003; Salo et al. 1999) involved the participation of >1000 infants, children and their parents. The intervention programme consisted of a primary care-based multidisciplinary team providing individualized dietary counselling during infancy and lifestyle and dietary counselling during childhood until age 7 years as part of routine ‘well child’ care. Intervention children were seen ten times in the first 36 months of life and twice per year thereafter. The control group received traditional care. At 12, 24 and 36 months of age fat intake (24 h dietary recall) and serum cholesterol measures were found to be significantly lower and remained lower at the 5-year measurements (P<0·05). The programme was ranked ‘mid’ in terms of scientific rigour as it was a cluster randomized trial, but the data available was based only on 70% of the cohort for some analyses and fewer in others. The outcomes at 5 years were impressive in that those in the intervention group compared with the control group had significantly lower serum cholesterol. Thus, ‘certainty of effectiveness’ could be rated medium, indicating a range of less to very promising. Programme logic was upheld; at 7 years parent nutrition knowledge and child intake of mono- and polyunsaturated fat in the intervention group was significantly better. The uptake of the programme was good with data available at 5 and 7 years and reach was broad with >1000 children but an unknown number of clinics participating. Thus, there is, at minimum, a moderate ‘potential for population impact, also ranging from very to less promising. When combined, the rankings would identify this programme as promising. However, those countries with a government-funded healthcare system and standard Well Child clinics providing multidisciplinary services might rate this programme as being most promising because of similarities in health systems.

Example 2

A ‘Know Your Body’ universal programme that had been adapted for use in Crete as a primary-school multi-grade programme was provided in over forty schools with 1046 children involved in the evaluation (Manios et al. 1999). There were twice weekly 45 min physical education sessions along with 4–6 h class time on fitness and 13–17 h education on healthy eating. The programme was evaluated after 3 years, comparing the intervention and control groups. Measures of serum total cholesterol and LDL were found to be significantly lower in the intervention group (P<0·001). There were also significant improvements on some fitness measures (EUROFIT; Council for Europe Committee for the Development of Sport, 1988), increased nutrition knowledge, decreased BMI and suprailiac skinfold thickness (P<0·05). There were no differences between intervention and control groups in daily nutrient intake (3 d weighed-food record), triceps skinfolds, serum triacylglycerols or parent knowledge. Based on a ‘mid’ ranking for scientific rigour and mixed but primarily positive outcomes this programme could be rated high on ‘certainty of effectiveness’. There were many schools, so reach was very good with good sample size for measurements. There was high parental participation in seminars. Based on these factors ‘potential for population impact’ could be ranked high, with the programme rated very promising.

Conclusion

The art of preventing obesity involves a blend of review, risk and political negotiation (Lobstein, 2006). To do nothing while waiting for evidence is unacceptable. For clinicians, programme planners and policy decision makers it can be difficult judging which interventions to adopt.

There are many challenges to identifying best, or better, practices to provide evidence-based practice for population health. There is a growing number of examples in which the broader content is evaluated beyond the traditional approach to critical appraisal that focuses entirely on scientific merit. In fact, an understanding of the context is crucial to making decisions about adoption of evidence. The Flynn et al. (2006) synthesis provides information from both traditional and broader contexts, which allows a more comprehensive examination of programmes to enable identification of promising programmes and interventions. This approach is particularly important in the area of childhood obesity prevention and intervention programming as the epidemic continues to grow at an ever-increasing pace.