TO CONTROL OR NOT TO CONTROL: HOW DO WE LEARN MORE ABOUT HOW AGRONOMIC INNOVATIONS PERFORM ON FARMS?

Summary

Our paper ‘Loading the dice in favour of the farmer: reducing the risk of adopting agronomic innovations’ revealed mean increases but also large variation in the impact of four agroforestry practises on maize yield, as experienced by farmers in Malawi. This prompted a response from Sileshi and Akinnifesi that was critical of the data and methods used. Their main concern was that farmers did not necessarily manage crops identically in plots with and those without trees, so the yield differences that we measured may be partly caused by these differences in crop management. We argue here that it is valid and useful to look at the actual effect on crop yield of farmers having trees intercropped with maize, rather than controlling for how the crop is managed, because this is what happens in the real world. Farmers respond to having trees in their field by treating their crop differently, so this is part of the system response to having trees in fields. Attempts to eliminate this will result in measuring an artefact rather than the real impact of trees on crop yield. By doing this, we revealed important variation in the impact of trees on crop yield amongst farmers, and we argue that it is important to explore, assess and communicate to farmers and development actors the extent and implications of this variation. Understanding the contextual factors that determine who is likely to benefit most from an innovation and for whom it is less suitable can then be incorporated in scaling up, so that targeting of innovations and the appropriateness of messages given to farmers are continuously refined.

INTRODUCTION

There is a long tradition of agronomic research, including recent work on agroforestry that progresses on the basis of demonstrating mean differences in yield between a proposed innovation, such as the introduction and management of trees in crop fields, and a control, such as crop fields without trees (Coe et al., 2014). Our recent research article ‘Loading the dice in favour of the farmer: reducing the risk of adopting agronomic innovations’ (Coe et al., 2019) broke with this tradition in going beyond looking at mean yield differences in controlled conditions, to explore the variability about mean differences in crop yield when trees were incorporated in crop fields across Malawi, as experienced by farmers who had adopted the innovation. This shows a wide range in the extent to which four agroforestry options affected crop yield amongst farms. We argue that seeking to understand the contextual factors that lead to these yield differences can improve targeting of agroforestry options to farmer circumstances and the messages about agroforestry transmitted to farmers, and thereby reduce the risk that farmers take when adopting these innovations. Research papers are designed to be read, discussed and debated, and we are pleased that Sileshi and Akinnifesi (2019) have started a public debate around this one through their response to it. We are also pleased that they recognise that the work is unconventional. Research progresses by pursuing new ideas using new methods so that convention is continually challenged. This represents innovation, where the new ideas and methods are sufficiently rigorous and useful to become accepted and inform future research. Sileshi and Akinnifesi (2019) appear to challenge that this is the case for the research that we presented (Coe et al., 2019) principally by questioning (i) the data and how they were collected; (ii) the appropriateness of the analysis of the data in determining effects of trees on crop yield; (iii) the definition of the risk of adopting agronomic innovations that we used and (iv) the relevance of focussing on the impact of trees on crop yield.

As we show in the detailed responses to each of their points below, rather than providing critical insight on the research that we presented, Sileshi and Akinnifesi (2019), seek to reframe the research questions in terms of (i) isolating biophysical effects of trees on crop yield, rather than the actual effects on crop yield when farmers incorporate trees in fields and (ii) the impact of agroforestry on the riskiness of farming rather than the specific risk, in terms of effects on crop yield, of an individual farmer adopting a new practise based on information about mean effects. It is not surprising that analyses and interpretation of data presented for one purpose are not relevant for a different purpose. While Sileshi and Akinnifesi (2019) pose interesting questions that, as they show, have already been addressed in previous research, this is a restatement of the conventional research that has been done, not a critique of the novel research that we presented. More problematically, they imply that seeking to explore differences in performance of agroforestry options on crop yield amongst farmers in some way detracts from the case for promoting agroforestry options to address food security in Africa. On the contrary, we argue that understanding more about the contexts within which different agroforestry options perform well, and those for which they are less suitable, is important in better targeting different agroforestry practises to farmer circumstances and providing more relevant information to farmers. If this type of research is pursued with rigour we anticipate it will lead to greater uptake and benefit to farmers. We conclude by showing how the new research that we presented builds upon and complements, rather than contradicts, the main thrust of Sileshi and Akinnifesi's (2019) response, indicating more convergence than is apparent from the adversarial tone of their contribution.

THE IMPORTANCE OF AGROFORESTRY

We agree with Sileshi and Akinnifesi (2019) about the emerging importance of agroforestry for addressing food security in Africa, and how growing recognition of this has been built on evidence, much of which they have generated. It is because of this that we are concerned about how research results are presented to policy makers and farmers. If only mean yield increases are communicated, without information about variation around these means, then an overly simplistic impression that all farmers can expect to achieve the mean increase, or something close to this, is implied. In fact, individual farmers will experience very different results depending on a range of contextual factors that determine performance of any agroforestry practise in their specific circumstances. The aim of our paper was to show this with an example and to show how more nuanced understanding of effects of trees on crop yield can be generated by understanding how contextual factors affect performance, giving rise to more useful messages. We firmly believe that this is necessary for agroforestry to reach its potential, and it should not be seen as an argument against agroforestry.

ORIGINS AND NATURE OF THE DATA

The points that Sileshi and Akinnifesi raise about the origins and nature of the data are curious. Sileshi was the nodal co-ordinator for Southern Africa at ICRAF, and line manager of Joyce Njolma, when the data were collected but left ICRAF before the analyses and interpretation of the two year dataset were done and the paper was written. The data were explicitly collected to evaluate performance of agroforestry options promoted by the Agroforestry for Food Security Program (AFSP) that Sileshi refers to, but through a supplementary activity and funding for impact evaluation. Coe and Sinclair discussed the design with Sileshi, and Njoloma was responsible for the bulk of field work. Contrary to the assertion by Sileshi, the data collection was never envisaged as a baseline and could not have been since it was done some years after the start of the interventions. As is clearly stated in our paper, the comparison presented is of yield of maize in plots with and without trees, on farms where the agroforestry interventions had been adopted for a period of at least four years and does not involve a temporal comparison with a baseline. The nature and limitations of such a comparison are fully discussed in the paper. However, regardless of the original intent when collecting the data, what is important is whether the analysis and interpretation that we presented are valid and insightful, which we address in the detailed response to Sileshi and Akinnifesi's points below.

HOW MUCH CONTROL IS DESIRABLE?

Sileshi and Akinnifesi point out that what farmers did on their plots was beyond the control of researchers. This is precisely what makes it possible for us to detect the variation in response to using trees for soil fertility that happens in the real world, outside the controlled conditions of researcher's experiments whether they be on station or on farm. We do not agree with their assertion that this form of analysis is not correct. It is certainly different from what is possible with more controlled experimental data but not less valid. We clearly stated in our paper what analysis had been done and discussed its merits and limitations. What is consistent is the use of trees for enhancing soil fertility in maize plots, a recommendation made in the promotion of agroforestry in Malawi. The myriad ways in which that practice and its context can vary is the main difference between what researchers can see in conventional trials and what farmers experience. When a study is not a randomised controlled experiment, as in this case, it is perhaps misleading to discuss ‘treatments’ and ‘controls’. However, we do take the farmer's plot without trees as representing what that farmer experiences without trees, compared with what they experience on their plot with trees. This difference is a measure of the difference the trees make as experienced by that farmer. There are many factors that influence that, and the paper is about quantifying, interpreting and explaining that variation. Of course, the validity of this comparison as a measure of the effect of incorporating trees on maize yield depends on the extent to which the plots were comparable before the intervention – that is, the extent to which crop performance on the no-tree plot is a good estimate of the counterfactual, which was not and cannot be observed. This is a common problem in many observational studies.

This focus on what farmers experience is indeed distinct from the focus typical in researcher's reductionist approaches. The extreme is the classical randomised trial in which an attempt is made to control at a constant level every factor except the one being examined as a treatment. This is, of course, a very sound design for some research purposes but does not necessarily generate good predictions of responses under the variable conditions and management that farmers use. An alternative design, that would have given data with a more obvious direct interpretation as a ‘tree effect’, would have been to encourage each farmer who adopted trees on one plot to keep a comparable plot, managed in the same way, without trees. This could have generated information for both farmers and researchers on the impact of trees. The variation in contexts (such as in soils, on-farm niche and crop management) across farms would have revealed context × tree effect interactions. This design was suggested at the start of the second phase of the AFSP project, but the implementers chose not to use it. That design, and the options for varying levels of control, is the subject of a paper in this issue (Vanlauwe et al., 2019), and it also represents a distinct change from traditional agronomic research practise.

However, the idea that the two plots should be managed in the same way is itself problematic, for two reasons. First, the presence of trees may well change how farmers behave, so that they manage crops differently in plots with trees than in those without. For example, planting and harvesting dates might need modifying as trees influence soil moisture regimes, weeding strategy may have to change as trees influence weed populations, and so on. Secondly, the trees and their impact on cropping will lead to different choices by farmers regarding allocation of resources. When considered part of a system, rather than in isolation, the ‘tree effect’ is indeed a response to many different factors beyond simple biophysical interactions with everything else held constant. This gross effect of trees on maize yield is what has been assessed in our paper, though other impacts of the trees have not been assessed.

Given all that, we do acknowledge that deciding ‘what to keep constant’ is difficult. In this study, the paired plots measured were said by farmers to be ‘comparable’ but exactly what that means was not defined or investigated. It would have been helpful to ask farmers their opinion on reasons for differences in maize performance on the pair of plots (Tiwari et al., 2004). Their observations might have provided insights that could then be investigated with data collected specifically to test their general validity.

SYNERGISTIC OR ANTAGONISTIC EFFECTS OF NITROGEN FIXING TREES AND FERTILISER USE BY FARMERS?

Sileshi and Akinnifesi identify fertiliser use as an important factor that influences maize yield and varies amongst plots, both within and between farms. We did take that into account in the analyses to the extent allowed by the data. As explained in the paper, the only credible data available on fertiliser use was whether or not it had been used on the plot that season. Rates of use reported by farmers could not be relied on, as many were unrealistically high. This is a common problem when trying to assess resource use from farmer recall. The results are summarised in Table 3 of the paper which, as suggested to be desirable by Sileshi and Akinnifesi, disaggregates farms that have different fertiliser regimes on different plots. Our point regarding variation is reinforced if we take the results in Table 3 and look more deeply. Of the nine farmers with gliricidia that had no fertiliser on either the sole or tree plot, six differences were positive and three negative. The sample size is too small to infer much but the fact that there is important variation is clear. Of the 51 farms that used fertiliser on both plots, 35 had positive and 16 negative differences between yield on tree and sole plots.

The evidence for a negative interaction between gliricidia and fertiliser use on crop yield is also presented in Table 3. The average difference between the + and – tree plots is higher when no fertiliser is applied to either (1.31 t ha⁻¹) than when it is applied to both (0.64 t ha⁻¹). This is a negative interaction. The difference between the two differences, −0.67, is the same as the difference in mean effect of fertiliser with and without trees. We are aware of and refer to alternative conclusions from others about synergistic effects, and that is why we were keen to point out that in this context of farmers experience the evidence points to the opposite. The inference is necessarily based on means across farms as no farms had all four combinations of +/− trees and +/− fertiliser.

DIFFERENT DIMENSIONS OF RISK

Sileshi and Akinnifesi (2019) correctly point out that farmers face numerous risks, these being ecological, social or economic in origin, but are presumptuous in suggesting that their conception of risk is correct whereas that presented in our paper is not. In our paper, we were considering the specific risk of an individual farmer in Malawi following a simple message such as ‘intercropping with gliricidia will increase maize yield’. The data we present show that sometimes it will and sometimes it will not increase yields, variation that represents risk to farmers acting on the recommendation. That risk can be reduced if we understand more about what determines variable performance amongst farmers, and hence for which farmers this message is most likely to be appropriate. Note that we do not equate this risk with adoption potential, nor are we looking at the effect of trees on the inherent riskiness of farming that Sileshi and Akinnifesi (2019) highlight in their response. This is a different issue and we concur with them that trees often mitigate risks associated with farming through diversification. As we point out in the paper, and Sileshi and Akinnifesi echo in their response, there are many benefits (and costs) to adoption of agroforestry beyond any impact on maize yield, and it was not the intention of our paper to assess those, nor is there information on them in the data that were available to us. This does not detract from the importance of understanding impacts of trees on staple crop yield, as is clearly explained in the paper. This follows precedent, including influential meta-analyses presented elsewhere by Sileshi and Akinnifesi themselves (see for example, Sileshi et al., 2008). The analyses that we present are trying to identify the factors that predict the variation in crop yields experienced by different farmers adopting different agroforestry practises. As we show in our paper, not many factors could be identified in the available data, symptomatic of a general tendency for agronomic data to be collected without sufficient contextual information to elucidate the reasons for crop yields to increase in some circumstances and not others (Sileshi et al., 2008). A richer data set that could have been generated by hypothesising those factors and measuring them might have revealed more. The central point we are making is that this variation (that represents risk to farmers adopting an innovation based on mean yield increases) should be assessed and included as part of the message to extension workers and farmers. In our paper, we suggest how it can be reduced by acknowledging what uncertainty exists about how contextual factors condition performance and then systematically evaluating this as scaling out progresses, through the use of planned comparisons.

WHAT COMPARISONS ARE VALID?

Sileshi and Akinnifesi (2019) criticise our comparison of present results with previous published data on effects of trees on crop yield, on the basis that the way in which yield differences were calculated were different, but that, as explained above, is the whole basis of our comparison and of our discussion of the observed differences. Similarly, contradiction of earlier results on risk is not in itself a problem, if it can be explained. Kamanga et al. (2010) and Sirrine et al. (2010) were considering different risks estimated in a different way than those we present, so it is not surprising that their results are different.

Finally, Sileshi and Akinnifesi (2019) state that we ‘. . . claim that agroforestry promotion was biased towards more fertile than average fields and more industrious than average farmers’. We do not. We stated that ‘This difference [in sole maize yields of farmers in the sample with national averages] is commensurate with a bias towards more fertile than average fields and more industrious than average farmers having taken part in the agroforestry promotion’. This is simply a possible explanation for why the yields in this study are higher than those often quoted for Malawi. It is not a comment on the process of promoting agroforestry. However, it would be unusual if these agroforestry technologies were neutral with regard to the type of farm and farmer interested in trying them. It is not usual for farmers participating in this type of project and trying new technologies to represent a cross-section of farmers within the communities concerned.

CONCLUSION

Papers in this special issue highlight the need to recognise and react to complexity and interactions when developing and accessing agronomic interventions, including agroforestry. While recognising variability in performance of agronomic options across contexts is not new, as Sileshi and Akinnifesi have themselves revealed (Sileshi et al., 2010), reacting to it needs a paradigm shift in how both research and development are done (Sinclair, 2017). This is overlain with the problem that agricultural research funded by development donors needs to impress them. We can expect research that challenges norms to be contentious (Sumberg et al., 2012) and it is clear that a step change in how research contributes to development is needed if the new Sustainable Development Goals to end world poverty and hunger (United Nations, 2015) are to be realised.