Evaluation was not a thing when the first author was a graduate student in the late 1970s. There was an Artificial Intelligence (AI) boom then, but that boom was quickly followed by a bust and a long AI Winter. Charles Wayne restarted funding in the mid-1980s by emphasizing evaluation. No other sort of program could have been funded at the time, at least in America. His program was so successful that these days, shared tasks and leaderboards have become common place in speech and language (and Vision and Machine Learning). It is hard to remember that evaluation was a tough sell 25 years ago. That said, we may be a bit too satisfied with current state of the art. This paper will survey considerations from other fields such as reliability and validity from psychology and generalization from systems. There has been a trend for publications to report better and better numbers, but what do these numbers mean? Sometimes the numbers are too good to be true, and sometimes the truth is better than the numbers. It is one thing for an evaluation to fail to find a difference between man and machine, and quite another thing to pass the Turing Test. As Feynman said, “the first principle is that you must not fool yourself–and you are the easiest person to fool.”

Research and extension personnel are beginning to look for new strategies to involve more farmers and the non-farm public in their programs. Two approaches we have used are wholefarm case studies (WFCS) and focus groups. WFCS in Oregon and Washington led to several research and educational programming ideas that are currently being pursued in both states. A focus group to study water quality, nitrate leaching and farming practices in Skagit County, Washington is one outcome of the WFCS process. It is made up of 16 people, including farmers, university personnel, members of environmental groups, and government representatives.

We review these two complementary participatory strategies for systems-oriented sustainable agriculture research and education programs. Both have been very useful for building problem-solving partnerships between the land-grant universities and agricultural and environmental constituents in our area. Noteworthy outcomes include: complementary applied on-farm and experiment station research; farming systems analysis; public education; new linkages with environmental and agricultural interest groups; additional grant funds to address identified priority issues; and interdisciplinary teams that cut across the biological and social sciences and include diverse citizen representation.

A 6-yr interdisciplinary field project using large plots was initiated in 1985 to assist Pacific Northwest growers in developing an integrated pest management system to control weeds, reduce erosion and grow crops profitably. Run by a network of 10 to 14 scientists from eight disciplines supported by five agencies, the project showed that wellmanaged conservation tillage systems had a higher average profit and lower economic risk than traditional conventionally tilled systems. Information was disseminated successfully to user groups by field days, plot tours and an educational video. Organizational strategies included an interdisciplinary design with three levels of objectives. Cooperating scientists were invited to participate in writing the proposal and followed strict field protocols and attended mandatory organizational meetings. This long-term study used large machinery, went beyond the transitional stage and allowed treatments to mature. Limitations included the restricted number of experimental variables, the location of the plots, the large labor force required, and publication decisions.

For more than two decades, emergency medical services (EMS) systems have proliferated primarily based upon governmental impetus and funding at the federal, state, and local levels. Although many of the foundations of patient care rendered in these systems have been based upon intuitive logic, the understanding of the impact on patient outcome is poor, at best. The reasons for the current status are varied, but five issues are preeminent:

1) The authority for the development of these medical systems has been based primarily in political and bureaucratic institutions which have little or no medical expertise;

2) Little attention has been paid to system evaluation, particularly in the area of cost-effectiveness;

3) Few academic medical institutions have become involved in EMS research;

4) Traditional approaches to medical research primarily are disease-specific and are not multidisciplinary. Thus these are not useful for evaluating and understanding the highly complex and uncontrolled environmental interactions that typify EMS systems; and

5) The process of efficiently and reliably collecting accurate data in the prehospital setting is extremely difficult.

The research agenda for agriculture remains an important concern for anyone interested in the future of sustainable agriculture. Differences in opinion remain on both prioritization of topics and on how and where the necessary research should be done. The inferences are that the research strategies of the 1960's and 70's were not appropriate. If we truly “learn from each other,” we must consider what recent reports, models, and recommendations from written sources and individuals seem to be saying. Examination of these sources suggests a number of characteristics for the research that is needed: 1) Research which addresses problems regional in scope and which are often site specific; 2) research prioritization must involve a broader cross-section of opinions than ever before and include farmers, private non-profit organizations and scientists; 3) research must be more multidisciplinary and include farmers, extension faculty, and staff more directly; 4) systems research requires more emphasis and adoption f or progress in many areas; 5) basic principles and theory for many agricultural disciplines will need examination and eventual modification; and 6) on-farm research is an essential part of the total research effort. The final obligation for all of us is a vigilance to assure that the research done is always of the highest quality.