7 - Problems and pitfalls

Summary

So far, we have assumed that everything will go smoothly; however, in practise, this is rarely the case. All will not go as planned, especially if this is your first experiment. Thus, this chapter discusses some of the things that can go wrong, and gives suggestions as to how to prevent them occurring, or how to deal with them if they do. Problems can be prepared for, and in many cases, following the advice given in this and previous chapters will put you in a good position to address them (be forewarned!). Pitfalls are those events for which you cannot prepare but that must be dealt with in order to rescue the situation.

Problems

Pilot tests show that the experimental design is fundamentally flawed. You may have put a great deal of work into preparing the experimental objects, tutorials, etc., only to discover that the task given to the participants is simply too difficult and takes too long, that the participants cannot understand what is expected of them, or that the tasks are actually inappropriate for the different conditions. The concept of pre-pilots (and even pre-pre-pilots!) is useful here. Piloting is an iterative process. Although you must pilot at least once with the full experimental method before running it, it is useful to run smaller, partial pilots on some aspects of the experiment before putting it all together. For example, get feedback on the tutorial from a colleague to find out whether it is clear, or ask someone to perform the tasks on the experimental objects on paper to determine whether they are appropriate. By the time you get to running the final pilots, many of the potential problems will have already been addressed.