To conclude, this chapter summarises the contents of this book by presenting a model and six key principles for designing and conducting experiments.

A model of the experimental process

The model, presented in Figure 8.1, shows the main stages of the experimental process and the important considerations that need to be addressed at each stage.

Six key principles for conducting experiments

This book presents specific advice to guide the researcher through the experimental process, and, subsequently, six key general principles emerge. These are listed as follows:

The first step in running an experiment is defining what you want to discover and how you will do so. This chapter presents an approach to experiments that begins by first defining a research question, and then basing the definition of the conditions, experimental objects, and tasks on that question. These elements will ultimately define the form of the experiment.

Several key concepts used throughout the book are introduced and defined in this chapter:

• The research question : a clear question that succinctly states the aim of the research;

• Conditions: the ideas of interest – these will be compared against each other;

• The independent variable: the set of conditions to be used in the experiment – there will always be more than one condition;

• The population : all the people who might use the idea; the sample: the set of people who will take part in the experiment;

• Generalisability : the extent to which experimental results can apply to situations not explicitly included in the experiment itself;

• Experimental objects : the way in which the ideas are presented to the participants – experimental objects embody the conditions so that they can be perceived;

• Experimental stimulus : the combination of an experimental object and a condition;

• Experimental tasks : what the participants will actually do with the experimental objects;

• Experimental trial : the combination of a condition, an experimental object, and a task.

Designing an experiment is more than creating stimuli and tasks and deciding on the participant experience. Before conducting the experiment, the exact form of data to be collected needs to be decided, and importantly, it needs to be confirmed as sufficient for answering the research question.

This chapter focusses primarily on data collection. It describes the different types of data that can be collected for different purposes and the means of collecting it.

We make the traditional distinction between quantitative data (represented by numbers; e.g., the number of errors, a preference ranking) and qualitative data (not represented by numbers; e.g., a verbal description of problems encountered in performing the task, a video showing interaction with an interface).

In practise, there are two distinct decisions to be made about data:

• What data to collect (a decision made in advance of the experiment), and

• How to analyse the data (a decision made after the experiment has been run).

These two decisions are inextricably linked because the potential means of analysis will influence the decision on what data to collect. Any discussion about data collection therefore necessarily entails discussion on how it will be analysed.

Some years ago, I presented a retrospective of the graph drawing (and related) experiments I had conducted since 1995 to an audience of information visualisation researchers, describing the process I went through in defining a new experimental research area and learning to run human–computer interaction experiments. This was an honest and reflective seminar in which I highlighted the mistakes I had made, the good and bad decisions, and how my knowledge of experimental design had increased and improved with every experiment. At the end of my presentation, a member of the audience asked, “So, Helen, what is the ‘Black Art’? What is it that you have learned about running experiments that we should all know?”

This started me thinking about how much expertise is embodied in experience and seldom communicated apart from in a master/apprentice model. PhD supervisors can advise students on how to formulate and conduct experiments, psychology and HCI research texts can be read, and other experiments in the research literature can be copied, but the actual step-by-step process of designing and running an experiment is rarely written down and communicated widely. Although I believe that one can never understand the process of conducting experiments without experiencing the process oneself, I also believe that experiences can (and should) be shared and that advice resulting from others’ experiences can always be useful.

So, now you have your results, and you want to tell everyone about them. This chapter discusses the way in which you report your research, typically in a research article for an academic conference or journal, or in a dissertation for assessment. In all cases, you need to keep in mind that someone else will be reading what you write, and that this person has not been party to your decision-making process. It is easy to leave information out because it appears obvious.

Reviewers’ concerns

It is the job of reviewers or assessors to make a judgement on the worth of your research, and it is your job to make sure that you have presented it sufficiently well that they can do so. This is true of all research; however, writing experimental papers brings with it its own particular issues:

• An experiment focuses on addressing specific research questions: reviewers may not believe that these questions are interesting or important.

• An experimental research question could be addressed in many different ways: everyone will have their own idea as to how best to address it.

• Different statistical methods are favoured by different people: beware the reviewers who are well versed in statistics when you have not analysed your data using their favoured method!

• No experiment can ever be perfect: reviewers can easily find faults on which to base a negative judgement if they want.

• Many reviewers have never actually designed and conducted an experiment themselves: they do not always appreciate the amount of work required for running experiments, the difficulty in making appropriate design decisions, or the constraints that apply to experimental design.

So far, we have assumed that everything will go smoothly; however, in practise, this is rarely the case. All will not go as planned, especially if this is your first experiment. Thus, this chapter discusses some of the things that can go wrong, and gives suggestions as to how to prevent them occurring, or how to deal with them if they do. Problems can be prepared for, and in many cases, following the advice given in this and previous chapters will put you in a good position to address them (be forewarned!). Pitfalls are those events for which you cannot prepare but that must be dealt with in order to rescue the situation.

Problems

Pilot tests show that the experimental design is fundamentally flawed. You may have put a great deal of work into preparing the experimental objects, tutorials, etc., only to discover that the task given to the participants is simply too difficult and takes too long, that the participants cannot understand what is expected of them, or that the tasks are actually inappropriate for the different conditions. The concept of pre-pilots (and even pre-pre-pilots!) is useful here. Piloting is an iterative process. Although you must pilot at least once with the full experimental method before running it, it is useful to run smaller, partial pilots on some aspects of the experiment before putting it all together. For example, get feedback on the tutorial from a colleague to find out whether it is clear, or ask someone to perform the tasks on the experimental objects on paper to determine whether they are appropriate. By the time you get to running the final pilots, many of the potential problems will have already been addressed.

Pleasures and Politics of Simulation

The Sims has generally been labelled a virtual doll's house and a real-life domestic simulator where the player creates and controls the lives of little AI-powered people. These kinds of epithets can be investigated through the concept of simulation, which in this context is regarded as an operating principle that connects the ingame occurrences to the ways players make sense of the real (that is, the actual, social) world. In semiotic terms, simulating real life refers to the subject matter of the simulation, ‘the signified’, whereas reference to, for example, the doll's house gives an idea of the particularities of the signification process and the representational system in question (‘the signifier’). In the context of The Sims, the association between the game (or, the game-as-process) and its referent in reality can fruitfully be investigated through the notion of simulation precisely as the explicitly stated source system of the game is the naturalised, albeit somewhat idealistic construction of the ideologically charged American suburbia that I investigated in the previous chapter.

The Sims can be categorised as an experiential (rather than algorithmic) simulation and a free-form game, deriving from the traditions of the adventure and RPG genres. Its RPG-qualities are most visible in its emphasis on the creation and development of characters and the important roles the characters are assumed to encompass as the player's representational devices. Another characteristic of The Sims that follows from its adventure-RPG-trait is its contextuality, visible in the constant transformation and expansion of the game's context (see Myers 2003, 35). The add-ons of The Sims have brought, for example, ‘The Shopping Centre’, ‘The Old Town’, and ‘The Studio Town’ in new gameplay spaces, as well as a constant flow of new gameplay elements. Although many game scholars rely on the idea of the incompatibility of game elements and the elements of simulation, in the case of The Sims I see these elements providing a multidimensional platform, a predisposition for toying and playing with the notion of reality and real life. As I am interested in ideology, I am also keen on finding out how these ideological propositions are inscribed in the game, and on the other hand, how they are reworked and redirected.

Categorising modding practices

What makes computer games specifically interesting pieces of digital code besides the actual gameplay is their inherent potentiality for malleability and alteration. At present, game software is structured and interconnected according to a specific logic: there is the game engine that controls everything that is to be experienced by the player by ordering the hardware to generate the appropriate images, sounds and movements, and mediating the content through designated interfaces such as the screen and speakers. The game engine thus renders the in-game world, its characters and objects, lets the story unfold according to the storyboard and the player's input, allows music and sound to be heard at the right moment - and it does all of this out of game data that is stored in files and libraries. This data, consisting of components such as 3D meshes, textures (surfaces) for objects and characters, scripts and rulesets, functions like raw material out of which the engine ‘spins its yarn’ for the player to experience in real time as the game is played. The fact that the contents of these data libraries can be altered makes games moddable (Knorr 2007, 3-4). Practically speaking, game modding takes place on the level of altering and tinkering with game data files as the access to the game engine is not normally granted to players.

Most of the theoretical notions on modding are structured around the level of adding game data. One of the most basic categorisations is the one provided by David B. Nieborg and Shenja van der Graaf (2008), as they make a differentiation in their study of total conversion (TC) mods between the fundamental elements of the games and mods as extensions of them, in other words, to ‘proprietary engines’ that are under corporate control. In their study, this is termed as the techno-economic dimension of modding and put under ideological scrutiny, as the TC modding is considered to entail free labour and unpaid development work done by players, granting benefits to the industry (see also Postigo 2003). Other, more general and perhaps more neutral terminology is used by digital media theorists such as Espen Aarseth (1997) and Joost Raessens (2005), who term modding as ‘addition’ or ‘construction’ of new game elements.

Playing with the Sims

As I have previously argued, interpretation and configuration, acting as the basis for the processes of meaning-making in gameplay and manifesting themselves in the physical activities of play, are closely tied to the representational qualities of The Sims. It may be true, however, that in the context of digital games, The Sims is an exception in this regard. James Newman (2002, 2), for instance, criticises the strong cultural studies tradition of theory building through concepts like (graphical) representation used to explain the pleasures of gameplay. He argues that for instance the appearance of game characters is not important to the primary player during gameplay, but the way the gameworld feels to the player is absolutely crucial – and in this process the game character acts merely as a vessel whose functionality is judged on the basis of the game's playability. The kinaesthetic experience of play is thus tied to the controls and mechanics of a game, and the character, the player's avatar, represents the possibility or a ‘capacity’ to undergo a game in a specific way (e.g. Jenkins 2005). The degree to which the player considers herself to ‘be’ her avatar, or the game character she plays, is not contingent upon representation, but a ‘character’ – be it a blinking white light or a MMORPG warrior tuned-up to the max level – is rather a set of characteristics, a tool with which to operate within the gameworld (Newman 2002, 7).

In principle, it is tempting to agree with Newman on the argument that the representation of game characters in the process of gameplay, for the experience of the player, is not a key issue in the study of games. However, it cannot be concluded that representation – associated with the interpretive dimension of gameplay and modding – does not matter in the context of games at all. In extra-game contexts, such as in the creation of gamics and machinima through the redirection of the game engine as well as game advertisements and dedicated fan websites, the representational aspect of game characters is often extremely important. It could be said that game characters, particularly the ones like Lara Croft and plumber Mario, outside the scope of game, are ‘representationalised’ in so far that they are assumed to incorporate an identity of their own.

Game and its modification

The Sims is a curious computer game. Originally published in 2000 by Electronic Arts, it has since become a global crossover hit and a cultural phenomenon. It has been termed a strategic life-simulation, a lifestyle simulator, an IKEA game and a virtual doll's house where the player's task is to provide little AI-controlled humanoid characters, called the ‘Sims’, with a house, household items and furnishings, and follow the daily activities of these pixellated people. The Sims also looks and feels like an exceptional game: its actual gameplay consists of orchestrating the Sims’ everyday lives by directing their paths, choices and relationships. It is especially untypical in the context of mainstream games and game genres. Nevertheless, due to the accessibility of its thematic and gameplay, The Sims has attracted millions of players worldwide, among them a number of people who, prior to its introduction, were not involved in playing digital games at all. It has been an unexpected success story – a conclusion that can also be deciphered from the press release excerpt above. Interestingly enough, even after a decade since its introduction, it is still not rivalled by any other game titles of the same design or composition. In other words, its success has not yet been duplicated (Croal 2008).

One of the main reasons why The Sims resists being categorised as a regular game is its design: it features subject matter, ways of interaction, characters and viewpoints that cannot be straightforwardly identified to replicate the militarised masculinity often regarded to epitomise the field of digital games. By militarised masculinity Kline et al. (2003, 247-248) refer to a self-amplifying cultural channel or ‘groove’ which interactive gaming has cut out for itself mainly by concentrating on game design practices that produce strongly gender-coded scenarios of war, conquest and combat.