Design at hackathons: new opportunities for design research

Abstract

Hackathons are short-term events at which participants work in small groups to ideate, develop and present a solution to a problem. Despite their popularity, and significant relevance to design research, they have only recently come into research focus. This study presents a review of the existing literature on the characteristics of designing at hackathons. Hackathon participants are found to follow typical divergence–convergence patterns in their design process throughout the hackathon. Unique features include the initial effort to form teams and the significant emphasis on preparing and delivering a solution demo at the final pitch. Therefore, hackathons present themselves as a unique setting in which design is conducted and learned, and by extension, can be studied. Overall, the review provides a foundation to inform future research on design at hackathons. Methodological limitations of current studies on hackathons are discussed and the feasibility of more systematic studies of design in these types of settings is assessed. Further, we explore how the unique nature of the hackathon format and the diverse profiles of hackathon participants with regards to subject matter knowledge, design expertise and prior hackathon experience may affect design cognition and behaviour at each stage of the design process in distinctive ways.

1. Introduction

Hackathons are rapidly growing in popularity. At these events, participants work in small groups to ideate, develop and present a solution to a problem. Thus, while not always explicitly promoted as such, hackathons provide participants with exposure to, and experience in, design (Artiles & Wallace 2013; Komssi et al. 2015). Hackathons present themselves as unique and authentic settings at which design activity can be studied.

Hackathons have only recently come into research focus (Trainer et al. 2016). Studies have investigated the individual experiences of participants (Olesen, Hansen & Halskov 2018), the effect of expertise diversity in hackathon teams (Legardeur et al. 2020) and the use of hackathons to solve problems (Artiles & LeVine 2015; Lewis et al. 2015; Uffreduzzi 2017; Kos 2019), teach design (Fowler 2016; Nandi & Mandernach 2016; Page et al. 2016; Gama et al. 2019) and facilitate a design process (Artiles & Wallace 2013).

We view hackathons as a setting that simulates aspects of design activity as it occurs in real-life practice; yet, the extent and nature of design in this unique environment is not well understood. Research on design activity as it occurs at hackathons can improve our understanding of rapid design decision-making and provide evidence-based prescriptions for more effective designing of the hackathon events themselves. In this paper, we aim to explore the extent to which design processes have been studied in hackathons and identify future directions and opportunities for design and design education research in this setting.

1.1. What is a hackathon?

There is no finite, agreed-upon set of characteristics that classify an event as a hackathon (Komssi et al. 2015), making it difficult to generate a definition inclusive of all variations of hackathon-like events. In general, a hackathon is a problem-focussed, short-term event where small groups work to develop a final product (Briscoe & Mulligan 2014; Komssi et al. 2015; Gama 2017; Izvalov, Nedilko & Nedilko 2017; Kollwitz & Dinter 2019). Hackathons tend to be of interest to computer programmers, designers and engineers; it is typical for the theme of the event to be tech-centric. They have been used in a wide variety of contexts and for different purposes, including education, networking and to accelerate innovation (Flores et al. 2018). Hackathons could be run as external events (open to anyone), or internally by organisations (Briscoe & Mulligan 2014). The duration of hackathons also varies greatly. Hackathons typically run continuously over 24–36 hours, but may also run for shorter periods of time over a span of weeks, months or even years (Truyen 2016; Taes & Colangelo 2017; Hölttä-Otto et al. 2018; Rennick et al. 2018; Taylor et al. 2018; De Oliveira et al. 2019; Richterich 2019).

The term ‘hackathon’ is a combination of the words ‘hack’ and ‘marathon’ (Briscoe & Mulligan 2014); whereas the usual interpretation of ‘hack’ is in reference to a cybercrime, in this context it refers to exploratory programming. The term ‘marathon’ implies a prolonged, race-like event. This accurately describes a hackathon as a tech-centric, speed design event. Since the first known uses of the term ‘hackathon’ to describe such events in 1999 (Briscoe & Mulligan 2014; Richterich 2019), the hackathon phenomenon has rapidly expanded globally and hackathons continue to grow in popularity. Hackathons may also be referred to as game jams, design jams, hacking festivals, hack days, design sprints and codefests (Briscoe & Mulligan 2014), among others. In this review, hackathons that are not labelled as such, but follow a format inspired by hackathons, are considered ‘hackathon-like events’ and are included in our analysis. As the hackathon format continues to be adapted for different uses and by different stakeholders, new names continue to emerge.

Typically, a hackathon starts with a presentation about the event goals, design challenges, sponsors, schedule and prizes. The theme of the event can either be announced beforehand, or at the start of the event, and could be general, or focussed on a specific task. The process of forming teams may have started before the event via existing relationships and the sharing of ideas via collaboration channels, or at the event based on common interests, skills and project ideas. Once teams are formed, work begins. In the typical hackathon, participants work through the night. Teams brainstorm, build prototypes and, at the end of the hackathon, present their work via ‘pitches’ in a competition for prizes (Briscoe & Mulligan 2014).

Hackathons have much in common with a related approach to design – Agile – which also prioritises the creation of working solutions over a short timescale (sprints; Sims & Johnson 2011, p. 40). Not surprisingly, many organisations that have already adopted agile practices will also use hackathons and hackathon-like events to accelerate innovation (Alkema, Levitt & Chen 2017). Prior studies have reported the emergence of agile practices during hackathons (Alkema et al. 2017), and attempts to integrate hackathon and agile approaches into one event (Ferrario et al. 2014; Hölttä-Otto et al. 2018). Despite the few instances when these two approaches come to overlap, of interest in this paper are the hackathons that follow the more typical structure – a large number of participants gathering over a short period of time (1–2 days) to engage in a design activity that is separate from their regular commitments, whether school or work. In contrast, the goal of design sprints as they are typically described in the literature is problem-solving and innovation within a specific organisational context; as such they are integrated with or complementary to existing organisational processes, and operate at different timescales of 1–4 weeks (Knapp, Zeratsky & Kowitz 2016, p. 40).

1.2. Design and hackathons

Since the 1960s when it first emerged as a discipline, design’s development and impact on industry, research and education has occurred in a number of ‘waves’ (Cooper 2019). The emergence of hackathons in the 1990s and their further gains in popularity in the 2000s (Briscoe & Mulligan 2014; Richterich 2019) coincide with the cusps of the third wave of change, a time of formalising the connection between design and innovation, and the fourth wave, a time of the acceptance of design outside of the discipline (Cooper 2019). The increasing interest in hackathons also aligns with the current (fifth) wave – applying design to understand the future (Cooper 2019) – in that hackathon events increasingly challenge participants to solve complex problems with a focus on benefitting the world and our futures; for example, solving systematic healthcare problems (Alamari, Alabdulkarim & Al-Wabil 2019), addressing sustainability issues such as deforestation (Lodato & DiSalvo 2015) and conceptualising products or services to aid those affected by self-harm (Birbeck et al. 2017).

Design reasoning has commonly been conceptualised as the process of developing an artefact (the ‘what’) that performs some function (the ‘how’) in order to solve the problem (the ‘outcome’; Dorst 2011). The designer must start with the desired ‘outcome’ and devise a new ‘how’, and in turn, design the ‘what’, a pattern known as design abduction (Dorst 2011). The typical design problem is ill-structured, meaning it is not well-defined, the goals are underspecified and the potential solutions are practically limitless (Goel & Pirolli 1992). The formulation of such ‘wicked’ problems cannot follow a linear path (Rittel & Webber 1973); design entails a continuous refinement of both the designer’s understanding of the problem and their ideas for solving it, in a process of co-evolution of the problem and solution spaces (Maher, Poon & Boulanger 1996; Dorst & Cross 2001). In other words, a designer must shift their thoughts between the required purpose/function and the appropriate ways to satisfy that purpose. Since there are two unknowns in design abduction (the ‘what’ and the ‘how’), there is a lot of uncertainty and limited knowledge, making design a long and iterative creative process (Dorst 2011).

At hackathons, the limited time available prevents long periods of exploration and iteration. However, the event is structured as an abductive task – participants may spend significant time ‘searching’ for a problem for which a solution must be designed. Because hackathons are a fairly recent phenomenon, a shared understanding of how this setting affects design behaviour and outcomes has not yet emerged. While many publications focus on outcomes of hackathons, few include discussion related to design activity at the event(s) (Olesen 2017). Existing literature often consists of experience reports or studies of a singular event, or within a specific domain (Raatikainen et al. 2013; Kollwitz & Dinter 2019). The one known attempt to synthesise this knowledge is a recent literature review by Kollwitz & Dinter (2019), who propose a taxonomy of hackathons. However, while their taxonomy accounts for the entire duration of the hackathon from initial idea generation to results, its focus is on how hackathons are/ought to be organised, and not on describing/understanding how hackers design. Other research on hackathons present ways to measure innovation during hackathons (Legardeur, Choulier & Monnier 2010) and investigate how variance in team members’ educational backgrounds influences team performance (Legardeur et al. 2020). We will revisit these concepts throughout this paper.

1.3. Aims

The study of design at hackathons is a logical extension of design research. The unique characteristics of hackathons bring about heightened and condensed design activity in which the typical design processes are challenged and adapted. A better understanding of design at hackathons can help clarify how these events engage participants in the design process (Olesen et al. 2018) and shed light on the consequences of a hackathon’s format on participants’ ideation, design judgement and knowledge of design. Studying how hackathon participants design can also more broadly enrich our understanding of how designers think during an applied design task when in a condensed time frame.

Motivated by the emerging popularity of hackathons, the unique limitations of time and resources during the events, and the lack of an established understanding of design at hackathons, in this paper, we provide a synthesis on the research seeking to uncover (1) the nature of design activity at hackathons and (2) the role of a hackathon’s format and structure on resulting patterns of design behaviour and learning. Based on our review, we then identify a number of knowledge gaps that inspire new potential design research directions in this unique setting. The rest of the paper is organised as follows: Section 2 explains the research methodology, with findings presented in Sections 3 and 4. Section 5 presents possible extensions of this research followed by a conclusion in Section 6.

2. Methodology

The aim of this study was to conduct a systematic literature review; however, such a review can only be conducted within established fields and the literature on design cognition at hackathons is limited. We nevertheless employed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework (Moher et al. 2009) as a means to structure our research. This framework prescribes a four-phase process for conducting systematic literature reviews, as shown in Figure 1.

Figure 1. PRISMA flowchart of our literature review.

Key search terms were identified as ‘hackathon’, ‘design’ (intended to encompass other terms such as design thinking, design process, design cognition and design activity) and ‘creative’ (to encompass creativity, creative thinking and creative process). While hackathons have also been identified with alternate names, such as game jams and hack days, these were eventually omitted from the search as ‘hackathon’ was found to encompass these words in the search results. The search was conducted on 19 November 2019 in the Scopus and Web of Science research databases. The terms were searched in the ‘titles’, ‘abstracts’ and ‘keywords’ fields. The inclusion criteria for the search were journal articles, conference proceedings, books or book sections, published in English. The exact search syntax is provided in Table 1.

Table 1. Search syntax by database with number of results

Searches in the Scopus and Web of Science databases yielded 232 and 109 results, respectively. After duplicates were removed, abstracts of the remaining 262 records were scanned for relevance. At this stage, 146 publications were removed, because they were proceedings of presentations at a hackathon (not publications about design at hackathons), the hackathon was a means to study a different phenomenon, or the publication was not about a hackathon.

A full-text screen was then done on the remaining 116 publications to further screen for applicability. The main reason for exclusion at this stage was the lack of discussion about design activity. After this process, 39 publications were selected as relevant to the topic of study. All 39 publications are included in the literature analysis.

3. Overview of the reviewed publications

Table 2 presents an overview of all reviewed publications and the hackathon(s) they study. All reviewed publications include information on design activity at a hackathon, or directly study design processes at a hackathon event. Most of the reviewed publications consider hackathons to be design or participatory design activities (e.g., Birbeck et al. 2017; Taylor & Clarke 2018). The publications varied in their motivations. A few publications aimed to propose a framework for hackathons (Johnson & Robinson 2014; Buttfield-Addison et al. 2016; Flores et al. 2018). They either presented a pattern of design behaviour as observed at a hackathon, or a structure to be followed at a hackathon. Other motivations included identifying the purpose of hackathons (Komssi et al. 2015), studying the effectiveness of hackathon-like events in teaching design (Artiles & LeVine 2015) and identifying potential benefits of hackathons (Carroll & Beck 2019). A few of the reviewed publications do not aim to study design activity at hackathons, but rather to describe how hackathons are used to find solutions to complex problems, such as accelerating healthcare innovation (Alamari et al. 2019), involving citizens in rural community development (Soligno et al. 2015) and supporting diversity in software development (Filippova et al. 2017). In these cases, discussions on the design processes of participants are limited.

Table 2. An overview of publications and hackathons included in the review. A check mark indicates a ‘Yes’, an ‘x’ means ‘No’ and ‘n/a’ means the information was not provided in the publication

The first section in Table 2 includes information on the publication: citation, subject area and the number of hackathons included in the study. Most of the publications were sourced from journals or conference proceedings in the subject areas of computer science and engineering. Only five publications focus on more than one hackathon, whereas the rest study individual hackathons.

The second section in Table 2 categorises publications based on methodology employed, which spans experimental (8 publications), case study (6), autobiographical (4) observational (14), interview/focus group (13), survey (9), review of literature (4), and even secondary data analysis (1) approaches. For example, in one study, researchers study healthcare innovation at a hackathon by following a set of selected teams throughout the duration of the hackathon and describing the design stages followed (Alamari et al. 2019). Similarly, Taylor et al. (2017) describe the observed behaviours of participants in a series of Inventor Days as they completed various tasks to learn about a community, identify issues and develop solutions for grassroot innovation. In the case of interviews and focus groups, hackathon participants are gathered after the event and asked key questions related to the authors’ research question(s). Surveys are also typically administered to hackathon participants post-event; however, in a few cases, surveys are used as a measurement tool (e.g., to assess the effectiveness of an intervention) and thus participants complete both pre- and post-event surveys (Artiles & LeVine 2015). Finally, in at least one case, secondary data analysis is conducted on registration information to study demographic characteristics of the hackathon participants (Izvalov et al. 2017).

The third section in Table 2 lists the topic of each hackathon under study. Five of the studied hackathons are either labelled as ‘game jams’, or have the aim of game development (Scott et al. 2015; Buttfield-Addison et al. 2016; Izvalov et al. 2017; Olesen 2017; Prieto et al. 2019). Another five publications present education-oriented activities that follow the hackathon pattern (Horton et al. 2018; Rennick et al. 2018; Taratukhin et al. 2018; Gama et al. 2019; Mielikäinen et al. 2019). These hackathon-like events aim to engage students in a heightened design activity, either in-class or as an out-of-class activity, where the specific topic is related to the students’ field of study, with the prompts ranging from addressing natural disasters to solving mechanical engineering problems. Five publications discuss ‘civic hackathons’ (Johnson & Robinson 2014; Soligno et al. 2015; Gama 2017; Taylor et al. 2017; Damen et al. 2019), meaning citizens are involved in solving problems related to their communities. Three hackathons are about healthcare (Piza et al. 2018; Alamari et al. 2019; McGowan 2019), and another three center around the sustainability topics of water quality (Carroll & Beck 2019), biodiversity (Thomer et al. 2016) and ecology (Lodato & DiSalvo 2015). Finally, one hackathon is in each of the following areas: mental health (Birbeck et al. 2017), education (Artiles & Lande 2016), film and television (Karlsen & Løvlie 2017), aircraft design (Saravi et al. 2018), diversity awareness (Safarova et al. 2015), museum artefact design (Rey 2017), student life (Aryana et al. 2019), greeting card reconceptualisation (Page et al. 2016) and app development and financial innovation (Frey & Luks 2016). Also included in the third section is if design was facilitated during the event, if the event was competitive, and the targeted participant profile.

Finally, section four categorises the purpose of the hackathons according to a classification proposed by Briscoe & Mulligan (2014). Hackathons are either ‘tech-centric’ or ‘focus-centric’. Tech-centric hackathons focus on software development and can aim to improve a single application (‘single-application’), a specific platform (‘application-specific’), or create an application within a specific programming language (‘technology-specific’). Applied (or ‘focus-centric’) hackathons focus on contributing to a social issue (‘socially oriented’), targeting a specific demographic (‘demographic-specific’), or addressing a business objective within an organisation (‘company-internal’). The 39 reviewed publications describe a balance of tech- and focus-centric hackathons, with the majority being application-specific or socially oriented.

4. Characteristics of design activity at hackathons

4.1. Participants’ design process

It is evident from the accounts in the publications that participants in hackathons or hackathon-like events follow a design process similar to ones followed in more common design tasks. The overarching pattern of design process at a hackathon includes an initial divergence in order to search for and understand a problem, convergence onto a specific design opportunity, divergence during the development of the solution, a final convergence via testing of the design and preparation and delivery of a final pitch. Specifically, the process typically begins with a team-building (Safarova et al. 2015) and problem identification (Mielikäinen et al. 2019) phase, followed by brainstorming and production (Safarova et al. 2015), ideation, prototyping and testing (Mielikäinen et al. 2019). The hackathon always concludes with a final pitch, termed the showcase (Safarova et al. 2015), or presenting (Mielikäinen et al. 2019) phase. Hackathon participants from the same team may be engaged in multiple phases simultaneously. Further, iteration occurs within and between phases (Buttfield-Addison et al. 2016). This pattern closely aligns with the four phases of the well-known Double Diamond design process (Design Council 2005). Below, we use that model to structure our overview of the characteristics of participants’ activities throughout the duration of a hackathon in each phase. The design process followed by hackathon participants, as described in this section, is illustrated in the (altered) double diamond design model shown in Figure 2.

Figure 2. The hackathon (altered) double diamond design process.

The first phase of the double diamond model is that of ‘Discover’, which describes a divergent activity with the primary goal to gain understanding of the problem or problem area. At hackathons, this phase includes a number of dependent activities. At the beginning of a hackathon, the first activity of significance for participants is to form and get to know their teams, if they have not already formed these teams prior to the event. Once teams are formed, participants identify team skills, knowledge and available resources, and then focus on need finding in order to identify potential topics for their hackathon project. Depending on the hackathon, participants may be given a topic or prompt (e.g., ‘find solutions to “smart villages and territories” problems’; Soligno et al. 2015, p. 737), or may need to begin their design process by first identifying a problem area. It is typical for industry to partner with hackathon events, for example, by having industry representatives attend hackathons and propose project areas. The highest degree of industry involvement is a company-internal hackathon, which is a hackathon exclusive to the organisation’s employees that aims to solve a shared problem. For example, at a hackathon hosted by a large IT provider in Germany, 24 employees from various departments split up into 5 teams that competed to develop an app that supported the sharing of a ‘thing’, such as a car or book (Frey & Luks 2016). In this case, the goal of the hackathon was predetermined, requiring less discovery for the participants. In general, teams either identify their problems from the industry partners (where applicable), from their personal experience (Raatikainen et al. 2013; Suominen et al. 2019) or by conducting user research (Karlsen & Løvlie 2017; Damen et al. 2019). Teams brainstorm potential ideas and use strategies, such as team voting, to converge on a direction for their project (Thomer et al. 2016). Altogether, this phase comprises the ‘finding’ of both the design problem and the design team itself. Hackathon participants engage in a process of perceiving the context (i.e., team and setting) to discover available resources – ‘space, skilled time, knowledge and tools’ (Prieto et al. 2019, p. 2) – to meet their goals.

In the second phase of the model – ‘Define’ – designers construct an understanding of the problem and converge into a problem formulation that defines the aims and requirements of the design and constraints on the solution space. Requirements and constraints may come from the task environment or the designers themselves. In the hackathon setting, a large constraint is time; thus, participants aim to identify a problem for which a viable solution can be developed in the short time frame of the event. Techniques highlighted in the reviewed publications for developing an understanding of the problem include interviews and stakeholder mapping (Damen et al. 2019), as well as user profiling (Rey 2017). These techniques allow for teams to identify key features and create a common reference of the problem. At hackathons, regardless if the topic is given or they have to conduct problem finding on their own, participants aim to develop a shared understanding of the problem with the group (Prieto et al. 2019). Completing this phase is needed in order for the team to establish a direction for the remainder of the design process (Damen et al. 2019).

The third phase – ‘Develop’ – describes another divergent set of activities. At hackathons, development comprise two main tasks: (1) concept ideation (Damen et al. 2019), which is the generation of possible solution ideas and subsequent iteration on those ideas and (2) embodiment (Prieto et al. 2019), which includes the development and demoing of initial prototypes. Common design tools used during this phase are sketching (Thomer et al. 2016; Karlsen & Løvlie 2017) and prototyping (Karlsen & Løvlie 2017; Rey 2017; Suominen et al. 2019). Teams may use provided materials, such as Post-it Notes (Rey 2017), to aid in brainstorming and cheap supplies to build prototypes.

In the fourth and final phase – ‘Deliver’ – teams focus on converging their solution ideas into a final artefact. Teams will test their designs, for example, by assigning a ‘tester’ role to a team member responsible for checking their code for bugs (Pe-Than & Herbsleb 2019), or by asking volunteers to use their prototype to identify any features that do not function as intended (Rey 2017). Test results help determine the design’s final functionality such that teams may have to remove some functionalities of their design in order to complete their projects on time (Rey 2017). The ‘Deliver’ phase is closely linked with the ‘Develop’ phase, and often, teams are observed cycling between the two phases. Once teams finish their evaluation and are satisfied with their results, they synthesise their final design. It is in this last phase that another set of important activities occur.

Hackathons tend to conclude with a final pitch competition, in which participants pitch their designs to a panel of judges (Gama et al. 2019) with the hopes of winning prizes and recognition. The presentation of a design solution to an audience (e.g., clients) is not unique to hackathons, but this activity is significantly emphasised at hackathons where it takes up a significant portion of the event. From both personal experiences and the literature, we have learned of the importance given to the final hackathon pitch. Prizes are dependent on the pitch; thus, teams will dedicate a significant portion of their resources to its development and final presentation (Richterich 2019). We argue, that at hackathons, the building and testing of the design is separate from its pitch. To many participants, a successful pitch is more important than a working final product, as it provides an opportunity to demonstrate what the product intends to do, regardless if that functionality is adequately implemented. Some hackathons may not even require a working prototype (Briscoe & Mulligan 2014). While not seen in traditional design processes, we propose pitching as a uniquely significant activity in the Deliver phase of the hackathon design process; therefore, we have altered the double diamond design process to include the hackathon pitch in order to emphasise this element of the hackathon and more accurately dedicate a portion of the design process to it.

4.2. Design process facilitated by hackathon structure

The structure of the hackathon event itself can encourage and shape the design process followed by hackers during the event. In this section, we will review studies in which a design process was encouraged or facilitated during a hackathon by the hackathon organisers.

The first way in which hackathon organisers influence the design process of participants is through education and suggested processes. Some hackathons offer participants workshops on specific design phases, such as ideation (Lodato & DiSalvo 2015; Filippova et al. 2017), or on the entire design process (Page et al. 2016). At these events, participants are not instructed to use the design process; rather, it is merely provided as a tool to help the teams. Other events suggest a design process to teams, such as Design Thinking (as popularised by Brown 2008) with an additional ‘pitch’ phase (Flores et al. 2018). When design instruction is offered to hackathon teams, there is a tendency for teams to use user-centered design practices (Page et al. 2016) and more closely follow an established design process (Flores et al. 2018). Further, a design-oriented structure on the event indicates an awareness of design processes on the part of hackathon organisers. Learning about the steps in a design process encourages participants to follow the process, which, in turn, improves outcomes at hackathons (Aryana et al. 2019).

Some hackathons include schedules or other types of structures that directly enforce a design process for participants. These hackathons divide the event into phases, which may be reflective of established design frameworks. For example, the timeline at some hackathons followed the phases of Design Thinking (Taratukhin et al. 2018; Alamari et al. 2019), the Logical Framework Approach (Soligno et al. 2015), the Challenge-Based Learning Framework (Gama et al. 2019) and the Mechanics Dynamics Aesthetics framework (Buttfield-Addison et al. 2016). There is evidence to suggest that an imposed hackathon structure that is design process oriented is conducive to effective designing (Soligno et al. 2015; Buttfield-Addison et al. 2016). Finally, some hackathons facilitate design-related activities and require participation in certain tasks, such as answering guiding questions and documenting design decisions (Gama et al. 2019). This strategy encourages hackathon participants to engage in activities typical of design processes without imposing a strict structure.

4.3. Hackathons as a tool to teach design

Event offerings at hackathons, such as workshops and schedules, among others, can encourage and facilitate (if not impose) the use of systematic design processes by hackers. Often, teaching hackers how to design is one of the expressed purposes of a hackathon. Similarly, one of the motivations participants have for attending a hackathon is gaining skills, including design skills. Therefore, the facilitation of design processes is unsurprising. In many cases, hackathons promote learning design processes as an official goal.

The hackathon model is sometimes used in educational settings to teach (engineering) design skills. For example, Gama et al. (2019) describe undergraduate course projects inspired by the hackathon framework. The hackathon-like events had four phases tied to respective course deliverables: information gathering, user needs identification, requirements specification and prototype building. Students employed design methods to aid in their process, including brainwriting, voting heuristics, SCAMPER (Substitute, Combine, Adapt, Modify, Put to another use, Eliminate, Reverse), persona building and physical computing cards. In another design education instance (Page et al. 2016), each day of a 4-day hackathon was dedicated to different steps in the double diamond design process model. While students were not required to follow the model, the aim of the instruction on the model was to increase awareness of design processes and to systematically support design learning.

In another example, the hackathon structure is used to facilitate project-based learning (PBL) in engineering. PBL requires significant time and sustained interest from both the instructor and students. Motivated by these drawbacks of PBL in engineering courses, Horton et al. (2018) make a case that hackathon events can be used to meet all elements of PBL. This includes (1) meeting student learning goals and (2) providing essential project design elements. These include a challenging and authentic problem that sustains students’ inquiry, independent thinking and decision-making, as well as opportunities for design critique and public presentation of the project results.

Perhaps, the most comprehensive use of hackathon-like events for teaching engineering design is described in Rennick et al. (2018). Here, curricular hackathons – named ‘Engineering Design Days’ – form the basis of a design ‘lattice’ around which other curriculum components (foundational math and science, discipline-specific engineering science, etc.) can develop in an integrated way (Hurst, Rennick & Bedi 2019a ). In this format, junior engineering students work in teams to solve an open-ended engineering design problem over the course of 2 days. These events are made possible through coordination among all course instructors in the term who contribute space in their course schedules such that students do not have to attend any additional classes or other curricular activities during the event. Design teaching here is more explicit. In many cases, the event includes a ‘warm-up’ period that familiarises students with the problem space and explicitly connects aspects of the design problem to concepts from their courses.

How effective are hackathons at teaching design? To the best of our knowledge, given the relatively recent use of hackathons in curricular settings, rigorous studies evaluating the format’s effectiveness in improving student participants’ design skills have been rare. Certainly, educational hackathon-like activities provide three major advantages to design education. First, at hackathons students experience the entire design process within a condensed time frame. This is particularly an advantage, as students have otherwise few opportunities to engage with all phases of design (Flus, Rennick & Hurst 2020). Second, design instructors can calibrate many aspects of the design problem and its representation to match them to the students’ academic level, context and educational goals (Hurst, Litster & Rennick 2020). Third, hackathons have been found to advance participants’ knowledge. At least one previous study has found that through participation in a design hackathon, nondesigner participants display an increased use of design thinking terminology and approaches (Artiles & LeVine 2015). A second study found that hackathons also supported education beyond design; that is, participants engaged in social learning and gained STEM knowledge, such as programming (Fowler 2016). Yet, due to the fast-paced format, participants may also overlook certain good design practices, such as gathering requirements from users or maintaining design quality (Gama 2017). A need arises to more deeply understand how the hackathon format can promote (and perhaps also hinder) design learning, as hackathons become more widely used in an engaging and fast-paced educational environment.

5. Future directions for design research in hackathons

In this review, we have identified the primary ways in which the role of designing is referenced in studies on hackathons. What emerges is a picture of hackathons as an authentic setting in which design activity occurs. How, then, does designing at hackathons compare to designing at other common settings in which design is studied (McMahon 2012)? On the one hand, given the range of participants’ design and other types of expertise, their self-motivation for participating in the events and personal interest in the chosen topic, and the significant interest and input from organisations who sponsor them, one could argue that the resulting design activity is more authentic than what is commonly studied in controlled experiments. On the other hand, the structure of hackathons themselves – in particular, the extreme time pressure under which they place participants – constrains design activity in unique ways that may not be representative of or generalisable to the authentic design activity that is typically studied through observations of (expert) designers in field studies. Nevertheless, studying designing as it occurs in hackathons may uncover unique processes designers follow during a heightened design event.

Despite this potential, we have identified a significant gap in the intentional study of design activity in hackathons. No publication included in this review had the main objective of studying the design behaviour of hackathon participants. While few publications outlined group processes, or included personal accounts of a hackathon experience, they do not develop an understanding of the common processes followed at hackathons, beyond the accounts of a few groups. In this section, we outline a number of future directions for design research at hackathons.

5.1. Methodological approaches to studying design at hackathons

A standard methodology for studying design cognition is protocol analysis (Ericsson & Simon 1984) in experimental settings. Unfortunately, protocol analysis is impractical in the case of long, slowly evolving design processes, and for large quantities of data (Goldschmidt & Weil 1998). Hackathons are attended by hundreds of participants, who continuously design for a significant amount of time in loud and difficult to control settings, thus making verbal protocol analysis infeasible. In order to study them, the publications included in this review employed methodologies such as interviews and ethnography. These methodologies require a lot of manual effort from the researchers; thus, only few groups are able to be studied at a singular event. This limits the size of the study population, the amount of data possible to collect and the applicability of the findings across many hackathons.

It is thus necessary for new methods to be developed that can study design cognition in ‘in-situ’ environments, such as hackathons. Legardeur et al. (2010) utilised a web-based application on which participants self-reported their design activities during hackathons. The advantage of this approach was two-fold: (1) it presented participants with a chance to reflect and assess their progress, while (2) collected data for the research team. While we see the benefit of this approach, we question the extent to which participants will understand their own design process without formal training, as we expect to be the case for software engineers and other typical hackathon participants that have limited design education and/or experience.

The inherent large quantity of data generated at hackathons – for example, question and answer in text-based participant–mentor help channels, final project descriptions and judging criteria and scores, points at opportunities to take new data-driven approaches such as text mining and natural language processing. Combined with more traditional protocol analysis approaches, such as the Function–Behaviour–Structure ontology (Gero & Kannengiesser 2014) or linkography (Goldschmidt 2014), the methods may allow for more efficient data collection and analysis at a larger scale in the challenging in-situ environments of hackathons. Further exploration into data-driven methodologies and how they may need to be adapted to suit a study on hackathons is needed.

5.2. Leveraging alternative hackathon formats

Another significant opportunity to develop new study approaches lies in the emergence of hackathons that are held in an online environment. This format has become particularly more common during the COVID-19 pandemic, during which previously scheduled in-person hackathons were adapted to be held in virtually. These hackathons function similarly to in-person hackathons, but with increased dependency on virtual platforms for communication and collaboration both between participants and event staff and mentors, but also between participants within teams. The online format presents an interesting opportunity for data collection. The digital age has introduced an ease to virtual collaborations. Designers have created ways to simulate physical collaborative experiences (e.g., through interactive Post-it Note software; Everitt et al. 2003) to facilitate virtual work. Video and audio calls can be recorded for transcription, data from ‘chats’ can be collected and digital work produced by teams can be submitted with the final design. These data collection methods would result in an abundance of data and allow for more data-driven methodologies to study design.

At the same time, the virtual and remote nature of collaboration between design team members adds a new dimension to design activity at hackathons, which must be carefully considered when conducting research. Asynchronous and remote work is common among distant collaborators, but compared to face-to-face settings, synchronous design activities conducted virtually may result in reduced interactions between collaborators. When compared with face-to-face interaction, communication via videoconference has been found to reduce the back-and-forth between participants and increase the length of each turn (van der Kleij et al. 2009). Asynchronous collaboration has many additional challenges beyond those typical of collaborative design (Hauck 1995). Collaborators may be in different time zones, have limited access to internet connection and experience limitations when building hardware and physical designs. These additional challenges are important to consider when studying data collected from virtual hackathons. At the same time, investigating virtual hackathons alongside in-person events could highlight the differences between virtual and face-to-face collaborative design.

5.3. The impact of fewer incubation opportunities

Participants grow significantly fatigued at hackathons, where they receive minimal sleep or breaks. Breaks enable people to recover from extended periods of concentration and can serve as incubation periods (Woodworth 1938) – a concept that is relevant to a cognitive activity such as design. While in an incubation stage in the creative process, the designer is no longer actively working on the problem (Wallas 1926; Tsenn et al. 2014). It is thought that the mind continues to work on the problem subconsciously. Research has found that taking a break during problem-solving may lead to a moment of insight (Burkus 2014). Additionally, incubation has been shown to help reduce design fixation and encourage the generation of a greater number and variety of ideas (Cardoso & Badke-Schaub 2009; Tsenn et al. 2014).

But what if the time available to the designer does not permit incubation? In a hackathon, breaks are rare and the breaks participants do take are much shorter than those in more relaxed design situations. In a traditional design project, incubation is naturally built into the project. Design occurs over days, weeks, months or even years. This implies that periods of rest can also span days, weeks, months or years. While these designers may not intentionally include incubation periods, they still receive rest from problem-solving. This helps lessen the effects from fatigue and fixation. We assume that these lengthy breaks are not possible at hackathons. Thus, a need arises to investigate whether the condensed nature of a hackathon causes fewer incubation periods, and how this affects the overall quality of the final design.

5.4. Design thinking in increased uncertainty

According to the dual-process theory (Kahneman 2011), there are two types of thought: fast, intuitive thinking (type 1), and slow, analytical thinking (type 2). Recently, Kannengiesser & Gero (2019) map this theory onto design thinking, what they call fast and slow designing. Experienced designers use more type 1 thinking, or fast design; they can more quickly transform requirements into design solutions (Kannengiesser & Gero 2019; Hurst et al. 2019b ) using their prior ‘canned’ designs, or ‘design types’ (Schön 1988). Novice designers, too, exhibit fast design behaviour; however, their ability to rely on past designs is limited.

What type of design thinking is employed at hackathons? Progressing from an ill-defined design problem to a final solution is dependent on the designer’s ability to manage the ‘inherent “uncertainty” that pervades real-world design problems’ (Ball & Christensen 2019, p. 36). The atypical setting of a hackathon (e.g., dramatically increased time pressure or a very ill-defined problem statement that is revealed at the hackathon) adds new sources of uncertainty for even experienced designers. This uncertainty would normally require an increased need for slow design. Yet, the added time pressure of a hackathon would suggest increased occurrences of fast design. Time pressure is not necessarily a unique factor of a hackathon, as most, if not all, design tasks have a deadline. However, while 24 total hours to design may not be considered a short design time frame, it is so when the 24 hours are condensed into one day, rather than over an extended period of time. The requirement to progress through the design process quickly and continuously may suggest an increased inclination for fast design. Thus, we reflect on how the conflict between both types of thinking is resolved. What is the predominant type of thinking at a hackathon? How does time pressure affect it? And how do designers manage increased levels of uncertainty during a heightened design task?

5.5. The role of expertise

As a designer’s mental function matures and they rise in their level of expertise (Dreyfus & Dreyfus 1980), their understanding of what it means to design may also change (Daly, Adams & Bodner 2012). An individual can have different levels of expertise in different skills; for example, a designer can be an expert in product design but a novice in web design. The variance in expertise demonstrates ‘the human ability to respond to an environment and adapt behaviour accordingly’ (Lawson & Dorst 2013, p. 82), such that one’s environment directly influences one’s expertise. This is evident within hackathons at three levels.

First, by design, hackathons typically enable collaboration between different topic experts (Frey & Luks 2016); thus, different team members may have different subject matter knowledge. Research on the impact of the diversity of expertise on hackathon performance is limited and inconclusive (Legardeur et al. 2020).

Second, we hypothesise that performance at a hackathon is related (to some degree) to one’s level of expertise in a relevant domain. For example, more experienced software engineers will be better prepared to complete a software design project within the hackathon environment. This extends to the degree of expertise in the discipline of design itself. In a study that we are currently conducting to compare the hackathon experiences of designers to nondesigners, the most notable preliminary finding is the ability of designers to critically discuss their processes and decision-making during hackathons. Designers are able to recall not just what decisions they made, but the rationale for each decision and how it served their event goal. Nondesigners lack the vocabulary and knowledge to participate in such conversations, and it appears as though the design decisions they make during hackathons are not as intentional as those of their design peers.

Finally, we argue that there is ‘hackathon expertise’; that is, valuable experience accumulated from participating in hackathons. Those who have attended more hackathons will have a more advanced understanding of how to approach a hackathon than participants that are new to this type of event. We hypothesise that these varying levels of expertise significantly affect participants’ design activities at hackathons.

In summary, a further exploration of team composition at hackathons in terms of diverse subject matter knowledge and expertise, as well as prior hackathon experience, will give more insight into how to optimise team performance at these events.

6. Conclusion

In this paper, we have reviewed the literature with the goal of identifying the role that design – as an activity and/or instructional goal – plays in hackathons. The publications reviewed describe studies on hackathons from a range of disciplines and with widely varying objectives.

The review demonstrated that hackathon participants follow typical divergence–convergence patterns in their design process. Often, hackathons will prescribe and facilitate a design process in order to guide participants and encourage more successful design and learning outcomes. However, the event structure also requires increased emphasis and time in a number of other activities, such as finding and getting to know both team members and problems to solve, and preparing for the solution pitch and demo.

As this review has demonstrated, design occurs at hackathons in expected and unexpected ways. The nature of hackathons – loud environments attended by a large number of participants – poses challenges for studying design cognition, as also demonstrated by the methodological limitations of current studies on hackathons. Yet, hackathons present a unique setting for studying designer activity at each stage of the design, especially under conditions of time pressure, reduced incubation time and across different levels of domain and design expertise. For these reasons, hackathons are valuable and promising opportunities for future design research.