Behavioural evidence to inform the COVID-19 pandemic response: Ireland's Social Activity Measure (SAM)

Abstract

Humankind's main defence against the virus that causes COVID-19 (SARS-CoV-2), besides vaccine development, was co-ordinated behaviour change. In many countries, co-ordination was assisted by tracking surveys designed to measure self-reported behaviour and attitudes. This paper describes an alternative, complementary approach, which was undertaken in close collaboration with officials in the Department of the Taoiseach (Irish Prime Minister). We adapted the Day Reconstruction Method (DRM) to develop the ‘Social Activity Measure’ (SAM). The study was conducted fortnightly for 18 months, with findings delivered directly to the Department. This paper describes the method and shows how SAM generated a detailed picture of where and why transmission risk occurred. By using the DRM, we built aggregate measures from narrative accounts of how individuals spent their previous day. SAM recorded the amount, location and type of social activity, including the incidence of close contact and mask-wearing, as well as compliance with public health restrictions by shops and businesses. The method also permitted a detailed analysis of how public perceptions and comprehension are related to behaviour. The results informed government communications and strategies for lifting public health restrictions. The method could be applied to other future situations that might require co-ordinated public behaviour over an extended period.

Introduction

The COVID-19 pandemic proved an immense challenge for governments and health systems the world over. Beyond the essential life-and-death nature of the threat, much of the challenge was the speed with which events unfolded and the accompanying need to understand them quickly. During the first 2 years of the pandemic in particular, this caused extreme difficulty for officials and politicians trying to devise, co-ordinate and implement a policy response. It also challenged researchers who produce evidence for public policy. The relevant medical science was constantly developing, while the epidemiological picture depended on non-linear transmission and the unpredictable evolution of new virus variants. Yet researchers and policymakers not only had to understand and respond to the latest movements of the virus, they needed also to focus on people. Humankind's primary defence was co-ordinated behaviour change, especially prior to the availability of COVID-19 vaccines, but also thereafter. From early in the pandemic, it was therefore apparent that behavioural science could perform a role by rapidly collating and producing evidence to inform pandemic policy (Lunn et al., 2020; van Bavel et al., 2020).

This paper describes a behavioural study undertaken by the Behavioural Research Unit (BRU) at Ireland's Economic and Social Research Institute, in collaboration with the Department of the Taoiseach (Irish Prime Minister). The project began in late 2020. Prior to December of that year, Ireland had fared better than most Western European nations in containing the virus and limiting the death toll. However, a policy to open up businesses for the holiday season coincided with the arrival of the Delta variant of SARS-CoV-2. A massive spike in COVID-19 cases, hospitalisations and deaths followed, along with a second lockdown. Around this time, after discussions with behavioural scientists in the BRU, the Department of the Taoiseach commissioned the ‘Social Activity Measure’ (SAM). Data collection began in January 2021 and was undertaken fortnightly for 18 months. As far as we know, the approach taken by SAM was unique and, from the use made of the data and findings, contributed meaningfully to the pandemic response.

The basic logic of SAM was that if behaviour change was humankind's best defence against the virus, we should measure behaviour accurately and in detail. Ireland, like many other countries, began a national tracking survey early in the pandemic that collected general self-reported behavioural measures together with some opinions and attitudes about the pandemic and the government's response. SAM took a different approach. The aim was to produce more granular and detailed behavioural data by adapting the Day Reconstruction Method (DRM) of Kahneman et al. (2004). The DRM is a specific method of prompted recall that helps survey respondents reconstruct an account of their experiences during the previous day using principles of cognitive science to improve recall and reduce bias. The DRM is well established to minimise memory bias, particularly for factual information such as time spent in specific locations (Lucas et al., 2021). Lades et al. (2022) provide a detailed discussion of the strengths of the DRM. For our purposes, we adapted it to generate an anonymous, online study to measure detailed behaviour relevant to the transmission of COVID-19. Although we cannot be certain, especially given that a volume of behavioural data collected by governments around the world during the pandemic remains unpublished, as far as we are aware, our use of the DRM for studying behaviours related to the spread of COVID-19 was unique internationally.

The contribution of the present paper is to describe how SAM was carried out and to illustrate its usefulness in generating evidence for pandemic policy. We show how the technique conferred some advantages over standard tracking surveys. The pattern of recall in SAM also indicates likely errors in surveys that require people to recall behaviour over multiple days. The design allowed the research team to study detailed protective behaviours and aggregate them across multiple locations. A set of psychological variables collected at the end of the survey also allowed us to model drivers of the relevant behaviours. Lastly, by describing their experiences when visiting different types of businesses (e.g. restaurants, shops and sports facilities), SAM respondents effectively acted like mystery shoppers to inform COVID-19 regulatory policy.

After outlining the method, we provide example findings to illustrate the above advantages and describe how they informed policy decisions. The paper is not intended to be a complete account of SAM findings and policy relevance (which together could fill a substantial volume). Nor is it intended to be a comprehensive empirical assessment of the relative accuracy of different data-gathering techniques – there are no ‘official’ figures for behaviour during the pandemic against which methods can be assessed. The aim is to describe and illustrate SAM sufficiently well to assist anyone who might consider deploying a similar or improved version in the future, whether in the face of a resurgence in the COVID-19 pandemic, another pandemic or indeed any emergency that requires a response centred on everyday behavioural change. The experience of the researchers involved and of the Irish Government, who continued to fund and run the study for 18 months, was that SAM provided beneficial behavioural evidence to support pandemic policy.

Study design

Since the purpose of this paper is to describe and illustrate the advantages of the behavioural technique, this section focuses more on questionnaire design and analysis than sampling; the method could be used with most sampling techniques. The SAM questionnaire differed from standard surveys in two main ways. First, the initial section of the survey adapted the DRM technique to prompt recall of all locations the respondent had visited during the previous 2 days. Second, it used the recollections of locations prompted by the DRM to direct the respondent to an appropriate subset of 13 branched survey modules that each covered a different type of location. Most participants only completed three or fewer of these modules. Thus, the branching was extensive and the bulk of the study questions was specific to the combination of locations that each respondent had visited.

At the start of the study, participants were given strong assurance that responses were completely anonymous. An adapted form of wording from the DRM (Kahneman et al., 2004; Lades et al., 2021) then directed them to revisit each of the previous 2 (named) days:

‘Because it might be difficult to remember exactly what you did on [yesterday] and [day before yesterday], we will do this in two steps for each day, starting with yesterday first.’

‘ … think of the day as a series of stages (like scenes in a film) and make a quick note of anything you did outside of your home each morning, afternoon and evening. If you stayed at home but someone called to your home (e.g. visitor, plumber, etc), take note of this too. You don't need to go into detail but try to write a few words that will remind you of where you went, what you did and who you met.’

Starting with activities carried out yesterday, participants typed into open text boxes, which were split into morning, afternoon and evening. These responses were not analysed; the purpose was solely to aid recall for the questions to follow. Following this exercise, participants were presented with a list of potential locations, such as shops, workplaces, other homes and pubs/restaurants/cafés, and asked to select any that they had visited. The list was designed and piloted to be fairly exhaustive but also contained two ‘other’ categories (one indoor and one outdoor). The process was then repeated for the day before yesterday. Responses to the two location lists determined which of the branched modules participants completed. In addition, all respondents answered questions about visits to their home and any walking, running or cycling in their neighbourhood. At this point, the study had not mentioned COVID-19.

The modules gathered details about the type of location participants visited and behaviour at that location. For instance, the module for shops recorded the type of shop, whether they had to queue and whether staff and other customers wore masks. Similarly, the module for cafés, pubs and restaurants asked whether the visit was to collect takeaway or, if not, whether the participant sat inside or outside. For locations visited more than once, one occasion was selected at random for these more detailed questions. For all locations, participants provided information on how many people from other households they had arranged to meet there or had met for more than 15 min, how long they were present, whether soap or sanitiser was provided for cleaning hands, whether they used it, whether they kept at least 2 m from other people, whether they wore a facemask and, for indoor locations, how well ventilated it was. Thus, for each location, questions proceeded in a narrative fashion for a specific occasion. As well as exploiting the DRM, the study therefore aligned with evidence that spatial context can serve as a scaffold for episodic memory (Robin, 2018). Question-wording was kept factual and neutral, with no evaluative questions that probed opinions or attitudes about behaviours.

In this way, as well as prompting recall, the study design differed from a standard linear survey. Respondents only answered questions about locations they had visited during the previous 48 h and identified by the DRM-prompted recall method. Aggregation of behaviours was done by the research team during data analysis. For instance, rather than asking respondents directly how many close contacts they had during the previous days, or how often they had worn a mask in public places, the analysis aggregated information across separate answers about, say, a specific shopping trip, meeting in a café, visit to a relative's house, etc., where on each occasion the participant provided information about how many people they met, for how long, in what context and whether they and others wore a mask.

Of course, we cannot be sure that aggregating responses like this produced more accurate measures of behaviour than asking respondents to aggregate their own behaviour in response to more general questions. However, researcher aggregation of behaviour from diary methods has been shown in other domains to be more accurate than self-aggregation (e.g., Fonseca et al., 2002; Heeb and Gmel, 2005; McNaughton et al., 2005; Wutich, 2009) and the prompted recall of events in reverse chronological order can improve recall (e.g., Loftus and Fathi, 1985). Moreover, behaviours relevant to the spread of COVID-19 are likely to be more complex for survey participants to aggregate. For a respondent to assess accurately, say, how often they had close contact or wore a mask during a recent reference period, they would need to undertake the sort of detailed memory search that our method explicitly walked them through, with the relevant episodes prompted. Furthermore, this modular approach allowed the research team to not only build up a picture on how protective behaviour varied across individuals and over time, but also across different types of locations.

Following the completion of the modules detailing behaviour, SAM asked a range of psychological and attitudinal questions that included, among other things, measures of personal worry about COVID-19, perceptions of risk, feelings of tiredness with restrictions and confidence in government. The survey finished with a set of standard socio-demographic questions.

The findings below derive from 36,000 observations collected in fortnightly waves of 1,000.¹ Respondents were recruited via existing online panels of two of Ireland's national market research agencies, which alternated between waves. Members of both panels were paid €4 for completing the survey. Quota sampling was used to obtain samples of adults (aged 18+) that were nationally representative by age, gender, social grade (defined by the occupation of the chief earner) and region. Ideally, data would have been collected from either a fresh cross-section of participants in each wave or from a longitudinal panel of repeat participants. However, given budget and, more importantly, time constraints, from the fifth wave of data collection onwards, respondents on the panels were permitted to respond to more than one wave. Repeat participation was always at least 4 weeks apart, but this was recorded, so any findings could be checked to ensure that changes over time were not confounded with willingness to be a repeat participant.² While the socio-demographic composition of the sample was matched to Central Statistics Office control totals on every wave of data collection, some hard-to-reach groups were doubtless underrepresented, in particular as the study was conducted only in English. The survey was programmed in Gorilla Experiment Builder (Anwyl-Irvine et al., 2020), spread across 7 days of the week for each wave, could be undertaken on mobile, tablet or PC and took approximately 15–20 min to complete.

Illustrative evidence for policy

This section provides examples of the types of findings that SAM generated and describes how they informed policy decisions. The aim is to present a subset of results to illustrate the advantages of the method. During the period concerned, the research team presented results from the latest wave of SAM to officials in the Department of the Taoiseach every fortnight. These meetings were two-way, with policymakers often raising specific questions they wanted to investigate using the SAM data. On one occasion, in March 2021, results were discussed directly with the Taoiseach. On multiple occasions, specific findings were sent to Cabinet Committees, to the Chief Medical Officer, to the Government Information Service and to Departmental Heads of Communication. Findings were posted on the Department's website in the form a slide deck and accompanying short commentary.³

Figure 1 demonstrates how the research team were able to aggregate data across individuals and locations to give the government a real-time picture of how the frequency and locations of close contacts were changing.⁴ For instance, from January to April 2021, Ireland was under ‘Level 5’ public health restrictions. People were confined to a 5 km radius of their homes (unless travel was essential), schools were closed, non-essential retail establishments were closed, workers worked from home wherever possible, hospitality (cafés, pubs, restaurants and hotels) and sporting venues were closed and gatherings and visits to other homes were not permitted. The SAM data revealed that despite these restrictions, aggregating across all locations, close contacts were increasing (Figure 1, left). Because the data were collected via modules for each location visited, we were able to uncover that the main reason for the increase was a minority who were having close contact while engaging in social visits to other homes, in violation of restrictions (Figure 1, right, blue line). Further analysis at the time showed that this behaviour was spread across socio-demographic groups and that people were less inclined to take precautions (e.g., ensuring good ventilation and wearing a mask) when in other people's homes compared to other locations. This evidence was used by the government communications team who used messages specifically about home visits. Care was taken to communicate widely that the large majority of the population was not going into other people's homes, given the prevalence of conditional co-operation in collective action settings (Chaudhuri, 2011; Thöni and Volk, 2018). Insights about where close contacts were most likely to occur and where people were more likely to avoid them also contributed directly to decisions about the timetable for lifting the Level 5 restrictions, which prioritised outdoor meetings and activities. Similarly, when COVID-19 cases began to rise again in Ireland in the Autumn 2021, the SAM data pinpointed the contribution that close contacts at work were making as more employees returned to workplaces after the Summer holiday period (Figure 1, right, orange line). Again, this information allowed interventions and communications to be targeted. Overall, the granularity by location that the modular design of SAM afforded was of clear benefit to policymakers.

Figure 1. Proportion of adults in Ireland who had at least one ‘close contact’ (spending 15 min or more within 2 m with an individual from another household) with and without wearing a facemask (left) and without a facemask by location (right). ‘Outside’ refers to outdoor locations such as parks, beaches and neighbourhood streets. ‘Hospitality’ refers to cafés, pubs, restaurants and hotels.

Each fortnight, similar analyses to Figure 1 were produced for the frequency of visits to different types of locations, where and how often individuals were meeting up with people from other households and, in each location, whether people were cleaning hands, maintaining social distance and wearing masks. Each of these measures could also be broken down by socio-demographic group and vaccine status. In general, the SAM data could be aggregated in multiple ways to provide a detailed picture of where, how and among whom transmission of the virus might take place. As researchers who also worked with data from Ireland's national COVID-19 tracking survey, we found this flexibility in the SAM data to be particularly helpful. As with many other high-quality COVID-19 tracking surveys (e.g. Blom et al., 2020; Kapteyn et al., 2020), Ireland's main tracking survey asked respondents themselves to aggregate behaviours over a reference period and multiple locations, providing a self-reported frequency. By instead asking detailed questions only about locations each individual had visited and then leaving the aggregation to the data analysis, the SAM data allowed us to address additional research questions.

In fact, the SAM data contained some interesting patterns in self-reported aggregate behaviour in the context of COVID-19. Figure 2 displays the number of locations that respondents reported visiting, broken down both by the day of the behaviour and the day of survey completion. The variations by weekday were of some interest to policymakers. However, we observed that respondents recalled visiting more locations when going back through yesterday than when thinking about the day before yesterday. Overall, this effect was highly statistically significant⁵ and implied that respondents were forgetting approximately 1-in-20 locations visited the day before yesterday. When considered across all days, this effect size is not particularly large, but there were systematic differences across days. The effect was stronger for weekdays, presumably because weekend activity was more memorable. We collected data only in relation to the previous 2 days, always asking about yesterday first and only within the context of the DRM, so we cannot know if this pattern might generalise to longer periods and other methods. However, this pattern may be of interest to others seeking to collect similar data that requires recall of behaviour. Given the disparities when respondents considered just the previous 2 days, it is possible that self-reports of behaviour over, for instance, a 7-day reference period might be subject to considerable measurement error.

Figure 2. Reported number of locations visited on each day of the week by day of survey completion. Error bars are 95% confidence intervals. The vertical axis range is set to approximately one standard deviation of the number of locations across individuals.

Other advantages of collecting multiple, relatively accurate measures of behaviour across contexts are the possibilities for investigating drivers of behaviour. It was apparent by Summer 2021 that movements in the behavioural measures collected by SAM were closely related to official COVID-19 case numbers, which were released by the Department of Health and reported on news programmes every evening in Ireland. Figure 3 displays the relationship across the period. The left-hand chart shows variations in monthly mean case numbers and a mean risk score for individual behaviour calculated from the SAM data. The right-hand chart is a scatter plot of month-on-month changes in the variables.⁶ Throughout the period, we experimented with different ways of aggregating the behavioural data into a single metric to capture variation in the riskiness of behaviour, in order to make comparisons of this sort. This score displayed in Figure 3 varied from zero to four, based on whether an individual had close contact the previous day, visited two or more locations per day, met up with more than one person from another household per day and took mitigation actions (wearing a mask, keeping 2 m from others and cleaning hands) less than two-thirds of the time in each location they visited. These criteria were set to capture variation, as each of these behaviours was undertaken only by a minority of the population, but the majority undertook at least one of them. As more people were vaccinated throughout 2021 and public health restrictions were gradually lifted, there was a clear upward trend in this risk score. However, on a month-by-month basis, the change in behaviour was strongly and negatively correlated with the (logged) case numbers.⁷ Of course, there would have been some reverse causality in this relationship. The extent to which increased activity led to higher cases (albeit with a time lag) would have dampened the estimated negative relationship.

Figure 3. Relationship between riskiness of behaviour and new daily cases of COVID-19 (as released by Ireland's Department of Health). Risk scores (0–4) are based on whether an individual had close contact the previous day, visited two or more locations per day, met up with more than one person from another household per day and took mitigation actions less than two-thirds of the time. The left-hand chart displays the time-course of new cases and risk scores. The right-hand chart plots the change in risk score from the previous month against the change in (log) daily case numbers.

For policymakers, the fact that behaviour was tied to reported case numbers was important. There was a debate within policy circles about the advisability or otherwise of continuing to release daily case numbers publicly. The SAM data suggested that ceasing to do so might remove a factor that was moderating behaviour.

The SAM data were able to offer further insights into the psychology behind social activity. The behaviours recorded in the modules could be modelled to identify relationships with socio-demographic background characteristics and the psychological scales collected at the end of the survey. Such exercises were undertaken at multiple points during the study period. An example is provided in Figure 4, which reports output from a model estimated for the period under Level 5 restrictions (January to March 2021). The chart shows standardised coefficients for multiple psychological scales in a model where the dependent variable was the risk score described above. The strongest psychological determinant we recorded throughout SAM was people's response to a question about how worried they were personally about COVID-19. Further work showed that this variable was linked to news about case numbers and was less associated with worry about contracting the virus than with worry about friends and family and whether the health service would be overwhelmed. For policymakers, this analysis therefore further supported ongoing communication of the epidemiological situation, including hospitalisations. Another variable of note was how tired people were with sticking to restrictions. This was a constant theme in public debate, as some politicians expressed doubt about the public's willingness to keep complying. These views were often reinforced by comments from industry lobby groups that stood to gain from restrictions being relaxed. They were also bolstered by some researchers, assuming that declining compliance must reflect some kind of ‘pandemic fatigue’ (World Health Organization, 2020; see also Harvey, 2020). While the SAM data suggested that fatigue was a factor, Figure 4 points to other important psychological factors. Notably, these included a scale that directly asked respondents which was more important to them, the burden of the restrictions or preventing the spread of the virus. Other important factors were whether people viewed the public health restrictions as easy to follow, whether they viewed the overall set of measures as coherent or contradictory and whether they thought other people were following recommendations to prevent the spread of the virus and confidence in the government's handling of the pandemic. Most of these factors were chosen for inclusion in the survey and for analysis based on how they mapped onto the literature on drivers of co-operation in collective action problems (e.g. Fehr and Schurtenberger, 2018): the value placed on the common gain (worry, burden vs spread), belief that the co-operative strategy will lead to the gain (ease and coherence of restrictions and confidence in the government) and belief by conditional co-operators that others will also comply. Other factors were included in the survey because they were receiving public attention at the time. These were generally less influential, including deterrence for non-compliance and whether the public understood how the virus transmitted, as captured in Figure 4 by variables assessing people's belief that they might get caught and fined for violations and their self-reported understanding of transmission (similar results were found for an objective measure of comprehension too).

Figure 4. Example of output from a model of psychological determinants of the risk score during ‘Level 5’ public health restrictions based on SAM data collected from January to March 2021 (n = 4,000). The risk score is as defined in Figure 3. Standardised coefficients (ordinary least squares) estimate individual-level relationships between the relevant psychological scale and the risk score, controlling for socio-demographic background characteristics. Error bars are 95% confidence intervals.

Figure 4 is illustrative; multiple exercises of this sort were undertaken with the SAM data across the 18-month period and the drivers of behaviour changed somewhat at different stages of the pandemic.⁸ However, the SAM results changed policymakers’ perspectives and assumptions about why people were behaving as they were, providing the government in Ireland with a different picture of what was behind public behaviour than they were getting from the contemporaneous public debate. The fact that the data consistently showed how important it was for compliance that the overall range of public health restrictions was perceived as coherent informed the order with which restrictions were lifted during Summer 2021 and how this was communicated. The stronger influence of worry relative to fatigue gave the government confidence that the public would stay the course if the epidemiological situation warranted ongoing caution.

SAM data were also used to assist regulations and communications with industrial sectors. Responses provided information about which kinds of shops and workplaces were making greater efforts to provide safer environments. A specific example arose in relation to the mandatory checking of vaccine certificates in cafés, pubs and restaurants, which was introduced in late July 2021. Respondents who had eaten out were asked in the relevant survey module whether they had been subject to a certificate check. Figure 5 shows that initial compliance was fairly poor. After SAM revealed a still higher proportion of establishments not performing checks, industry representatives were publicly called into meetings with the government to address the issue. Non-compliance declined markedly thereafter.

Figure 5. Proportion of occasions that indoor diners reported not having COVID-19 vaccine certificates checked on the door of pubs, restaurants and cafés by month.

The findings shown in this section offer a selection of the sort of information that SAM supplied directly to the Department of the Taoiseach, which had overall responsibility for pandemic policy. Many other findings informed policy and government communications. One important element was myth-busting. For instance, the SAM data repeatedly demonstrated that non-compliance with restrictions and more risky social activity was not concentrated within specific socio-demographic groups, despite some public discussion that pointed fingers at young people, men and specific geographic locations. Another finding was the consistently higher price being paid by younger adults in terms of lower mental well-being while restrictions were in place. As the vaccination roll-out progressed, SAM provided detailed comparisons of the behaviours of vaccinated and unvaccinated people who were subject to different restrictions on activities. The detailed information on social activity by location was also used to develop a risk model to determine where rapid testing systems would generate the most benefit (Mallon et al., 2022).

Discussion

This paper demonstrates how a study based on an adaptation of the DRM provided behavioural evidence for policy in Ireland during the pandemic. As researchers who worked with data both from SAM and from Ireland's COVID-19 tracking survey, we found that the technique generated more granular and flexible measures of behaviour, allowing us to address more research questions. Findings were fed directly into pandemic policy development and government communications. The behavioural measures lent themselves also to modelling the psychological determinants of protective behaviours in real time, which helped the government to promote compliance with public health rules and to design restrictions and advice, such that they were more likely to be followed. We hope that others might see merit in the approach if an emergency policy response requires adaptation of everyday behaviours in the future.

A further element of SAM not yet mentioned was the prominence that the results received in the Irish media. Every one of the 36 waves of data collection led to newspaper coverage and many of them were covered also on television and radio. This was important in the context of the government's efforts to co-ordinate collective action through voluntary guidance as well as enforceable regulations. From time to time, the media focused heavily on isolated incidents of non-compliance, involving parties, drunken street scenes and prominent public figures caught breaking restrictions and/or guidance. Such incidents threatened to undermine the collective effort, in line with behavioural research on conditional co-operation. Indeed, the SAM data showed that a large majority of people believed themselves to be more compliant than most others. However, media reports of the SAM results consistently communicated to people that the majority of their fellow citizens were overwhelmingly compliant and cautious in their behaviour, helping to reinforce the collective approach. A particular example of this was communicating the finding during the vaccine roll-out that many people were waiting for their vaccine before beginning to increase their social activity again.

While the BRU conducts almost all of its work in collaboration with policymakers, in over 10 years of the unit's operation, this collaboration was easily the closest. In addition to fortnightly briefings, there were frequent communications between researchers, senior officials and, on occasion, politicians. These often involved providing answers to specific research questions ahead of a decision or checking exactly what could be said in public in order to keep faithful to the data. SAM data also featured in letters to the government containing advice from Ireland's Chief Medical Officer. Our perspective, as researchers, was that the evidence provided by SAM was highly beneficial, not least because the findings frequently challenged beliefs based on individual perceptions or anecdotes. On the downside, the study placed the unit's researchers under substantially more stress and time pressure than usual for an extended period. Moreover, given the time pressure to produce results every 2 weeks, the researchers were not able to follow open science practices that are standard for projects in the BRU, including pre-registering hypotheses and analysis plans. Hence, analyses were mostly exploratory rather than confirmatory and the research questions addressed were those thought to be important by the researchers and policymakers involved.

This collaboration also offered us a perspective on broader issues in behavioural public policy. The pandemic understandably generated debates about the appropriate use of behavioural science in informing policy (e.g. Harvey, 2020; Ruggeri et al., 2022), with some authors explicitly cautioning policymakers against reliance on behavioural science (IJzerman et al., 2020). As a behavioural research team working directly with central government, we observed some disjunction between these debates and the reality of undertaking behavioural science for pandemic policy. Firstly, while much debate focuses on whether behavioural evidence is strong enough to underpin specific interventions, often of the ‘nudge’ type, our experience during the pandemic was that the primary contribution we made as behavioural scientists was not to devise interventions but to provide good measurement of behavioural phenomena. This consisted not only of adapting the DRM but of deciding what else to measure based on the literature on collective action and risk perception. Although the research team undertook multiple other studies on COVID-19, including tests of behavioural interventions, many were diagnostic studies that sought to measure relevant phenomena (Robertson et al., 2021; Lunn and Timmons, 2023), of which SAM was the largest but not the sole example. Our experience was that such studies played a greater role in informing policy, often because they questioned influential assumptions and opinions that we could show were not matched by evidence. Secondly, much of the debate either knowingly or unknowingly adopts a ‘supply and demand’ type framework that views researchers as suppliers of expert evidence for use by policymakers. Our experience with SAM underscores findings in the public administration literature (e.g. Best and Holmes, 2010) that the effective use of evidence for policy depends on a closer relationship between researchers and policymakers. In particular, where there is a two-way system for sharing information and perspectives, our experience with SAM (and other studies) is that behavioural science is likely to make a much more telling and useful contribution.