Crisis as driver of digital transformation? Scottish local governments’ response to COVID-19

Abstract

The response to the COVID-19 pandemic has, from the outset, been characterized by a strong focus on real-time data intelligence and the use of data-driven technologies. Against this backdrop, this article investigates the impacts of the pandemic on Scottish local government’s data practices and, in turn, whether the crisis acted as a driver for digital transformation. Mobilizing the literatures on digital government transformation, and on the impacts of crises on public administrations, the article provides insights into the dynamics of digital transformation during a heightened period of acute demands on the public sector. The research evidences an intensification of public sector data use and sharing in Scottish local authorities, with focus on health-related data and the integration of existing datasets to gather local intelligence. The research reveals significant changes related to the technical and social systems of local government organizations. These include the repurposing and adoption of information systems, the acceleration of inter and intraorganizational data sharing processes, as well as changes in ways of working and in attitudes toward data sharing and collaborations. Drawing on these findings, the article highlights the importance of identifying and articulating specific data needs in relation to concrete policy questions in order to render digital transformation relevant and effective. The article also points to the need of addressing the persistent systemic challenges underlying public sector data engagement through, on one hand, sustained investment in data capabilities and infrastructures and, on the other, support for cross-organizational collaborative spaces and networks.

Policy Significance Statement

The responses to the COVID-19 pandemic exposed the significant opportunities as well as persistent challenges associated with data sharing and (re)use within the public sector. By examining Scottish local government’s data engagement in response to the crisis, this article offers useful insights on the dynamics of digital transformation in the public sector. In particular, the article points to the importance of identifying and articulating specific data needs in relation to concrete policy questions in achieving digital transformation. The article also highlights the need for long-term investment in data capabilities and infrastructures and the value of cross-organizational data collaborations and networks of data practitioners to support and sustain change in local governments’ data engagement.

1. Introduction

The response to the COVID-19 pandemic has been characterized by a strong focus on data intelligence and data-driven technologies (Ada Lovelace Institute, 2020; Frith and Saker, 2020; Leszczynski and Zook, 2020; Lyons and Lăzăroiu, 2020). From the outset, near-real data intelligence derived for example from contact tracing applications, urban sensors, and so forth, were seen as key to understanding the unfolding of the pandemic, informing public health policy, and monitoring socioeconomic measures introduced to manage the crisis. At the same time, the increase in data collection, use, and sharing (especially health-related) raised concerns about privacy, ethics, surveillance, and the potential for data harms (Frith and Saker, 2020; Newlands et al., 2020; Eom and Lee, 2022).

Emerging research indicates that data engagement formed a significant part of local governments’ initial response to the crisis (e.g., Cretu, 2020; CDEI, 2021; OECD, 2021). For example, local authorities developed dashboards displaying real-time data on local infection rates, integrated various datasets to identify patterns of vulnerability in the community and used data to monitor and tailor the delivery of essential services (CDEI, 2021; Gangneux and Joss, 2021). This engagement, however, occurred within the context of long-standing funding cuts (Lowndes and Garner, 2016) and significant preexisting organizational barriers, including insufficient data-related skills, gaps in data capabilities (e.g., infrastructures, processing, and analytics), inconsistent data quality, complex data governance mechanisms (e.g., data protection, privacy, and security) and data siloes (Giest, 2017; Jaansen et al., 2017; Klienvik et al., 2017; van Ooijen et al., 2019). Unsurprisingly, given these barriers, research conducted prior to the pandemic found that data practices in local public administrations were uneven, and the contribution of data intelligence to policy- and decision-making often limited (Malomo and Sena, 2017; Durrant et al., 2018; Bright et al., 2019; OECD, 2021).

Against this background, this article seeks to examine—with the example of Scotland, where digital transformation has been national policy for several years—whether local governments responded to the COVID-19 pandemic with increased data engagement and, in turn, how the crisis may have triggered digital transformation. We hypothesize that the pandemic, as crisis, potentially acted as a driver of digital transformation by increasing demands for data intelligence, encouraging greater data sharing and collaboration and, conversely, lowering intra and interorganizational barriers. Since the pandemic unfolded suddenly, requiring rapid action from (local) government, it provides a unique opportunity to analyze digital transformation processes within a heightened time period. Consequently, the analysis should provide useful insights into what are likely organizational enablers and barriers when local government (and other public sector organizations) pursues digital transformation. Our study used a mixed-method approach to obtain information from, and the views of, local government officers as well as representatives of public and third sector organizations collaborating with local government. Underlying the research were three main questions: (RQ1) What importance has data played in Scottish local government’s response to COVID-19? (RQ2) What organizational changes, in both technical and social systems, have occurred in response to the crisis? (RQ3) What do local government officers, and wider stakeholders, consider key enablers of, and barriers to, sustaining digital transformation beyond the initial stages of crisis?

The research is situated within the literature on ICT-enabled organizational change and digital transformation in public organizations. The public sector has long been the focus of reforms designed to harness digital technologies and data to enhance public sector efficiency and improve public service delivery (Dunleavy et al., 2006; Tassabehji et al., 2016; Vial, 2019; Liva et al., 2020). Overall, the literature points to (relative) progress on the digitalization of public services and processes—that is, at the technical systems level—while finding less evidence of wider changes associated with organizational culture and structures, at the social systems level (Mergel et al., 2019; Tangi et al., 2020, 2021). The article brings this body of work together with the literature on crises and their impacts on public organizations (Boin et al., 2016; Christensen et al., 2016; Tokakis et al., 2019). The latter provides mixed evidence of how crises affect public sector organizations, pointing particularly to leadership, and preexisting organizational arrangements, as key factors that determine whether public organizations manage to take advantage of the conditions generated by crises.

Following the literature review in the next section, Section 3 sets out the methodology, including the choice of Scottish local government as case study and the mixed methods of data collection and analysis. Section 4 then presents the findings in three main parts. This is followed by Section 5, which discusses the findings in relation to the wider literature, and Section 6, which identifies lessons for policy and practice.

2. Crisis as Possible Driver of Digital Transformation in the Public Sector

2.1. Partial digital governmental transformation

ICT-enabled organizational reforms of the public sector have had a long history in the UK (as elsewhere), closely connected to New Public Management goals of efficiency, target delivery, outsourcing, and competition (Dunleavy et al., 2006; Tassabehji et al., 2016). These reforms have been discussed using several overlapping frameworks, including “e-government,” “digital government,” “digital era governance,” and more recently, “transformational government” and “digital government transformation” (Dunleavy et al., 2006; King and Cotterill, 2007; Meijer and Bekkers, 2015; Vial, 2019; Liva et al., 2020; Tangi et al., 2021). Notably, ICT-enabled organizational change has been discussed from a sociotechnical perspective that understands organizations as complex sociotechnical systems (see Sawyer and Jarrahi, 2014). The technical system comprises “processes, tasks, and technologies needed to transform input into output” (Nograsek and Vintar, 2014, p. 110), while the social system encompasses people, structure, and culture (Nograsek and Vintar, 2014, p. 110). For organizations to change, transformation must occur within both systems (Nograsek and Vintar, 2014; Weerakkody et al., 2016; Faro et al., 2021; Tangi et al., 2021). For example, in their empirical analysis of Italian public sector organizations, Tangi et al. (2021) showed that, while digital technologies enabled the transformation of the technical system, the corresponding social system was left largely unaffected. Digital transformation, thus, needs to go beyond the mere digitalization of services and processes to include structural and cultural organizational changes (Weerakkody et al., 2016; Mergel et al., 2019; Tangi et al., 2021). Following this literature, this article understands local government’s data engagement during the pandemic as embedded in broader processes of digital transformation (whether successful or not) at both technical and social system levels.

The literature identifies several challenges and barriers to public sector digital transformation, which broadly map onto the social–technical perspective of organizations. Technical barriers include the complexity and incompatibility of technologies, security threats, lack of and outsourcing of infrastructural facilities, lack of technical standards and interoperability, and system legacies. Barriers related to organizational structures encompass lack of managerial and/or political support, lack of skills and knowledge, lack of available resources (e.g., funding and staff), organizational complexity and fragmentation, while cultural barriers include resistance to change, bureaucratic culture, and risk aversion (Gil-García and Pardo, 2005; van Veenstra et al., 2011; Meijer, 2015; Barcevičius et al., 2019; Wilson and Mergel, 2022). The literature, furthermore, points to external factors that can drive, or impede, innovation and change, including policies and regulations, collaborations and engagement in interorganizational networks, and pressure from external stakeholders (Cinar et al., 2019; Tangi et al., 2020). Crises, as part of the external environment in which organizations operate, has been identified as a potential driver of digital government transformation (Pittaway and Montazemi, 2020; Tangi et al., 2021; Eom and Lee, 2022), as the following section elaborates.

2.2. Mixed evidence of the impacts of crises on public organizations

Crises are characterized by three key components: threat, uncertainty, and urgency (Boin et al., 2007, p. 43). Importantly, they need to be understood as extended periods and not just limited to a triggering event (Boin et al., 2007; Williams et al., 2017). The literature provides mixed evidence of the impacts of crises on organizational change. On one hand, crises are often viewed as external shocks that generate “windows of opportunity” for organizational reforms as well as wider policy change (Boin et al., 2007). Accordingly, as disruptions to normal routines and practices, crises demand novel and rapid responses which, in turn, can foster innovation (Bessant et al., 2012; Avelino et al., 2014; Gkeredakis et al., 2021; Oborn et al., 2021). Orlikowski and Scott (2021), for example, argue that crises create opportunities to engage in “liminal innovation,” by leveraging the tensions arising from disruptions to improve established organizational practices and processes. This may include the use of iterative experiments, open-ended sense-making and alternative ways of doing things (Orlikowski and Scott (2021)). Drawing on this literature, emerging research documents how the COVID-19 pandemic has reconfigured ways of working and generated the conditions for innovation, experimentation, and collaborations (Mazzucato and Kattel, 2020; Gkeredakis et al., 2021; Orlikowski and Scott, 2021).

On the other hand, the notion of crisis as window of opportunity has been criticized with the literature pointing to the complex dynamics at play in relation to the transformative potential of crises in public administrations (Boin et al., 2007, 2009). The organizational capacities to transform and adapt in response to crises depend on factors such as leadership, communication, and legitimacy, as well as on preexisting organizational arrangements and culture (Deverell and Olsson, 2010; Boin et al., 2016; Christensen et al., 2016; Tokakis et al., 2019). Furthermore, crises often generate “political framing contests” in which various actors seek to “interpret events and the responsibilities and lessons involved in ways that suit their political purposes and visions of future policy directions” (Boin et al., 2009, p. 82). These discursive framings can create competing pressures for continuity and change (Boin et al., 2009). Thus, whether crises are “triggers of systemic change, or if they serve to forestall such change” remains an “open question” (Boin et al., 2007, p. 52). This debate aligns with the question of incremental versus radical change in the wider public policy and organizational change literature (Kuipers et al., 2014; Van der Heiljman and Kuhlmann, 2017).

Emerging research findings on the impacts of the COVID-19 pandemic point to accelerated digitalization in public sector organizations, particularly related to the needs to rapidly provide public services digitally and to the move to remote working (Agostino et al., 2021; Aristovnik et al., 2021). At the same time, the effects the crisis has had on, and the challenges it poses to, broader organizational processes associated with digital transformation are not yet clear (Gabryelczyk, 2020; Eom and Lee, 2022). In response, this article examines local governments’ data engagement in the face of the crisis and analyses whether this led to organizational changes in both technical and social systems (see research questions, Section 1). In doing so, the article contributes to the analysis of the dynamics (enablers and barriers) of digital government transformation, and the potential role of crises as enabler of systemic, sociotechnical change in public sector organizations.

3. Methods and Data

3.1. A case study of Scottish local government

This research focuses on local government since it was on the frontline of efforts to combat the COVID-19 pandemic and experienced heightened demands for data intelligence. As such, it lends itself to studying how the crisis affected data engagement in a public sector setting and what, if any, digital transformations emerged from this. Scotland was chosen as case study for three reasons: first, the digital transformation of the public sector has been a key policy of the Scottish government for over a decade (see below). Second, as a relatively small nation (5.5 m inhabitants), Scotland consists of 32 local authorities which, for the purpose of systematic research, is methodologically manageable. Third, in practical terms, an opportunity arose to implement the research with support from the Digital Office for Scottish Local Government (henceforth “Digital Office”), thereby helping to ensure access to local authorities and other public sector organizations and, afterwards, to disseminate the research results and, thus, inform ongoing data practices.

The Scottish government published its original digital strategy “Scotland’s digital future” in 2011 (updated in 2017), followed by the publication of the Open Data Strategy in 2015 (Scot Gov, 2011, 2015). This translated into several initiatives to support digital transformation in local government, most notably the establishment in 2016 of the Scottish Local Government Digital Transformation Partnership involving all Scottish local authorities. From this emerged the Digital Office which provides support to councils in four areas of digital transformation: digital leadership; digital foundations; digital services; and digital telecare. On its part, the Improvement Service, the national improvement organization for Scottish local government, also supports local authorities in their digital transformation, particularly relating to the use of spatial data. Additionally, the Scottish Cities Alliance (SCA), jointly established by the seven Scottish cities and the Scottish Government in 2011 to advance the Agenda for Cities (Scot Gov, 2016), includes in its smart city programme collaborative projects on open data, smart communities, smart services, and smart infrastructures. Early 2021, during the pandemic period, the Scottish government published its latest digital strategy, “A changing nation: how Scotland will thrive in a digital world” (Scot Gov, 2021). The strategy was jointly developed with the Convention of Scottish Local Authorities (COSLA), the Digital Office, and the Improvement Service, thus underscoring the importance attached to local government in the pursuit of digital transformation in the public sector. The document refers to the pandemic as a key moment that demonstrated the importance of good data, data intelligence and digital transformation (Scot Gov, 2021, pp. 91–92). The strategy puts forward several practice-oriented recommendations, including: building common platforms across the public sector; developing and accelerating the adoption of common digital and data standards; fostering greater collaboration; securing data sharing within and across organizations; and implementing a Data Transformation Framework to ensure data quality and encourage the expansion of data reuse across the public sector (Scot Gov, 2021).

3.2. Mixed method approach

The research was conducted during the height of the pandemic, under a national lockdown. In line with institutional ethical guidance, a decision was taken not to conduct research with frontline staff directly involved in public service delivery, given the added pressures under which they operated. Instead, staff with more strategic or coordinating roles was invited to participate. This had the benefit that they had a good overview and understanding of their typically large and complex local government organizations. Research participants were selected based on their active involvement in data use/management (“data cohort”) and pandemic response (“recovery cohort”). They, thus, provided situated information and perspectives on local government’s data-related responses to the crisis. As such, however, their responses do not represent the full range of experiences and viewpoints from across the multidepartmental local government organizations. Finally, it should be noted that participants were invited to engage in the research in an individual professional capacity. Their responses, therefore, should not be read as official viewpoints of the local authorities concerned.

The research used a mixed method approach starting with a survey of all 32 local authorities and followed by three focus groups and five semistructured interviews. The fieldwork, for which institutional ethical approval was obtained, was conducted between October 2020 and April 2021.The Digital Office provided practical assistance, including recommending the survey to the local authorities and facilitating contacts with potential participants for the focus groups. They also provided policy- and practice-relevant feedback on the draft survey questionnaire and supported the dissemination of results. Responsibility for the research (design, ethics, implementation, and analysis), however, rested solely with the authors.

3.2.1. Survey

The survey was designed based on (a) a review of relevant academic and policy literatures, and (b) advice from data scientists, public policy researchers and Digital Office staff. The survey was piloted with six volunteers and adjusted. The final version contained 20 questions and was structured around six main themes: “General data response to COVID-19”; “Data uses and needs” (including specific data types, sources, policy domains); “Data capabilities” (incl. gaps and opportunities); “Data sharing and collaborations”; “Data opportunities and challenges”; and “The role of data in the context of COVID-19.” The full questionnaire is available in https://researchdata.gla.ac.uk/1147/?_ga=2.12681719.1187732427.1657116747-1022996968.1651148229. The rationale was to include all Scottish local authorities (32) and in each authority capture information from, on one hand, a staff with an experienced role in data management (digital services, business intelligence, innovation, and research) and, on the other, a staff with a central role in COVID-19 recovery planning. The sample, therefore, comprised 64 local government officers: 32 in the “data” cohort, and 32 in the “recovery” cohort (the questionnaire contained questions common to both cohorts, as well as specific to each, see questionnaire). The officers were identified from the database of the Digital Office as local authority personnel involved in the Scottish digital transformation partnership (data cohort) and COVID-19 response (recovery cohort). To safeguard anonymity, the names were withheld from the researchers (the Digital Office sent out the survey invitation on behalf of the researchers). The survey response rate was 70.3%, with 31 local authorities (96.8%) participating (see Table 1). Thus, it was above average of expected response rates in organizational research (Holtom et al., 2022).

Table 1. Survey responses per cohort

3.2.2. Focus groups

Three separate focus groups (FG) were held with local government officers, representatives of public sector organizations, and representatives of third sector organizations (see Table 2). FG1 brought together officers from five local authorities (different sizes, geographic locations, and urban–rural makeup), to elicit qualitative information and in-depth discussion about local authorities’ data needs, capabilities, and uses resulting from COVID-19. Participants were drawn from the survey sample. The other two focus groups were designed to elicit information and discussion about data sharing practices and collaborations between local authorities and public sector organizations (FG2), and between local authorities and the third and voluntary sector (FG3). Participants for FG2 were recruited with the help of the Digital Office; they included, alongside three local government officers, four staff with strategic and/or coordinatory roles in their respective public sector organizations (government, police, health service). Participants for FG3 were recruited through desk research; they included, alongside two local government officers, six staff in voluntary and community organizations that participated in the pandemic response and recovery either by supporting local communities directly (e.g., food community charities) or coordinating voluntary organizations at regional or national levels (e.g., SCVO and SCDC).

Table 2. Focus groups participants

Each focus group started with an open prompt inviting participants to reflect on their organizations’ overall data response to COVID-19 and ended by asking participants to discuss emergent opportunities for data engagement and digital transformation. This common structure was used to allow for comparison across the three focus groups. Additionally, FG1 included two bespoke discussion prompts, on types of data used to respond to COVID-19, and on challenges related to data use and capabilities. FG2 and FG3 also encompassed two additional prompts; one concerning the accessing and sharing of external public sector data (FG2) and third sector data (FG3), and the other concerning data-related collaborations between local authorities and their respective sector.

3.2.3. Semistructured interviews

Expert interviews were arranged with four organizations that are active in the field of digital transformation at Scottish national level: the Digital Office; the Improvement Service; the Scottish Cities Alliance; and the Society for Innovation, Technology and Modernisation (SOCITM). The interviews were designed to elicit information about interorganizational data collaborations across Scotland. Discussions were structured around the role of the organization in supporting data collaborations across local government, the impacts of the pandemic on local authorities’ data practices, and main challenges faced and opportunities arising.

As the research was undertaken in the first year of the pandemic, it provided an opportunity to study the effects of the unfolding crisis on Scottish local government’s data engagement. Accordingly, survey participants were prompted, in response to questions about the relative importance of different data types (public sector; third sector; private sector; and smart data), to consider three different phases of the crisis: in the beginning (March 2020); at the time of the survey (October–November 2020); and projecting forward to the “coming months” (Spring 2021). The focus groups and interviews, too, were designed to prompt participants to reflect on their (organizations’) experiences during different phases and discuss their expectations for the near future.

3.3. Data analysis

The quantitative data collected from the survey was aggregated, using descriptive statistics, and analyzed to identify key themes. On this basis, the survey findings initially resulted in five main categories:

1. 1. Importance of public sector data;

2. 2. Intensification of data use mainly to deliver essential services;

3. 3. Key challenges of data quality and data harmonization;

4. 4. Significance of third sector data (albeit underutilized)/missed opportunity of private sector data use;

5. 5. Limited use of novel data.

The data from the focus groups and interviews were transcribed and coded in NVivo, using an iterative thematic analysis approach: the coding matrix was first developed based on the initial five thematic categories (see above), following which it was refined based on the summary notes of each focus groups and interviews. This resulted in five additional themes as follows:

1. 1. Interorganizational governance;

2. 2. Intraorganizational complexity and legacy systems;

3. 3. Data ethics, security, and public trust;

4. 4. Opening of conversations and change of perspectives in relation to data;

5. 5. Defining and aligning the purpose of data use/collaborations with existing needs.

Each theme emerged from the clustering of coding nodes across the datasets. For example, theme 2 included the following nodes: “increased data use,” “high pace response,” “accelerated data sharing and collaborations,” “focus on essential services,” “operational needs,” “initial response,” and “collection of new types of data.” Analytical and visualization tools in NVivo and Microsoft Excel (crosstab and frequency queries, bar charts, sunbursts, tree maps, etc.) were used to identify trends and patterns across the quantitative and qualitative datasets. Methodological triangulation (Flick, 2007) was applied to ensure the robustness of the research and cross-examine the validity of findings. This included checking emerging themes across datasets, comparing and ensuring consistency of research notes, independently reviewing survey data aggregation and analysis, documenting the development of the coding matrix over time, and presenting methodology and initial analysis of findings to colleagues and stakeholders.

4. Findings

4.1. Crisis response: Intensification of public sector data use and sharing

A key finding of this research is that Scottish local government’s data engagement in the initial phases of the COVID-19 crisis was predominantly centered on public sector data. Survey respondents identified internal and external public sector data as the most important types of data during the first month of the lockdown (89 and 67%, respectively) and in the period October–November 2020 (80 and 76%). Third sector data (e.g., data on food and medicine distribution and numbers of volunteers) was also seen as important, with two thirds of respondents rating it “quite/very important” in the same periods. However, only 44% of survey respondents reported using it, thus highlighting a discrepancy between perceived importance and actual use. Strikingly, respondents rated the importance of public sector data far higher than that of private sector data and novel data (e.g., cellular, crowdsourced, and sensor data). Private sector data was considered by a significant proportion of respondents as either “not at all important/fairly unimportant” (31% concerning the first month of lockdown; 20% concerning the period October–November 2020) or “neither important nor unimportant” (38 and 31%, respectively). In addition, participants reported relatively low usage of private sector data (27%; survey, Q. 11). The findings on novel types of data are similar, with two thirds of respondents rating them as “neither important nor unimportant” or “not at all important/fairly unimportant” in both periods. The reported use of novel types of data was limited, namely in descending order: social media data (24%), data from connected infrastructures (18%), cellular and Wi-Fi data (4%), and crowdsourced data (4%). Significantly, 36% of respondents reported that their local authority had not used any (survey Q. 15).

The prevalence of public sector data (internal and external) is in large parts explained by the public health nature of the crisis and the associated responsibilities of central and local governments and health boards. The survey data confirms that local government experienced an acute need for data concerning essential public services, including adult social care, education, and children services. Data needs also increased in the economic domain, in relation to business and corporate services and emergency support for local businesses (see Table 3). Data needs were made more acute by new data requirements prompted by the Scottish government and COSLA, which requested regular data returns from local government. Local government participants described these requirements as an onerous challenge, especially at the beginning of the crisis.

Table 3. Ranking of policy areas with increased data needs resulting from COVID-19 (survey Q.7, both cohorts)

The need for rapid data intelligence was met by both an intensification of data use within local government and an increase of data sharing across the public sector. Public sector data used by local government included various operational and administrative data on: the uptake of public services (e.g., adult social care data and customers contacts data); social benefits recipients (e.g., free school meals and housing benefits); local government employees (to facilitate staff redeployment); education (e.g., teachers’ absence and infection outbreaks in schools); and local businesses (e.g., business grant and rate relief data). Local authorities also used external public sector data, notably personal data provided by the NHS to local government on vulnerable people required to shield (self-isolate) and data related to the spread of the pandemic (e.g., local infection and death rates).

Seventy percent of respondents reported an increase in the integration and analysis of existing internal data, and 83% an increase in internal data sharing. This occurred alongside the acquisition of new data, with 78% of respondents reporting using new data sources and 74% indicating an increase in data collection (see Table 4). As one focus group participant (local authority) put it, “our use of data almost from day one really went through the roof.” Local government’s main data sources were other public sector organizations, including NHS Health Boards (93%), the Scottish Government and its agencies (78%), and Special NHS Boards (67%) (Survey Q. 10, both cohorts). Relatedly, 70% of respondents described an increase in data sharing externally and 65% reported the implementation of data sharing agreements in response to COVID-19 (see Table 4).

Table 4. Local government’s data response to COVID-19 (survey Q. 6, data cohort)

Local government’s response in the initial stages of the crisis was focused on using data to identify and support vulnerable groups, as well as monitoring and adapting service provisions:

Initially, we needed to get off-the-ground data in terms of how did we record what people were requiring. Obviously, it was the responsibility of the local authority to set up a helpline, so then it became what did we record on that helpline to enable us to provide the services that were required? Particularly in relation to food, and pharmacy and medicine, and any other needs. (Focus group, local authority)

Across all datasets, participants spoke of the rapid and collective efforts put in place across organizational boundaries to use and share data to identify and support vulnerable groups. While, however, there was an alignment of purpose, the initial period was marked by a time-consuming case-by-case approach to developing data sharing agreements between local government and NHS health boards concerning shielding data. As one participant recounted:

At the start of the pandemic, a lot of work went into establishing the data sharing to allow the NHS to tell local government who were being shielded […] The sharing agreements have been very point to point, as you’d expect, and that requires a lot of effort because every time you got a new requirement, a new DPIA (Data Protection Impact Assessment) [Data Protection Impact Assessment], you had to do a new data sharing agreement etc. (Focus group, local authority)

Furthermore, local governments had to meet new data reporting requirements by the Scottish Government and COSLA, which created additional challenges. As one participant described, “it was new data, it was different data and we did not have the systems to handle this data” (Focus group, local authority). Another explained that the key challenges in the initial response to the crisis were the “rapidity that people required data in, how quickly they needed it” as well as “the transition from manual things” to more systematic ways of organizing the delivery of essential services (Focus group, local authority). To meet these challenges, local government had to develop processes to match and integrate existing datasets with newly acquired shielding data, subsequently expanding to other national datasets:

There were lots of different datasets coming at us about the nature of the pandemic in our area. Some of that was open data that we were consuming; some of it was local datasets; some of it was the Public Health Scotland test and protect data at different levels of sort of anonymisation. And we pulled that into a dashboard hub of the corporate view of [Council] in terms of trying to understand where the epidemic was playing out and what kinds of impacts that was having. (Focus group, local authority)

The integration of different datasets, then, was often accompanied by the development of dashboards to facilitate day-to-day operational decision-making and strategic leadership. Dashboards were used in different areas, for example, to monitor COVID-19 local infections, school absences and educational capacity, local economic and business impacts, and to support the redeployment of staff. The survey data confirms a significant increase in the use of data visualization tools (see Table 4).

Altogether, in response to the first research question (RQ1), these findings confirm the important role data played during the initial stages of the pandemic: local government experienced an intensive demand for data matched by new data reporting requirements, especially related to essential services. Significantly, the increased data engagement was centered mainly on public sector data (internal and external) and to a lesser extent on third sector data, with private and novel data playing only minor roles. This can be explained by the fact that the crisis was centered on health and health-related data. Private sector and novel data were both perceived by respondents to acquire a growing importance in the future (69 and 62% of survey respondents rated private sector and novel data respectively as quite/very important’ for the third period).

4.2. Changes in socio-technical systems: Shortening of information sharing processes and development of new ways of working

Turning to RQ2, the findings point to several organizational changes, in response to increased data demands, at both technical and social system levels within local government, as well as in the interactions with other public sector organizations. Changes were particularly apparent in relation to information systems and agreements to share data within and across organizational boundaries. Participants noted a significant acceleration of processes related to data sharing and governance, but also related to more technical aspects, such as data cleansing and integration:

Previously anything to do with data sharing, and obviously particularly to do with sensitive data, has often taken significant amounts of time to overcome various governance barriers. Certainly, from my direct experience and from speaking to others it has massively shortened the amount of time needed to get certain bits of information together, to gather data, to collate data together. (Interview)

Another participant recalled how they were now able to “turn something around in a few days that normally would have taken about 4 or 5 months” (Focus group, local authority). The rapid demands for data also led to the adoption of new cross-organizational systems and agreements, including the adoption of a new Data Sharing Framework between thirteen local authorities and NHS NSS, and the development of a Local Government Data Portal to deposit regular COVID-19 data returns to Scottish Government and other regulatory bodies (Digital Office, 2020). Another example of changes to technical systems was the adoption of new, or the repurposing of existing, internal information systems, particularly Customer Relationship Management Systems (CRM):

We had started a process of standing up a CRM for us, so again, what we did was we accelerated some of that in relation to developing with Agilisys Helping Hands, which allowed us to then do some of the monitoring for the shielded category and the shielded list and the data that was coming in for shielded. And then allowed us to turn that round and report back out externally. (Focus group, local authority)

Other participants reported the deployment of systems and solutions to manage helplines and customers’ contacts, for example Oracle Service and Verint CRM.

The research also sheds light on changes more closely connected to social systems. This included the development and strengthening of cross-organizational networks and collaborations. Participants spoke about an increased openness to collaborate and share data:

Certainly, from our perspective, COVID pushed forward the whole issue about collaboration. And I think there’s been more openness in sharing than we probably had previously and our willingness to actually sit round a table and share and discuss. (Focus group, local authority)

Another participant explained that prior to the crisis data sharing used to be seen as something very difficult to achieve, whereas “now it is seen as something that we just work through and get done” (Interview). Collaborating was viewed as a pragmatic and efficient way of rapidly developing complex data sharing agreements and avoiding the duplication of efforts. One participant recalled how stakeholders from different local authorities joined forces to develop a data sharing agreement template that could be shared with others:

You know the work that they [other local authority] were doing with NHS NSS […] We ended up doing the data share agreement at the same time and we worked with their Information Governance Unit [of other local authority], so there’s been a lot more sharing of data, sharing where we actually turn round and say, we’ve got the template, does everybody agree it? Yeah, that’s fine, let’s all just sign it. (Interview)

Apart from the practical benefits, collaborations were also welcomed more broadly as spaces for peer support and knowledge exchange across the public sector. As another participant explained:

It’s that shared learning environment, the shared experience, the shared support, learning ourselves about how data exists within other local authorities. So, we’re not specifically sharing data as such but were learning just about the construct of the data. (Focus group, local authority)

Significantly, these social system-level changes were instrumental in facilitating changes at the technical systems level, thus highlighting the interrelationship between the two systems: the aforementioned Data Sharing Framework (btw. local government and NHS NSS) was led by the COVID-19 Data and Intelligence Network (Scottish government), and the Local Government Data Portal by the Digital Office’s Local Government Data Taskforce.

As to wider organizational culture, some participants also indicated a shift in the recognition of the value of data intelligence for decision-making. As one put it: “suddenly people recognize the value of data and the reason for having data and using it” (Focus group, local authority) while another emphasized that the pandemic had “moved the conversation about data” beyond data practitioners (Focus group, local authority). Another participant saw this shift among senior management and the political leadership: “[spatial data] is becoming really well regarded by the senior people who [had] perhaps thought [that] these geospatial things were, you know, nice to have, but they are now seeing the value of it” (Interview). One focus group highlighted the opportunity for data intelligence and data accessing across the local government organization, exemplified the successful implementation of dashboards and their adoption by the leadership:

There is a real desire to consume it [data]. And so, from our perspective what we have now sent round to our wider leadership team are weekly dashboards with data in them, partly compiled by my team. And that goes back to cases, absence and a year ago nobody would have ever wanted that. Nobody would’ve conceived that it would be useful, whereas we now go, oh, look, the dashboard’s in and open it. (Interview)

Other participants, however, argued that still more was needed to increase the recognition of the value of data as part of the wider organizational culture, especially from people “at the very top” (focus group, local authority). One commented that there was greater need for “a more open approach” to data sharing (Focus group, local authority). Survey respondents also pointed to the need for “greater awareness of the power and importance of data” and for recognition of “how powerful data can be” (Survey Q.19, recovery cohort). Altogether, this suggests that while there were some notable shifts in the organizational culture and social practices occurred, these changes were not evenly distributed and often seen as at risk of backsliding, as discussed further below.

4.3. Sustaining transformation: Addressing persistent challenges and nurturing collaborations

In response to RQ3, the findings provide insights into perceived enablers of, and barriers to, sustaining digital transformation beyond the initial stages of crisis. First, participants pointed out several persistent, preexisting challenges related to data engagement which the crisis had exposed and/or amplified. Inconsistent data quality, and the lack of data standards and common identifiers, were singled out as particularly problematic. Survey respondents rated “ensuring data quality/standards” and “integrating data” between 4 and 5 (on a scale 1–5, from “not at all challenging” to 5 “very challenging”). These two main challenges were followed by “identifying data needs,” “mapping existing internal data,” and “analyzing data” which received slightly lower ratings (see Figure 1).

Figure 1. Data related challenges experienced by local government (percentage, survey Q. 17, data cohort).

These challenges were especially felt in the initial stages of the crisis when local governments had to match and integrate data from their own organizations with shielding data from NHS health boards. Preexisting gaps in data infrastructures and skills were also felt acutely:

It was the lack of infrastructure and having to rely on very basic tools to process things rapidly that was a particular feature and focus, which also illuminated data literacy and skills issues along the way. (Focus group, local authority)

Participants also described significant challenges from multiple legacies of past and existing IT systems as well as from the organizational complexity and siloed nature of local government. Looking ahead, participants, therefore, pointed to the need for appropriate and long-term investment, associated with both the technical and social systems, to be able to address these persistent challenges and support local government in adopting data standards:

The immediate issues for us are to get investment, sustained investment, to standardise as much as possible, where viable, across our own organisation. But it’s just the cost of all this, and you have to be able to resource it, and that really is a major concern. (Interview)

Another participant spoke of the needs to invest in data literacy and related skills across local government to ensure that data is used in support of policy and decision-making:

I think, if we’re investing in data, and data platforms, we equally need to invest in, you know, being able to make sure that people who are making decisions, and policy decisions, and operational decisions, are able to use that data, and know how it can be used and manipulated. So, it’s not just a technology platform driven solution, but it actually impacts on policy. (Focus group, local authority)

A lack of sustained investment was raised as a major risk of reverting to precrisis practices, or leading to a new, but unsustainable situation where local government would be expected to maintain its data engagement without further resourcing. One participant described this as “trying to make the transformational change while also providing a business-as-usual service with a small team” (Interview). Some participants perceived the crisis as an opportunity to make a strong case for investment, directed at senior leadership. Participants also viewed the pandemic as a catalyst for change in local government’s data engagement and broader digital transformation. One participant described it as “a flash of data maturity modeling” (Focus group, local authority), while another argued that the pandemic had brought 10 years’ worth of digital transformation within local government (Interview). However, as reflected above, other participants were concerned that local government would “retreat back into business as usual after this and keep on doing things as we were” (Interview). The risks of reverting to pre-Covid status quo were seen as particularly acute with respect to data sharing and collaborative working across the public sector:

I’m sure some of them [barriers] were justified, but what I would be hoping is that we don’t regress, that we don’t go back to where we were previously, where it did take us six months or a year, or if we ever got it, to come to a collective agreement around a particular dataset or a particular data sharing. (Focus group, local authority)

Participants identified working collaboratively across organizational boundaries as an important factor to maintaining newly gained practices and transformation. Forty-three percent of survey respondents identified collaborating with other stakeholders to collect, use and/or analyze data as a significant opportunity (survey, Q. 18, data cohort), while a focus group participant stated that:

It’s what we can do together, as opposed to what we do individually, if we can build on the collaboration that we have had during this crisis and building some of that openness. (Focus group, local authority)

Participants also saw the sharing of learning and good practices as a particular benefit of collaborations which, in turn, would reduce unnecessary duplication of efforts. As one participant put it:

There’s all this learning from each other that we can have. Some people going through the pain of a variety of DPIAs or whatever, again, doing that collectively, having a way of shortcutting some of the issues for us would always be very helpful. (Focus group, local authority)

In this context, participants spoke to the importance of strengthening and developing cross and interorganizational “networks of data practitioners” (Focus group, local authority), to share learning and work collectively toward solutions. Finally, local government collaborations were also discussed as a means to gain collective bargaining power and negotiate better data accords and agreements with data owners, in particular private sector organizations, to access and reuse relevant data:

I do think there’s a real opportunity for authorities to look and come together as one, and say, where are the things that we could focus and do something together, and maybe chip in our pennies and have a much stronger position with private sector companies or private sector resources. (Focus group, local authority)

5. Discussion

Overall, our findings evidence a significant intensification of data engagement by Scottish local government in the early phases of the COVID-19 pandemic (RQ1). This resulted from rapidly increasing data needs and new reporting requirements in response to the unfolding public health crisis. Consequently, increased data use and sharing mainly involved public sector data and, conversely, did not significantly involve private sector and novel types of data. The research also shows that, due to the intensification of data engagement, several organizational changes in the technical and social systems of local government occurred (RQ2). Changes closely related to technical systems included the matching and integration of datasets, the adoption of dashboards and the repurposing of CRM systems. Changes related to the social systems included the collaborative development of data sharing agreements, new ways of working (e.g., collaborative networks and shared learning) as well as changes in perceptions related to data sharing and the role of data in decision-making. However, digital transformations were often hindered by preexisting data-related challenges and gaps in local government’s data infrastructures, processes, and capabilities. The crisis both exposed and further amplified these barriers. Notably, these include a lack of data standards and a standardized approach to data sharing, as well as patchy data quality, which together rendered the matching and integration of data within and between organizations complex and time-consuming. Addressing these challenges with long-term investment, and by building on cross-organizational collaborations, were identified as key to sustaining digital transformation over time (RQ3).

Our findings speak to the literature on digital government transformation by pinpointing organizational changes in both technical and social systems of organizations. The literature indicates that generally fewer changes occur at social system level—concerning organizational structures and cultures—compared with changes at technical system level, thus hampering full digital transformation (Nograsek and Vintar, 2014; Tangi et al., 2021). Our findings, in contrast, show that, at least during the initial phases of the COVID-19 crisis (Spring 2020 to Spring 2021), both the technical and social systems of local government organizations in Scotland experienced some significant changes. Indeed, by creating an urgent need for working across organizational boundaries to generate data intelligence, the crisis first prompted changes at social system level, especially through significantly accelerated data sharing processes and interorganizational collaborations. In turn, this prompted changes in the technical systems (e.g., development of a common platform to return data to central government and repurposing of CRM system). Our findings, thus, demonstrate the capacity of public sector organizations within the context of an external crisis to significantly adapt and change in a short time frame concurrently at social and technical system levels.

It is important, however, to acknowledge that, to be successful, digital transformation needs to be sustained over time, going beyond a temporary state created by crisis. While this research shows that the COVID-19 pandemic acted as a trigger for increased data engagement and organizational changes in the short-term, it is too soon to say whether this turn out to be long-lived. Here, our findings point to significant ambivalence: research participants welcomed the positive organizational changes achieved in the short term, but also raised significant concerns about the risks of reverting to a precrisis situation once the pandemic was in retreat. At national level, the publication of the Scottish government’s latest digital strategy a year into the pandemic (February 2021) shows a political framing of the crisis (Boin et al., 2009), that puts renewed strategic emphasis on digital transformation in the public sector.

The research provides lessons for digital transformation processes in local government, and more broadly in the public sector, beyond the initial crisis triggered by the pandemic. First, the findings show that local government’s increased data engagement and related organizational changes were driven by the emergence of specific data requirements (i.e., related to essential services and new requirements for health data reporting). This suggests that digital transformation may be more successful when focused on specific data needs and distinct areas of policy. Second, the findings highlight the essential role of interorganizational collaborative networks in creating, implementing, and adopting, new technical processes as well as in facilitating broader changes at social system level. Such networks have the potential to act as communities of practice (Wenger, 2010), in which practitioners can share their experience, generate collective knowledge, and discuss data-related solutions to specific operational needs and policy questions. Collaborative networks can also play a significant role in building collective bargaining power to negotiate with the private sector (Micheli, 2022) as well as in sharing leadership know-how on digital transformation in local government (Pittaway and Montazemi, 2020). Relatedly, third, our findings point to the importance of buy-in by senior management and political leadership to achieve sustained organizational changes. This requires active involvement, sufficient data literacy to engage with data, and a broader recognition of the role of data intelligence in policy- and decision-making. This aligns with other research findings on the key role of strategic leadership in effecting digital transformation in the public sector (Pittaway and Montazemi, 2020; Tangi et al., 2021), as well as in crisis management (Deverell and Olsson, 2010). Fourth, our findings emphasize the indispensable technical groundwork required to develop data sharing protocols and standards within the public sector. Related to this, the research exposes the need for long-term financial investment in local government’s data infrastructure (e.g., data system management, analytical tools, and support to adopt standards) and capacity development (e.g., staff, data-related training, and data literacy).

6. Conclusion

This article investigated how acute data needs and new data requirements that arose in the initial stages of the COVID-19 crisis impacted on Scottish local government’s data practices and, in turn, led to organizational changes at both technical and social system levels. Importantly, while demonstrating local government’s capacity for digital transformation, the crisis nevertheless exposed persistent challenges facing their data engagement that may hamper long-term transformation. Methodologically, the research centered on the views and experiences of local government and public sector officers who closely work with data or were involved in COVID-19 recovery planning. As such, while providing essential first-hand insights, it does not provide a full spectrum of views and experiences (including of frontline staff) from across the varied parts of local government organizations. Furthermore, as the research focused on the initial stages of the COVID-19 crisis and its unfolding until early 2021, it does not speak to the longer-term impacts of the crisis on local government’s digital transformation.

The article, while grounded in the COVID-19 pandemic, provides key insights on digital transformation relevant to local government—and by extension, other public sector organizations– as they move into recovery and prepare for future crises. Most notably, the findings reveal how transformation, both in the technical and social systems of organizations, crystallized around specific data needs and policy areas (i.e., the provision of essential services and support of vulnerable groups). Beyond the COVID-19 crisis, this points to the importance of identifying and articulating specific data needs related to concrete policy questions, to help ensure the effectiveness of digital transformation. The findings also highlight the need for sustained investment in local government’s data capabilities and infrastructures, in order to address persistent challenges, as well as the importance of developing and nurturing cross-organizational collaborative spaces and networks to support and sustain digital transformation over time.