Digital & data-driven transformations in governance: a landscape review

Abstract

Data for Policy (dataforpolicy.org), a global community, focuses on policy–data interactions by exploring how data can be used for policy in an ethical, responsible, and efficient manner. Within its journal, six focus areas, including Data for Policy Area 1: Digital & Data-driven Transformations in Governance, were established to delineate the evolving research landscape from the Data for Policy Conference series. This review addresses the absence of a formal conceptualization of digital and data-driven transformations in governance within this focus area. The paper achieves this by providing a working definition, mapping current research trends, and proposing a future research agenda centered on three core transformations: (1) public participation and collective intelligence; (2) relationships and organizations; and (3) open data and government. The paper outlines research questions and connects these transformations to related areas such as artificial intelligence (AI), sustainable smart cities, digital divide, data governance, co-production, and service quality. This contribution forms the foundational development of a research agenda for academics and practitioners engaged in or impacted by digital and data-driven transformations in policy and governance.

Policy Significance Statement

The landscape review established by the Data for Policy Conference series highlights the state-of-the-art research in the data for policy research, for example, the area of digital and data-driven transformations in governance as well as proposes future areas of interest for researchers and policymakers alike. In this context, the paper points toward policy-relevant research as well as highlights areas that will be a major component in decision-making with and on data. Specifically, the review highlights three areas: (1) public participation and collective intelligence, (2) transformation of relationships and organizations as well as (3) open data and government.

1. Introduction

Governments have always relied on data to run the state efficiently and effectively (Headrick, 2000; Higgs, 2001). This is a process that is now increasingly shaped by digital innovation enabling the collection and use of data at scale. These digital and data-driven transformations have required the development of new strategies for the use of data and digital technology to enhance public sector efficiency, improve service delivery, ensure equal access, aid decision-makers, and reduce costs. This integration has given rise to concepts like “Digital Weberianism” (Muellerleile and Robertson, 2018) and “Algocracies” (Lorenz et al., 2021), fostering the emergence of “Information States” (Braman, 2009) or “Dataist States” (Fourcade and Gordon, 2020) and where data and technology are increasingly inseparable from processes of government.

Normative expectations drive these changes; the digital and data-driven future for government is one that is effective and efficient, but also aligned with societal goals of transparency, accountability, and participation. The rationale is that data use and digital service development enhance decision-making processes, facilitate citizen participation, expand government service availability, and decrease operational costs. To facilitate these changes, new networks and forms of governance have been developed. These, in turn, have altered, businesses processes, decision-making, and the policy making process. What results is the creation of new power relationships and redesigned public sector institutions.

However, these changes also bring risks as the public sector becomes increasingly datafied and dependent on digital technologies. Real-world examples illustrate that automated decision-making tools can yield biased outcomes (Grimmelikhuijsen and Meijer, 2022), that reliance on data can lead to detachment between service users and service providers (Cuéllar, 2016; Fourcade and Gordon, 2020), and that digital and data-driven transformations may enable new privacy risks or opportunities for surveillance (Feldstein, 2019; Koenigs, 2022; Kosta, 2022).

While there is a large amount of research that addresses these issues, much of it remains fragmented and dispersed across disciplines, and what is needed is a comprehensive and transdisciplinary research agenda that explores the positive-potential of digital and data-driven transformations of governance, while also considering the negative effects (Engin et al., 2024).

This landscape review, developed as part of the Data for Policy Area 1: Digital & Data-Driven Transformations in Governance, helps to address this by exploring and mapping the area as part of the emerging Data for Policy research landscape and providing scholars a transdisciplinary opportunity to engage in discourse on the topic.

To achieve this goal, this paper is driven by three primary goals:

• • first, to conceptually clarify and set a strong foundation for the Data for Policy Focus Area of Digital & Data-Driven Transformations in Governance;

• • second, to provide an overview of the current state-of-the-art and future trajectories of research topics relevant to this strand of academic and practical inquiry;

• • third, to set and propose a future research agenda by identifying and outlining areas of research where there is a clear need for further development.

To achieve these goals, the paper integrates the specialized knowledge of the authors (also editors of the Area 1), the thematic orientation of the journal, and valuable insights from the wider community. Our analysis is based on a scoping review of papers selected for their relevance to the journal’s Area 1, and ensuring alignment with the current academic discourse. Additionally, we have incorporated feedback and contributions from the academic and professional community associated with the journal, enriching our study with diverse perspectives and expertise. This collaborative approach enables a nuanced exploration of the subject matter, reflecting both the depth of authorial expertise and the broader community’s insights. With this, the landscape review can used as a precursor to a systematic review and to develop targeted review questions. Ultimately, this paper provides a starting point for scholars, practitioners, or interested persons who are directly working with, researching, or influenced by data and its interactions or effects on policy and governance.

2. Mapping of topics, trends, and actors

This paper defines and understands digital and data-driven transformations in governance as approaches and endeavors that utilize data and digital technologies to enhance the efficacy and efficiency of the public sector. This involves improving service delivery, ensuring equitable access, providing support for decision makers, and implementing cost-cutting measures. The review identifies three main themes in the literature together in combination with research papers submitted to Area 1 of the Data and Policy Journal. These are (1) public participation and collective intelligence; (2) transformation of relationships and organizations; and (3) open data and government. In addition, to these three themes there are several consistent cross-cutting aspects, such as governmental capacity to address changes in both technologies and processes. Figure 1 summarizes the core themes.

Figure 1. Trends in digital and data-driven transformations in policy and governance.

2.1. Public participation and collective intelligence

The first thematic category is that of public participation and collective intelligence. Themes of interest within this category include, but are not limited to, digitally enabled public participation or crowdsourcing.

2.1.1. Digitally enabled public participation

New data-driven and digitized forms of public participation have emerged as a result of the increased availability of data and the digitalization of many government processes (Welch, 2012; Ingrams et al., 2019). This includes new methods for engaging citizens in the policy-making process (e.g., through the provision of digital platforms to solicit and respond to stakeholder feedback) (Tseng, 2023), data-driven approaches to service development, or the use of data to drive accountability and transparency (open government data, as an example). Many of these are discussed in more detail in future sections of this paper, further demonstrating this area of research’s interconnectedness.

In many cases, these methods for participation can be further subdivided into ‘passive’ and ‘active’ forms (Grêt-Regamey et al., 2021). In the case of the latter, one would see citizens being directly involved and engaged in new data-driven initiatives, for example providing feedback to a proposed piece of legislation during an open review process. Whereas the former would see citizens only playing a more passive role, for example, by using a sensor or app on their phone to generate data for a public sector organization on, for example, air quality (e.g., citizen science projects).

In addition, research highlights that there are pressing questions around equity as studies show a clear relationship between socioeconomic status and participation (e.g., Brady et al., 2012), and, more specifically, a relationship between individual characteristics, socio-economic status, and stratification theory (education, income, and occupation) thereby illustrating that a citizen’s political efficacy and participation is reflective of their personal history (Brady et al., 1995; Clark, 2017; Toots, 2019).

Similarly, also from a political perspective, numerous approaches have been proposed to understand how citizens engage with public services. Some focus on the ‘administrative burden’ that citizens experience and highlight how “hidden politics in the policy process affect and sustain persistent inequities in policy access” (Burden et al., 2012; Moynihan et al., 2015; Nisar, 2018; Masood and Nisar, 2021, 56; Giest and Samuels, 2022). Other research looks more generally at the ‘administrative barriers’ that potentially exclude or burden some groups—specifically, those with lower education levels, arguing that such barriers are further linked to outright ‘administrative exclusion’ (Brodkin and Majmundar, 2010).

2.1.2. Crowdsourcing and collective intelligence

In addition to digitally enabled public participation, crowdsourcing and collective intelligence is another area of emerging research. Crowdsourcing, as defined by Mergel and Desouza (2013), is an approach that allows “collective intelligence to be leveraged to solve complex problems when resources or expertise are otherwise lacking.” Collective intelligence is understood as a “form of universally distributed intelligence, constantly enhanced, coordinated in real-time, and resulting in the effective mobilization of skills” (Levy, 1997).

Digital technologies have led to the creation of new collective intelligence platforms that provide a place for individuals to coordinate, collaborate, and communicate in real-time to, for example, generate and gather knowledge from the “wisdom of the crowd” (Suran et al., 2020). There are many examples of how digital platforms can harness collective intelligence to develop value-producing platforms, such as Wikipedia or OpenStreetMaps (Budhathoki and Haythornthwaite, 2013). Increasingly, data generated from the crowd is playing an important role in the development of a digital world, for example, in training state-of-the-art AI-based systems (Verhulst, 2018; Verhulst et al., 2021).

Looking at the success of these distributed collective intelligence platforms, governments have begun to experiment with similar initiatives and initial research does seem to suggest that when new services engage with citizens, it may generate better public services (Dutil, 2015; Liu, 2017). While there appear to be some benefits to crowdsourcing and collective intelligence, and the development of new digital technologies is improving the accessibility of these techniques, it is also clear that the applicability and usefulness of crowdsourcing are still limited (e.g., van den Berg et al., 2020). Potential weaknesses include a lack of support from internal stakeholders (Mergel and Desouza, 2013), questions around data quality (Leibovici et al., 2017; Jacobs, 2016), and in the case of the public sector, a lack of interest from citizens (Gelman et al., 2011; Liu, 2017).

Due to these weaknesses, some scholars warn that the cost of engaging citizens drains resources from core administrative work (Liu, 2017) and may cater specifically to a rather idiosyncratic group of highly motivated civic influencers and/or groups that have been uniquely galvanized around an issue that affects their economic, political, or social interests (Cunningham, 1972; Fiorina, 1999; Cleaver, 2005; Lodge and Wegrich, 2015).

2.2. Transformation of relationships and organizations

Key topics, including the co-creation of public services, trust and power dynamics, organizational structures, and the reimagination of bureaucratic structures, fall under the category of citizen-government-business relationships. The impact of digital and data-driven applications on these interactions is evident and has led to a heightened interest in concepts like “human centricity,” GovTech, data exchange, data ownership and consent, and automated decision-making.

Two emerging paradigms encapsulate these transformations. Osborne’s “new public governance” emphasizes a more inclusive, involving, and human-centric government (Osborne, 2006; Osborne, 2010). The second paradigm, outlined by Dunleavy and Margetts, describes the “third wave” of Digital Era Government, marked by data-intensive policymaking, robotic state development, intelligent delivery structures, and wider administrative holism (Dunleavy and Margetts, 2023). This presents a complex dichotomy where the public sector is expected to be more inclusive and citizen-centric, yet simultaneously automated and reliant on algorithmic systems (Vogl et al., 2020; Meijer et al., 2021).

2.2.1. Co-creation

A core area of research within this domain is co-creation, particularly in the context of new public services (Osborne et al., 2016). Defined as “the involvement of outside, non-typical, stakeholders in the initiation, design, implementation, and evaluation of a new public service” (Toots et al., 2017), co-creation holds the potential to enhance service accessibility, personalization, and generate higher public value for administrators and users (Casiano Flores et al., 2021; Jarke, 2021). The extant literature explores how co-creation processes unfold, particularly in the context of new digital systems. Lember et al. (2019) present technologies that could diversify co-creation opportunities, while cautioning against a potential future where decision-making relies on complex algorithms and collective data.

Studies on digitally enabled co-creation, such as those by Khayyat and Bannister (2017), McBride et al. (2019, 2023), Nikiforova (2022), demonstrate how open government data (OGD) fosters innovation ecosystems supporting co-creation. This has been further demonstrated during the COVID-19 pandemic, where co-creation and OGD-based ecosystems emerged as strategic crisis management tools (McBride et al., 2023) and opportunities to promote innovation through, for example, hackathons (Kangro and Lepik, 2022; Nikiforova, 2022).

Though initial evidence suggests clear benefits for data and digitally-enabled forms of co-creation, there is limited evidence regarding the tangible and sustainable outcomes of co-creation initiatives (Cingolani, 2021). Further research is needed to understand how co-creation unfolds in public service development and delivery, including its process, and its effects on stakeholder relations, especially regarding power dynamics, information asymmetry, and trust. In this context, Castells’ work (2013, 2022) on power relations offers a critical lens through which to examine the role of technology in reshaping interactions between governments and citizens. His insights into the network society and the shifting nature of power in the digital age can illuminate the complexities of co-production, particularly how technology can both empower and marginalize different stakeholders. Understanding these dynamics is crucial for assessing the true impact of co-creation initiatives on public service development and delivery, and in ensuring that such efforts genuinely enhance democratic engagement and equity.

2.2.2. Digital transformation of public services

Against this backdrop there are two sets of factors relevant for the citizen-government relationship in the context of digital public service delivery: first, the design and usability of the system matters. Second, personal experience and characteristics that affect comprehension and navigation abilities when confronted with the digital system. These include experience with (similar) systems (administrative capital), as well as individual, demographic characteristics.

From the public servant perspective, bureaucrats, and especially those at the front-line or so-called ‘street-level bureaucrats’, which have traditionally been directly involved in the experiences that citizens have with a public service are seeing their roles and relationships with citizens transform (Olabe, 2017). This change is driven by today’s digital world where screens might replace personal engagement on both sides of the citizen-state interaction. For bureaucrats in particular, studies show that digitalization has come with managerial demands for efficiency coupled with performance metrics (Jewell, 2007; Brodkin, 2011, 2012). This has effects on their responsiveness to citizen-clients, their discretionary space in the decision-making process as well as their position as mediators between citizens and managers as well as policymakers (Gofen and Lotta, 2021).

Some foresee the emergence of new bureaucratic roles aligned with digital processes. Bovens and Zouridis (2002) delineate three employee groups: “(1) those actively involved in data processing, including system designers, legislative specialists, legal policy staff, and system managers; (2) managerial roles overseeing production processes; and (3) the ‘interfaces’ engaging citizens with information systems, encompassing public information officers, help desk members, and legal staff handling complaints and objections on behalf of the organization” (Ibid, 180).

This suggests that, as per Bovens and Zouridis (2002), system designers, legal policy staff, and IT experts assume roles akin to modern-day street-level bureaucrats. Recent research, however, emphasizes that even as face-to-face interactions diminish in public service delivery—hallmarks of street-level bureaucrats (Lipsky, 1980)—individual bureaucrats retain influence over policy implementation and decisions concerning citizen-clients, given that many applicants do not neatly fit eligibility criteria (Keiser, 2010). Studies on street-level bureaucracy and digital office technologies support this, asserting that digital applications aren’t just managerial tools impacting these bureaucrats but an infrastructure shaped by frontline workers (Hoybye-Mortensen, 2019).

2.3. Open data and open government

The third thematic category encapsulates the broader shift that we are seeing in today’s public sector to embrace ‘openness’; with topics of interest such as open innovation ecosystems, transparency, OGD, and personal data ownership.

2.3.1. Emergence of open government

The openness paradigm outlines a future for government that is open, responsive, and transparent, where government data is freely available for any to use and create value from, and where technological innovations and developments funded by the public sector are able to be freely reused by any who are interested. From a technological standpoint, “openness” is primarily associated with the open-source software movement originating in the late 1980s and early 90s which placed a strong emphasis on “opening up” software so that it was free, reusable, and non-discriminatory, amongst other things (Kelty, 2001; Stallman, 2002; Tozzi, 2017). Viewing the open-source software movement as a success, many were interested in how this belief system could be applied in other areas of society.

One sector with perhaps the strongest interest was the public sector and the idea of an “open government” began to emerge together with the associated topic of “open government data”—data collected, produced, or paid for by the public bodies and made freely available for re-use for any purpose¹ compliant with principles² of (1) completeness; (2) primariness; (3) timeliness; (4) availability; (5) machine-readability; (6) non-discriminatory; (7) non-proprieteness; (8) license-free (Bauer et al., 2020). In line with this transformation, there is a large amount of literature that helps untangle the complexities associated with openness in the public sector.

Scholars have explored how openness and open government data can drive (open) innovation and economic value creation (Jetzek et al., 2014; Susha et al., 2015; Ahmadi Zeleti et al., 2016; Ruijer and Meijer, 2020), promote accountability and transparency in the public sector (Mayernik, 2017; Matheus and Janssen, 2020; Lnenicka and Nikiforova, 2021), be used to assist in crowdsourcing and collective intelligence applications (Handler and Ferrer Conill, 2016; Hemmings et al., 2019; Colovic et al., 2022; Nikiforova, 2022; McBride et al., 2023) within open data ecosystems (Dawes et al., 2016; Najafabadi and Luna-Reyes, 2017; Linåker and Runeson, 2021; Lnenicka et al., 2024a, 2024b), and lead to the creation of new digital services or create new public service co-creation ecosystems (Turki et al., 2017; Lember et al., 2019; McBride et al., 2019; Miasayedava et al., 2022).

Research has similarly highlighted that along with these benefits, there are also risks, with some scholars highlighting the potential “dark side” of openness (Zuiderwijk and Janssen, 2014; Charalabidis et al., 2018; Schnell, 2020). According to Schnell (2020), this includes the idea of “autocratic openness,” where open government initiatives hide more authoritarian norms behind the scenes. To Zuiderwijk and Janssen (2014), the “dark side” includes potential risks to privacy, wasted resources, or poor decision making, amongst others, which are even increasing in the current age of Generative AI.

2.3.2. Creating value from open government data

One potential way to create value from OGD is through the support and development of innovation infrastructures, also called public data ecosystems open data ecosystems, data spaces, and innovation platforms. Yet, there is a clear need for these innovation infrastructures, data ecosystems, and data exchange platforms (Bharosa et al., 2020; McBride et al., 2022). One key component of these ecosystems is individual data ownership and consent management. With increased interest in the opening up of data, there has been a drive to develop and implement the means for citizens to “own” their data, being able to use it as they wish and share it only as needed (Lehtiniemi and Haapoja, 2020).

While research does exist on new forms of data ownership and management, especially in the smart city literature where ideas such as data collaboratives or data banks are often discussed (Paskaleva et al., 2017; Susha et al., 2017; Klievink et al., 2018; Artyushina, 2020; Bozkurt et al., 2022), there is still a lack of research that explores and conceptualizes the personal ownership of data.

2.4. Case study of digital and data-driven transformations: data dashboards

The case study of ‘data dashboards’ incorporates several of the points raised in the above themes. Data dashboards are defined as” the visualization of a consolidated set of data for a certain purpose, which enables one to see what is happening and to initiate actions” (Matheus et al., 2020, 2). A dashboard application is often developed and offered in collaboration with private stakeholders and combines citizen-generated data with government data. This government data could be based on open data of a city or government unit for example. Dashboards can further be used for decision-making by the government and the public through summarizing and visualizing a certain issue (in real-time).

Given these characteristics, some say that the use of data dashboards promotes larger public goals, such as transparency, performance monitoring, reporting, planning and policymaking. In fact, dashboards “are often aimed at empowering the public by creating transparency and accountability” and for reducing information asymmetry (Janssen and van den Hoven, 2015; Matheus et al., 2020). At the same time, there are risks in using dashboards to achieve these goals. Dashboards might even be useless if they are not paired with government-internal organizational changes. Even more, if dashboards are not designed properly, they might result in misinterpreting data, which can affect public trust in government.

The application of data dashboards highlights several aspects that apply more generally to digital and data-driven transformations in governance, which are:

• • more collaborative efforts with (private) stakeholders required in order to gain access to and understand data as well as build digital applications;

• • organizational changes need to accompany digital and data-driven transformations in order to make sense of data use;

• • opportunities as well as challenges in engaging the public in digital and data-driven transformations;

• • availability, interoperability, and accessibility of data plays a key-role in driving data-driven applications.

2.5. Key research questions emerging in the area

The “digital and data-driven transformations in policy and governance” area is incredibly diverse, with research interests that necessarily draw on multidisciplinary inputs, research themes, methodologies, and philosophies. To create and conceptualize an inclusive research regiment within the area, it is important to identify larger trends and categories that are driving the digital and data-driven transformation in policy and governance within which increasingly granular research topics, questions, or interests can be situated.

In the above sections we have identified three of these categories that appear to be essential: (1) public participation and collective intelligence; (2) transformation of relationships and organizations; and (3) open data and government. Importantly, these categories are not mutually exclusive, but rather form a co-evolutionary cycle increasingly driving change and transformation.

To provide a foundation for future research, Table 1 gives an overview over example research questions emerging from the trends highlighted here, however this is not an exhaustive list. Ultimately, by engaging with key questions in these Data for Policy focus areas it will be possible to help untangle and bring clarity to the broader Data for Policy focus areas of digital and data-driven transformations in governance.

Table 1. Overview of topics and potential research questions

3. Concluding remarks

In conclusion, this review establishes a forward-looking agenda for Data for Policy research, specifically within the realm of digital and data-driven transformations in governance. Adopting a multi-disciplinary perspective, the paper delves into both established and cutting-edge research lines in public administration, policy studies, information systems, and computer science. The exploration reveals that digital and data-driven transformations in governance are intricately linked to internal institutional and bureaucratic processes, as well as their intersection with citizens and stakeholders affected by these changes.

At this intersection, the paper identifies participation and interaction procedures impacted by digitization and datafication, such as crowdsourcing and co-production mechanisms, along with broader societal impacts on the agency of both citizens and bureaucrats. The evolving organizational context driven by digital and data-driven applications carries implications for public sector employees, influencing their roles in the service delivery process. Notably, citizens’ experience with public services are crucial for fostering trust and confidence in government.

The digital and data-driven applications further introduce managerial demands for efficiency and performance metrics for bureaucrats, with the automation of processes contributing to a sense of disenfranchisement. For citizens, a new digital divide emerges, not in terms of access or skills, but in terms of their voice in decision-making processes. The review also highlights defining events in the open data and open government movement, raising questions about the realization of benefits associated with openness and potential trade-offs concerning privacy, data ownership, and quality. The role of AI in data quality, transparency, and explainability in a government context is also examined, alongside challenges related to cross-border data exchange.

Emphasizing government and its ecosystem, the review outlines three overarching future research areas: (1) public participation and collective intelligence; (2) transformation of relationships and organizations; and (3) open data and government. These areas delve into how data is shared across stakeholders, addressing questions of interoperability and the public’s role in co-designing and participating in these practices. The paper recognizes that these practices impact relationships, organizations, trust, power dynamics, and the design and delivery of public services. Changes in capacities and competencies, both in government and among citizens, including digital access, data literacy, and data management, are also highlighted.

In summary, technological advances in the public space are shaped by existing institutional dynamics and organizational structures. Values such as legitimacy, accountability, and transparency play a crucial role in the utilization of data-driven innovations. Simultaneously, digital and data-driven transformations reshape how the government interacts with citizens and other stakeholders, encompassing regulatory settings, digital communication, data responsibility, and a broader understanding of digital access and data literacy. The review emphasizes the presence of institutional and technological legacies in different contexts and the concurrent creation of new legacies alongside existing ones, a critical aspect often overlooked in research on digital and data-driven applications.