Research directions in policy modeling: Insights from comparative analysis of recent projects

Abstract

With the increased availability of data and the capacity to make sense of these data, computational approaches to analyze, model and simulate public policy evolved toward viable instruments to deliberate, plan, and evaluate them in different areas of application. Such examples include infrastructure, mobility, monetary, or austerity policies, policies on different aspects of societies (health, pandemic, skills, inclusion, etc.). Technological advances along with the evolution of theoretical models and frameworks open valuable opportunities, while at the same time, posing new challenges. The paper investigates the current state of research in the domain and aims at identifying the most pressing areas for future research. This is done through both literature research of policy modeling and the analysis of research and innovation projects that either focus on policy modeling or involve it as a significant component of the research design. In the paper, 16 recent projects involving the keyword policy modeling were analyzed. The majority of projects concern the application of policy modeling to a specific domain or area of interest, while several projects tackled the cross-cutting topics (risk and crisis management). The detailed analysis of the projects led to topics of future research in the domain of policy modeling. Most prominent future research topics in policy modeling include stakeholder involvement approaches, applicability of research results, handling complexity of models, integration of models from different modeling and simulation paradigms and approaches, visualization of simulation results, real-time data processing, and scalability. These aspects require further research to appropriately contribute to further advance the field.

Policy Significance Statement

Through an analysis of 16 recent projects on policy modeling in Europe and beyond, the research provides an overview regarding the area of application, relevant findings and possible future research for each project. The authors also synthesize the main directions of future research in policy modeling, providing insights into the future development of computational modeling of policy. The research is useful for policy makers as an analysis of current research and as exploration of possible future developments in the field, thereby spotting the biggest challenges and opportunities of policy modeling for data-driven and evidence-based policy decision-making.

1. Introduction

Policy modeling can be applied to make decision-making processes during the design, planning, and creation of a policy more intelligent, evidence-based, and agile (Lampathaki et al., 2011). Still, while 10 years ago, researchers voiced great optimism in computational policy modeling, tools and simulations are still not extensively used by policy makers. Hamill argues that one of the reasons for that is the complexity of the domain policy modeling (Hamill, 2010).

In regard to models dealing with complex social phenomena, accurate predictions are often impossible. Gilbert et al. (2018) argue that the probabilities of certain events and directions of influence and interdependencies of different factors are the main findings of policy modeling. Still, the value of these findings can be very high: for example, learning that a certain event is in the range of the possible can help make a policy more flexible and responsive (Gilbert et al., 2018), or finding evidence that certain (desired) outcomes seem impossible may lead to the adjustment of a policy. Gilbert et al. (2018) even argue that the process of modeling is more valuable than the actual output of the model as the modeling contributes to the understanding of complex social processes.

On the political level, policy modeling can increase political legitimacy through realizing data-driven and evidence-based decision-making: relying on data when making policy decisions (Starke and Lünich, 2020) and providing data confirming the effectiveness of enacted policies (Papazoglou Chalikias et al., 2020). Another important aspect of policy modeling is the value of a model as an educational tool. Models can be useful for educating the audience and communicating how certain systems work (Voinov and Bousquet, 2010). The process of creating a model can also contribute to the understanding of phenomena by the scientists. Models, when created collaboratively, can synthesize the knowledge of many experts and provide a glimpse into the functionality of complex systems such as climate systems (Stocker et al., 2013).

Analyzing and modeling a policy or creating simulation tools to run and test policy models is a significant endeavor, which requires interdisciplinary teams and collaboration among different stakeholders. Such collaborations are often realized as projects funded through external funding programs. With the increased availability of data as well as tools and technologies to deal with these data, policy modeling has evolved into an attractive and interesting research avenue to pursue. This exploratory paper aims to shed light on research needs, challenges, and opportunities in policy modeling through a literature review and analysis of recent research and development projects in the domain. The ultimate goal is to deliver a comprehensive understanding of the current developments in policy modeling. The research is motivated by the following two research questions:

RQ1: What particular contributions did/do recent research projects in policy modeling aim to deliver?

RQ2: What future research needs in the domain of policy modeling can be identified from literature review and from the analyzed research projects?

To answer the first question, research projects in policy modeling were identified and analyzed (partly in the scope of the Gov 3.0 project ¹). The analysis included 29 research projects mainly funded at EU level since 2010. For the research in this paper, the projects were further limited to 16 projects that ended in 2016 or later. For the collection of the data about the projects, a template was developed to collect the main aspects (formal data on each project, type of project, general description, technologies used, soft factors studied, technology readiness level, and pilot application area) of the projects in a comparable format. For the further analysis toward identifying research needs, a second template was developed to extract the identified research needs, the research needs the project addressed already, as well as the identified training needs and training needs addressed in the project. The analysis included a qualitative and thorough study of the deliverables of each project to distil relevant issues and distinguish the commonalities and differences between the projects. The rationale for this approach is that projects can be a useful source for future research needs of the domain for several reason: (a) joint research and piloting projects explore solutions that may be subsequently implemented on a larger scale and (b) project deliverables often discuss future research needs or future directions in the area. To find this information, the deliverables and reports of each project were analyzed for any indications such as “future directions,” “further research,” “limitations,” or similar. The authors’ discussions of these topics were summarized in the corresponding column of the second template, which is presented as an adjusted table in Section 3. As these insights are based on the findings of the project team working in the domain for several years, they are particularly valuable and shall be shared with the scholarly community.

For the second research question, a literature review, guided by (Webster and Watson, 2002), was conducted (again initially in the Gov 3.0 project ² and subsequently expanded), based on keyword search in Google Scholar, Scopus, Web of Science, ACM Digital Library databases. The suitability of each article for inclusion in the reference dataset was evaluated by the researchers after perusing the article’s abstract and keywords and a quick review of the article’s full text.

With this in mind, the paper is structured as follows: Section 2 reviews relevant literature on the different aspects of policy modeling to provide the necessary foundations for the project analysis. It establishes the foundational aspects for the project description and comparison, which follows with explanations and a comparative table in Section 3. A synthesis of the information relevant for answering the research questions is provided in Section 4. In Section 5, we conclude with a summary of the research conducted and findings generated, and we spot some limitations of the research.

2. Literature Review

Ruiz Estrada defines policy modeling as “an academic or empirical research work that is supported by the use of different theories as well as quantitative or qualitative models and techniques, to analytically evaluate the past (causes) and future (effects) of any policy on society, anywhere and anytime” (Ruiz Estrada, 2011, p. 524). Computational policy modeling characterizes the primarily technological means in the policy analysis and modeling, which is—compared to classical policy trials, pilots and experiments—more flexible, faster and cheaper to implement (Gilbert et al., 2018). In literature, some ambiguity exists between the terms policy modeling and computational (policy) modeling: while the former is a broader term, which, beyond technological tools, includes also nontechnological methods, instruments, and tools to generate results of a policy implementation (e.g., scenario building, serious games played without computers), the former is also used to mean the latter (i.e., policy modeling done with a computer system). In the paper at hand, we use the broader term policy modeling since our considerations embody the combination of technical and non-technical approaches to achieve more sophisticated, more informed, and more reliable policy making.

Academic research on policy modeling and on the development of technological tools provides insights into how information technology can assist policy makers. Along the policy lifecycle (Howlett and Ramesh, 1995), policy modeling can be employed on different stages: not only during the design of a policy, but also in the evaluation (or appraisal) of policies (Koliba and Zia, 2013; Gilbert et al., 2018). The key goal in applying comprehensive policy modeling approaches is to improve a policy or experiment with possible variations of the policy in iterative steps (HM Treasury, 2013).

The subsequent sections summarize key insights from literature on (a) different fields of policy modeling, (b) policy modeling along the policy lifecycle, (c) methods and instruments for policy modeling, and (d) different theories and paradigms of policy modeling and simulation.

2.1. Fields of policy modeling

Ruiz Estrada (2011) reviewed 1,501 papers of the Journal of Policy Making to classify the existing policy modeling research in the Journal with respect to the field in which it contributes. The author established the following twelve fields of research: (a) domestic and international trade policy modeling; (b) energy, communications, infrastructure, and transportation policy modeling; (c) environmental and natural resources management policy modeling; (d) fiscal and government spending policy modeling; (e) institutional, regulation, and negotiation policy modeling; (f) labor, employment, and population policy modeling; (g) monetary, banking, and investment policy modeling; (h) production and consumption policy modeling; (i) technological and R&D policy modeling; (j) welfare and social policy modeling; (k) economic growth and development policy modeling; and (l) miscellaneous policy modeling (Ruiz Estrada, 2011).

While Ruiz Estrada already provides a sophisticated overview of fields of research of policy modeling from before 2011, his research was limited to the Journal of Policy Making and new areas may have emerged over the past decade. However, gathering a comprehensive overview would require an extensive literature study involving different databases of academic literature, which is not the objective of this paper.

2.2. Policy modeling along the policy lifecycle

Different scholars describe the policy lifecycle. Howlett and Ramesh (1995) for example distinguish agenda setting, policy formulation, decision making, policy implementation, and policy evaluation in an iterative cycle. Agenda setting is about establishing the need for a policy or policy change. Policy Formulation includes gathering evidence from the relevant stakeholders, collecting data to better understand the political context and formulating or updating a policy. Decision making refers to the final approval by legal or strategic actors and the enactment of a policy. The policy is then implemented through its application in the relevant domain (Policy Implementation). And finally, policy evaluation looks into monitoring performance, success, impact, and acceptance of the enacted policy, which may initiate another cycle. Similar descriptions with nuances of differences can be found in Macintosh (2004). Other scholars build on one of the two models, for example, Janssen and Helbig (2018).

Policy modeling techniques can be used in each stage of the policy lifecycle (Janssen and Helbig, 2018). Practically, policy modeling is most useful on the planning and evaluation stages, when modeling and simulation techniques can support the assessment of how a particular policy enactment will affect a given situational context or to extrapolate the effects of a policy over time. In both cases, models are useful to go beyond the available information and generate new knowledge, leveraging sophisticated modeling techniques (Thalheim, 2011).

2.3. Methods and technologies for policy modeling

Policy modeling is about studying a complex policy field and trying to understand the main influence factors and how these are interacting toward particular policy objectives. Different methods can be employed to gain insight into the consequences of a particular policy. In the following, we outline several approaches discussed in literature:

A common approach is to rely on data from the past: extrapolations based on existing historical data either relating to the existing situation or a similar situation. This data-driven modeling approach is best suited for predicting repeatable situations, for example raising the taxes has been done often, so historical data can be used to reliably predict the societal consequences of the possible changes in fiscal policies. Data-driven inference is a booming strand of research as more data (big data) become available and technological capacity for the data analysis improve (Pruyt, 2016). Data-driven modeling can be used for ex-ante policy evaluation, when the actual data related to the implementation of the policy is compared to the counterfactual, describing what would happen in the absence of the policy (Brandt and Dlugosch, 2020).

A different approach is to use methods able to generate novel insights that are not directly based on historical data. This approach may be based on the opinions of experts (like in the Delphi method or in the scenario-building approach), who produce insights and ideas that go beyond extrapolation from pre-existing data. This approach is better suited for predicting the unpredictable and unexpected, also termed “unanticipated knowledge” (Zabell, 1992). For example, the effects of an invention of artificial intelligence (AI) may be difficult to predict based on data, because it is a unique situation that has never happened before. This area has also been transformed with the increase of computational capacity: the input from the experts can be used to build formal simulation models improving and validating the predictions (Kopec et al., 2010; Stach et al., 2010).

In relation to these two approaches, Kim et al. (2017) distinguish between data modeling and simulation modeling. Simulating the different processes in society is difficult as there are a lot of variables to consider. The simulation models continue to improve and evolve over time. Simulation models often involve citizens at some point. The involvement can take different forms: from the weakest (providing feedback about the model structure or results) to the stronger like actually modifying or co-producing a model (Mureddu, 2019). For the stronger forms of involvement, effectively engaging citizens is a challenge in itself: it is necessary to balance the complexity and usefulness of the feedback. If the desired feedback requires expert knowledge of the simulation model, the feedback from the citizens will be limited. While more general feedback is easier to collect, its usefulness is limited. To tackle this challenge, different approaches may be used: from supporting the simulation models with narrative scenarios and visualizations to the creation of platforms for facilitating the feedback to the specific aspects of the model (Scherer et al., 2015; Mureddu, 2019).

2.4. Theories and paradigms of policy modeling and simulation

Policy modeling and simulation can be distinguished by their underlying theories and paradigms of modeling and simulation, which started to develop as a field in the nexus of activities between government, military, and industry after World War II (see Gilbert and Troitzsch, 2005; Majstorovic et al., 2015 for a comparative analysis of the different paradigms). Scholars differentiate Microsimulation, Agent-based Modeling, Macrosimulation, and System Dynamics.

Microsimulation is a paradigm, which mostly builds on (large) data from the past. The modeling focus is either on static (e.g., evolution of tax volume over time in a given context or territory) or dynamic factors (i.e., behavior of people towards paying tax). Bae et al. (2016), for example, describe individuals in population with micro population data and simulate their behaviors based on population statistics. Microsimulations have a long history, first mentioned in 1950s when they were used in economic modeling, they are used effectively in taxation, welfare pension and other policy areas (Birkin and Wu, 2012). If dynamic behavior is applied in microsimulation, microsimulation models can transition to the paradigm of agent-based modeling (Heppenstall et al., 2012, p. 6).

Agent-based modeling (ABM) focuses on dynamic aspects/behavior of actors in complex system environments. Batty et al. (2012) argue that ABM offers “the capability of reflecting the richness of the world in a way that appears essential to any good explanation of how spatial structures such as cities, regions, the global system itself as well as all its physical components evolve and change”. ABM models the behavior of individual units (agents) involved in these processes. For example, modeling crowds by modeling the behavior of individual people. The pivotal challenge with the ABM paradigm is determining the unit of analysis; a larger number of agents leads to more complex simulation. Therefore, individual actors are often aggregated into actor groups with less granular behavioral characteristics, which bears the risk that the simulation model does no more reflect the real behavior. Scholars therefore argue that modeling the individual agents’ properties close to the reality reflects reality more accurately (Macal and North, 2014; Scherer et al., 2015).

System Dynamics is about looking at the problems on the global scale and viewing the world as a supersystem consisting of a number of complex interconnected systems (Jaeger et al., 2013). Simulation models based on system dynamics model a situation on a global level to describe a real-world system using analytical means via systems of differential equations (Gilbert and Troitzsch, 2005). A real-world system is described and analyzed as a whole on the macro-level (Forrester, 1961) and represented using flow diagrams and internal feedback loops (Coyle, 1997; Harrison et al., 2007). System Dynamics models do not require much data and the output of the model consists of plots describing behavior and the changes of the initial values of the variables and parameters of the model over time. To describe the behavior of a real-world process accurately, a model needs to be run many times with different parameter values (Maria, 1997). A typical use of system dynamics models is macro-economic modeling as well as for describing impact of policies during, for example, a spread of a disease. According to Astolfi et al. (2012), system dynamics models are well suited for predicting short-term policy impacts.

It is also possible to combine different approaches within a simulation model to improve accuracy or analyze the system on different levels, realizing multilevel modeling (Majstorovic et al., 2015; Bae et al., 2016).

3. Overview of Projects in Policy Modeling

In order to gather insights into recent research on policy modeling (mainly in European Research), we studied 16 research projects, of which 14 are/were funded within European funding schemes and two are projects conducted outside the EU. The projects ending 2016 or more recently have been identified based on search in the funding database Cordis ³ and through web search using “policy modeling” keywords and year of finalization as an exclusion criteria for the selection of the projects.

Table 1 summarizes the results of the project analysis. The template used for the analysis in this paper is based on the initial templates developed in the Gov 3.0 project and is adjusted to contain the following elements: Acronym provides the short project name along with a footnote to the project website or the Cordis database entry. Project duration lists the timespan in years, in which a project has been executed. Stage in policy lifecycle and technologies lists the stage in the policy lifecycle at which the project is centred (based on the stages presented in Section 2.2) and technologies used in the policy modeling aspect of a project. Areas list the policy modeling areas as described in Section 2.1. Projects focusing on theory development or projects providing a generic solution to “any” policy area are indicated as well. Summary outlines the focus of the project, while the Artefacts/Results lists key models/prototypes/solutions elaborated in the course of the project. Finally, the last column Identified future research lists the policy modeling-related future research directions extracted from deliverables, reports, or publications of a project.

Table 1. Overview of analyzed projects on policy modeling, which ended 2016 or later

4. Insights from the Project Analysis

In this section, we synthesize the findings of the project analysis and discuss them in relation to the existing literature on the subject.

4.1. Stages in policy lifecycle, technologies/methods, and modeling paradigm

The majority of the projects developed solutions that can be used in two or more stages of the policy lifecycle. The stages mainly addressed are agenda setting, policy formulation, and polity evaluation.

The most frequently mentioned technology is big data, used as input for simulations or models. Other technologies applied in the projects analyzed are machine learning, social data analysis, and natural language processing. The technologies used are dependent on the stage in the policy lifecycle: social media analysis is used in projects focusing agenda setting to collect inputs from citizens, while big data is more often mentioned by projects addressing policy evaluation.

Projects that involved simulations, applied either the paradigm of agent-based simulations or addressed serious games and gaming-based simulations. For example, the SYMPHONY project ⁴ produced a serious game simulating the global financial market. Serious games can also be used as a modeling and prediction tool. However, often the focus of the games lies on educating the players. Agent-based games, for example, have already been used in other complex environments, like management (Jager and van der Vegt, 2015).

4.2. Domain of application and project results

The focus of the majority of the projects in the analysis is on the development of a policy simulation model of the specific domain. The main outcome of such projects is usually either a theoretical model or a prototype that is tested on specific use cases in the domain.

The areas of application, mapped to the classification presented in Section 2.1, include energy, infrastructure and transportation policy modeling [Consensus, INSIGHT, ModeRATED Policy], health policy modeling [FRESHER, REPOPA, PoliVisu, Prevention Policy Modeling Lab], financial and monetary policy modeling [SIMPOL, SYMPHONY], labor, employment and population policy modeling [COMPASS NZ], and environmental and natural resources policy modeling [SIMPOL, SUPREMA]. Some projects focused on solutions that can be applied to many different policy areas [SENSE4US, Policy Compass]. Other modeling areas include cross-cutting aspects like crisis management [Crisis] or risk modeling [RISK].

Several projects are not oriented toward modeling a specific policy issue, but rather focus on the policy modeling domain as a whole, such as Big Policy Canvas.

Most of the literature dealing with policy modeling takes a technical research path and focuses on the technologies that can support policy modeling for Government. In addition, as underlined by Brooks et al. (2018), the policy modeling strand is fragmented as it encapsulates different disciplines which are distinct from each other and meet in their own conferences and specialist fields. Realizing these limitations, scholars have attempted to strengthen or set up international communities for ICT solutions for governance and policy modeling (e.g., in the eGovPoliNet project, see Janssen et al., 2015 and Majstorovic et al., 2015), aiming to build on experiences accumulated by leading actors and bringing together innovative knowledge of the field. Beyond bringing together different research communities working on policy modeling topics, their work aimed to provide a barometer for effectiveness for public governance and policy modeling in Europe and worldwide. The eGovPoliNet project provided important theoretical and empirical contribution in policy modeling research (Janssen et al., 2015). Mellouli et al. (2015) produced a comparative analysis of different technology frameworks for policy modeling, classifying them into three categories: (a) frameworks focusing on policy process stages, (b) frameworks focusing on institutions, and (c) those that focus on human factor and collation.

A more recent project, Big Policy Canvas (2017–2019) looked at the needs in data-driven policy making and created a roadmap of future research needs and research actions. Policy modeling was named as one of the challenge clusters in relation to data-driven policy making. Specifically, Big Policy Canvas looked into the gaps that hinder the rapid and effective uptake of policy modeling. The apparent need to accelerate the policy modeling process has been identified as one of the research gaps in the project (Mureddu, 2019, p. 87). Policy modeling is also one of the six research clusters related to the big data use in public policy (Mureddu, 2019), specific relevant research challenges associated with this cluster are Collaborative model simulations and scenarios generation and Integration and re-use of modeling schemes.

4.3. Future directions of research: challenges and opportunities

Summarizing the future research envisioned in the Gov 3.0 project, several trends can be formulated, which can be broadly grouped into two categories: (a) overcoming existing challenges and (b) realizing opportunities.

The challenges-related research directions include:

Stakeholder Involvement. Gilbert et al. (2018) argue that the value of policy models will be very low, unless the decision makers are involved iteratively in the development of the policy model. It is also often necessary to involve the affected stakeholders both for improving the quality of the model and fulfilling the demand and expectation of participation (Voinov et al., 2016). Stakeholder involvement therefore requires common understanding (modeling standards, procedures, and methodologies; Mureddu, 2019), likely the unification of modeling theories (Wimmer and Majstorovic, 2015) and improvements in usability of the tools [PoliVisu].

Transparency and applicability. The public demands certain level of transparency from the policy makers. Policy modeling however produces results that might be difficult to explain, policy models can be black box systems that produce certain output based on the (often very large amount of) input, while the actual way from the input to output may be very difficult to comprehend (Armenia et al., 2014). Creating more transparent models is necessary to justify policy making decisions based on them, however, this still remains a challenge [COMPASS]. At the same time there is a significant issue of public mistrust of science and the validity of scientific findings in relation to political life (Ibrahim and Larsson, 2019; Edis, 2020).

Complexity. As the subject of modeling is a society or its part, it is intrinsically difficult to consider all the relevant variables involved in the model: necessarily reducing the range of considered factors to a manageable amount (Easton, 1957). Thus, the policy model would always be an imperfect attempt to simulate the reality. Models are continually becoming better with the advancements in both computing power and sophistication of the modeling approaches. Still, a number of projects listed the complexity as one of the research challenges [CRISMA, SUPREMA]. Other related significant research challenge indicated by a number of projects is the need for an integrated model in a domain. SUPREMA project report (Jongeneel et al., 2020), for example, mentions the diversity of models and the challenges of maintaining them as a biggest challenge in agricultural modeling. (Mureddu et al., 2012), FRESHER (Mattioli et al., 2018), and INSIGHT (INSIGHT, 2013) projects on the other hand discussed the importance of data integration from different sources and disciplines as a way to improve the interpretation of complex data.

Visualization. Directly as a result of the previous challenge, even the most sophisticated model is useless if its users are not be able to make sense of it (Burkhardt et al., 2013). Visualizing the results of the modeling process remains a significant challenge as suggested by projects like EU Community, SYMPHONY and Big Policy Canvas (Mureddu, 2019).

Comparing the identified research directions to the findings of a policy modeling roadmapping effort conducted a decade ago (Lampathaki et al., 2010), it is worth noting that fewer challenges now are purely technological, while the majority of the research findings still remain relevant even ten years on: visualization, stakeholder involvement and model integration are present both in the CROSSROAD roadmap (see Lampathaki et al., 2010) and in the current results.

The opportunities include the research directions focusing on taking advantage of the new technological developments, particularly in relations to the big data analytics, increasing computational capacities and Machine Learning algorithms (Mureddu, 2019). This is particularly significant when real-time data is used [INSIGHT, PoliVisu] (INSIGHT, 2013).

Sustainability is a concept mentioned often in the project descriptions. Using policy modeling techniques is perceived as a way to reach sustainability in society or one of the domains, while sustainable models are one of the goals of policy modeling. Connected to sustainability is the need to reach wider audience with the results of policy modeling and simulations. This can be achieved through simplification of the presentation and providing visualizations of the processes and the results. Making the complex models transparent and understandable to the general public (citizens as stakeholders) is a significant challenge, not limited to the domain of policy modeling, however very often it is a challenge that needs to be addressed for the desired outcome. Connected opportunity is scalability, when the existing simulation platforms and solutions can be adapted to be used on a larger scale or in a different environment, which can also lead to significant savings (especially useful in the developing countries).

Ibrahim and Larsson (2019, p. 215) discuss the results of the SENSE4US project and argue that there is still “a lack of decision support tools that can provide policy-oriented modeling, simulation and decision analysis capabilities”. So, the possibilities for future work include the ways of extracting data for policy model from texts in policy reports and research literature, incorporation of information from social media, implementation of applications for stakeholder analysis (e.g., Crowdsourcing applications) (Frank, 2013). With the significant progress in machine learning and natural language processing in relation to social media analysis, incorporating these technologies into the policy modeling solutions can be a significant opportunity to increase the accuracy and relevancy of the models.

Finally, projects such as RISK see the possibility for the application of the created model or theoretical framework in a different area or domain. This is a distinct opportunity for more theory-oriented projects, however the successful realization would often require cross-disciplinary collaborations and involvement of a wider range of stakeholders.

It is also worth discussing the research directions not identified in the projects, most notably the ethical issues. Policy models may use personal data as an input. In this regard, it is important to ensure the proper ethical standards and anonymization of data. More generally, policies affect the lives of people, and decisions based on policy modeling inform the adoption of policies. Hence, attention should be paid to avoid possible bias in the input data and ensure transparency. Thus, while absent from the results of the project analysis, ethical considerations remain an important future research direction that needs to be considered in relation to policy modeling.

5. Conclusions, limitations and further research

While policy modeling has come a long way and there have been significant advances in relation to both the development of technology and computational capacities on one hand, and approaches and theoretical policy models on the other, there are many aspects that still need further research. For example, a thorough literature review on policy modeling to complement and enrich the highly valuable research of Ruiz Estrada (2011) by extending the scope of papers beyond the Journal of Policy Making to include research from different disciplines and publication outlets is such a future research need. Such substantial research would also lead to a domain model to conceptualize and describe the breadth and depth of the research on policy modeling. Such a research could also contribute to build up a classification of types and categories of policy models used along respective modeling paradigms.

This paper introduced an analysis of 16 policy modeling projects along with literature review of the topic. Along this, four core characteristics of policy modeling have been identified (RQ1) and future research directions in policy modeling have been extracted and synthesized (RQ2).

As accurate modeling and simulation is complex, policy models are created with a specific application area in mind. And even then, the creation of a working prototype or a tool requires significant effort and funds, which often can only be realized in projects over several years, which can be seen in the most projects focusing on the specific policy area and policy-making stage. Projects that involve more than one policy area are usually more theory-oriented, not offering applied modeling or simulation solutions. Such specialization of the projects in turn sets restrictions on the re-use of the created tools and adaptation of models to other policy areas or sometimes even other geographical areas. Another related big challenge is the communication of the results of modeling, and implementing these results into real policy changes. The inherent complexity of the domain may make the communication of the findings difficult, thus limiting potential benefits of the modeling. In this regard, further research is needed on stakeholder involvement in policy development and modeling: to collect relevant information, to establish trust and to ensure that the results of policy modeling are used in real-world policy making processes. Additionally, research in visualization of the results and process of policy modeling may address further issues in relation to communicating with the key stakeholders and society as a whole.

Compared to the earlier research roadmapping efforts, technological issues seem to be less significant research targets in the analyzed projects, which shows the notable technological advances in the field of policy modeling and also changes in the focus of some of the projects. Real-time data analysis is an example, which was not included in earlier research roadmaps, but is a hot topic now with increased popularity of the smart city concept (see e.g., Nam and Pardo, 2011; Ojo et al., 2015, p. 13; Caragliu and Del Bo, 2019 for descriptions).

The findings of the analysis in the paper are not comprehensive and some of the less evident aspects or research needs of the policy modeling domain were likely not identified due to the limitations of the research methodology. Still, the number and variety of the projects in the analysis allowed to identify the major future research directions in the field and can serve as a useful overview of the state of research in policy modeling.

Another limitation of this research is that, while the effort was made to provide as complete picture of existing policy modeling projects, it is possible that some projects were not included in the analysis. For some projects, the deliverables and final reports were not readily available, which produced gaps in the table. Future research may include an in-depth comparison of the models and simulation approaches designed or discussed in the projects similar to the approach taken by Majstorovic et al. (2015). Alternatively, the research can be extended by using papers as a unit of analysis, as many more models and approaches are not directly connected to a specific project, but result from the ongoing research at the research groups and centers.