The need for a broad perspective when assessing value-for-money for out-of-hours primary care

Abstract

Background:

Out-of-hours primary care (OOH-PC) has emerged as a promising solution to improve efficiency, accessibility, and quality of care and to reduce the strain on emergency departments. As this modality gains traction in diverse healthcare settings, it is increasingly important to fully assess its societal value-for-money and conduct thorough process evaluations. However, current economic evaluations mostly emphasise direct- and short-term effect measures, thus lacking a broader societal perspective.

Aim:

This study offers a comprehensive overview of current effect measures in OOH-PC evaluations and proposes additional measures from the evaluation of integrated care programmes.

Approach and Development:

First, we systematically identified the effect measures from published cost-effectiveness studies and classified them as process, outcome, and resource use measures. Second, we elaborate on the incorporation of ‘productivity gains’, ‘health promotion and early intervention’, and ‘continuity of care’ as additional effects into economic evaluations of OOH-PC. Seeking care affects personal and employee time, potentially resulting in decreased productivity. Challenges in taking time off work and limited access to convenient care are often cited as barriers to accessing primary care. As such, OOH-PC can potentially reduce opportunity costs for patients. Furthermore, improving access to healthcare is important in determining whether people receive promotional and preventive services. Health promotion involves empowering people to take control of their health and its determinants. Given the unscheduled nature and the fragmented or rotational care in OOH-PC, the degree to which interventions and modalities provide continuity should be monitored, assessed, and included in economic evaluations. Continuity of care in primary care improves patient satisfaction, promotes adherence to medical advice, reduces reliance on hospitals, and reduces mortality.

Conclusion:

Although it is essential to also address local settings and needs, the integration of broader scope measures into OOH-PC economic evaluations improves the comprehensive evaluation that aligns with welfare gains.

Background and aims

Global commitment to out-of-hours primary care (OOH-PC) and recognition of the importance of this healthcare format is increasing (Hong et al., 2020; Steeman et al., 2020). Additionally, research has demonstrated that its implementation has the potential to improve the quality of care, optimise efficiency, and reduce the strain on emergency departments (EDs) (Mohsin et al., 2007; Guttmann et al., 2011; Whittaker et al., 2016; Hong et al., 2020; Allen et al., 2021). Although there is no consensus on the definition of ‘appropriate’ or ‘inappropriate’ use of the ED, several studies find that many medical problems presented in the ED could be managed in a primary care setting, as they do not always require specialist care (Derlet & Ledesma, 1999; Carret et al., 2007; Durand et al., 2011; Kraaijvanger et al., 2016).

OOH-PC is operationally defined in terms of time frame, as primary care delivered on weekdays outside business hours, approximately from 6:00 PM to 8:00 AM, during weekends or public holidays (O’Donnell et al., 2015). Such care is classified as unscheduled, meaning that no appointment or forward planning is arranged beforehand (O’Donnell et al., 2015). However, there is a discussion in the literature on whether out-of-hours care should only provide urgent care or include non-urgent care as well (Keizer et al., 2015; O’Donnell et al., 2015; Barnes et al., 2022). Nonetheless, for this paper, we adopted a broad and commonly accepted definition of OOH-PC based solely on the time frame, specifically excluding care delivered in the ED or other secondary or tertiary levels.

OOH-PC encompasses several models of delivery, such as practice-based services in which physicians within an individual or group practice look after their own and each other’s patients during OOH times (Berchet & Nader, 2016). Another model is general practice cooperatives (GPCs), which are large-scale self-organised groups of general practitioners (GPs) providing out-of-hours care in a region (Berchet & Nader, 2016; Colliers et al., 2017). Additionally, there are also retail or medical clinics located within grocery stores or pharmacies, typically staffed by nurses or other health professionals (Berchet & Nader, 2016).

Many countries have now adopted OOH-PC (Steeman et al., 2020), necessitating its inclusion in the decision-making process regarding future investments. For decisions that aim to maximise welfare, the perspectives of the healthcare payer, the hospital, and society are informative, in which the latter incorporates the full range of relevant costs and effects, including patient long-term outcomes and productivity losses (Byford & Raftery, 1998). However, evidence on the economic evaluation of OOH-PC service delivery is scarce and limited, despite numerous suggestions to robustly assess the causal impact of improving access to primary care on the use of other services, outcomes, and costs (WHO & UNICEF, 2022). Some studies in this domain have focused only on the estimation of cost implications (Brogan et al., 1998; Scott et al., 2004; O’Dowd, 2006; van Uden et al., 2006; Eichler et al., 2010; Moth et al., 2013; Lin et al., 2021; Morreel et al., 2022). Others have incorporated effects by including immediate health system effects that manifest within a short time frame of patient presentation, such as admissions or practice attendance (Hansen & Munck, 1998; Lattimer et al., 2000; Moore et al., 2021; Flaherty et al., 2022). Although OOH-PC is commonly considered a short-term intervention, it is also crucial to capture long-term societal outcomes (Deidda et al., 2019).

This study aims to first provide a comprehensive overview of the effect measures currently used in the economic evaluations of OOH-PC interventions and, second, propose an additional set of societal effect measures to capture the broader economic value of this healthcare service. The additional measures relevant to OOH-PC are identified and discussed based on an overview of economic evaluations of integrated care programmes.

Approach and development

Systematic literature search

We consulted PubMed, SCOPUS, Web of Science, EconLit, Cochrane reviews, NHSEED, and Health Technology Assessment databases for articles that performed economic evaluations of OOH-PC. See Appendix A for details on the search strategy. Eligibility criteria were agreed upon by two researchers (JP and LW) using the PICOTS framework (Population, Intervention, Comparator/Context, Outcome, Timing, and Study Design/Setting) to define inclusion and exclusion criteria (Table 1).

Table 1. PICCOTS

The online search identified 2717 unique results; we identified one additional article by screening the references of seven systematic reviews on OOH-PC effectiveness. After removing 299 duplicates, we selected 101 from the title and abstract screening, and finally, we included 13 in this overview after reading the full text. See Appendix B for the PRISMA diagram.

We evaluated the quality of the 13 studies using the Consensus Health Economic Criteria (CHEC) list, which is a 19-point checklist researchers use to evaluate the reporting and methodology of published economic evaluations (Evers et al., 2005). Of the studies, 10 received a score greater than or equal to 75%, while three received a score of between 51% and 74%. (See Appendix C for the evaluation approach and the assessment results). Note that scores cover both the quality of the study conducted and the completeness of reporting. However, if a paper does not discuss context because it seemed obvious, we flagged the corresponding item as not applicable and excluded the item from the denominator following the CHEC guidelines.

The 13 included studies comprised 11 research papers that evaluated the cost-effectiveness and efficiency of various OOH-PC healthcare interventions and modalities (Broekman et al., 2017; Chesteen et al., 1986; Flaherty et al., 2022; Flynn, 1998; Hansen & Munck, 1998; Lattimer et al., 2000; Moe et al., 2019; Moore et al., 2021; Patwardhan et al., 2012; Poole et al., 1993; Sterner et al., 2012) and two protocol papers proposing full economic evaluations (Reuter et al., 2016; Wijers et al., 2012). Since our focus was on the effect measures and not on the results, these protocols are sufficient and studies that published the results were not included.

Among these studies, five compared alternative ways of providing OOH-PC (Broekman et al., 2017; Hansen & Munck, 1998; Lattimer et al., 2000; Reuter et al., 2016; Wijers et al., 2012), while seven compared OOH-PC with ED or with a ‘no OOH-PC’ scenario (Chesteen et al., 1986; Flaherty et al., 2022; Flynn, 1998; Moe et al., 2019; Moore et al., 2021; Poole et al., 1993; Sterner et al., 2012). Additionally, one study compared OOH-PC with urgent care centres, primary care physicians, ED, and without intervention (Patwardhan et al., 2012). In total, these studies evaluated six types of OOH-PC: telephone support, nurse-delivered care, telemedicine, home delivery, late night/weekend/holiday clinics for alcohol intoxication, after-hours clinics/family practices or urgent care centres, and jointly operating ED and GP care. See Appendix D for a further description of the studies.

Measures previously used in the economic evaluation of OOH-PC

With the formalisation of OOH-PC implementation, numerous published studies have concentrated on its effectiveness. In the upper section of Table 2, we have compiled common effectiveness measures based on the findings of seven systematic reviews (Foster et al., 2020; Fry, 2011; Garratt et al., 2007; Hong et al., 2020; Huibers et al., 2011; Leibowitz et al., 2003; O’Donnell et al., 2015). In contrast, the literature on cost-effectiveness is comparatively limited and systematic reviews are absent (O’Donnell et al., 2015). Therefore, we performed a systematic search of the literature and retrieved 13 studies as outlined in Section 2.1. The lower section of Table 2 compiles the effect measures based on the findings of these studies.

Table 2. Overview of effect measures related to the process, patient outcome, and healthcare resource use in the effectiveness and economic evaluation of OOH-PC interventions

Table 2 shows the effect measures related to process, patient outcomes, and healthcare resource used to evaluate OOH-PC interventions’ effectiveness and economic evaluations. According to the Donabedian paradigm, we categorise the measures into three groups according to the consensus-based entity submission types outlined in the Centers for Medicare & Medicaid Services Measures (CSM) Inventory Tool (Measures Management Systems, 2023). A process measure refers to the evaluation of specific steps, procedures, and environment essential to provide quality care (Measures Management Systems, 2023). When a process measure is managed effectively, the likelihood of achieving the desired outcome increases. For example, more general access to healthcare services can contribute to reduced mortality (Measures Management Systems, 2023). A patient outcome measure focuses on assessing a patient’s health status or any changes in their welfare resulting from healthcare interventions (Measures Management Systems, 2023). Examples of outcome measures include mortality rates and gains in quality-adjusted life years (QALYs). A measure of healthcare resource use quantifies the utilisation of healthcare services expressed in terms of natural units (Measures Management Systems, 2023). This encompasses various aspects, including diagnoses, procedures, or healthcare encounters, and can be exemplified by metrics such as the number of GP visits (Measures Management Systems, 2023).

As shown in the upper part of Table 2, effectiveness studies mostly include the use of healthcare resources, a few patient outcome measures, and a limited number of process measures. Similarly, the lower part of Table 2 shows that OOH-PC economic evaluations focus on the use of healthcare resources and a few patient outcome measures, while process measures remain relatively underutilised.

The need for a broader scope for OOH-PC

OOH-PC falls within the umbrella of integrated care, and it is necessary to measure and evaluate its broader effects at various levels. Given this, OOH-PC has been shown to improve access to care for those in need, reduce ED visits, and promote efficiency (Dent, 2010; Lowe et al., 2005; Piehl et al., 2000). Moreover, integrated care represents care that is coordinated across professionals, facilities, and support systems, is continuous over time, and is responsive to people’s needs, values, and preferences (Schneider et al., 2011). It encompasses treatment plans, methods, and models of care that enable improvement in patient experience, promote efficient service delivery, reduce healthcare expenditures, and improve population health through enhanced coordination and continuity of care (Plochg et al., 2009; Shaw et al., 2011). These types of interventions impact various outcomes at various levels, necessitating the measurement and evaluation of multiple outcomes (Tsiachristas et al., 2016). Furthermore, they alter existing care processes and pathways and impact providers, patients, and communities (Baxter et al., 2018), henceforth the recommendation for a broader evaluation. A broad scope of evaluation has been applied to various areas of integrated care for public decision-making (Nolte & Pitchforth, 2014).

Table 3 presents an overview of the effect measures used previously or recommended for use to measure the effects of integrated care in economic evaluations. We derived this information from two systematic reviews and several individual research studies (KPMG, 2018; Nolte & Pitchforth, 2014; Rocks et al., 2020; Steuten et al., 2006; Tsiachristas et al., 2013). Like Table 2, the measures are categorised into process, healthcare resources, and patient outcome. Although healthcare resource use measures are important indicators of health system performance, it is also essential to consider patient-centred outcomes, including population health, patient, or community well-being. The combination of patient-centred and process-related measures allows for a more comprehensive evaluation.

Table 3. Overview of effect measures related to the process, patient outcome, and healthcare resource use in the economic evaluation of integrated care interventions

Additional key measures relevant for OOH-PC economic evaluations

Comparison of Tables 2 and 3 reveals that integrated care interventions employ effect measures beyond immediate outcomes. These effects are more extensive, covering broader health system and patient effects compared to those used in OOH-PC. Consequently, we advocate for the inclusion of additional effects in the evaluation of OOH-PC to improve the comprehensive understanding of the economic value. Additionally, it will create synergy between OOH-PC and conceptually similar integrated care interventions, which can complement and/or substitute OOH-PC.

Following this mapping of the effects of integrated care initiatives and OOH-PC, we identified eight additional effect measures for integrated care initiatives not yet used in OOH-PC. These measures include two process measures, one healthcare resource use measure, and five patient measures. Specifically, two identified process measures are ‘continuity of care’ and ‘health promotion’, while the additional measure of using healthcare resources is ‘informal caregiver time’. The innovative patient outcome measures for OOH-PC studies are ‘diagnosis and disease management’, ‘frequency of complications and exacerbations’, ‘self-efficacy’, ‘patient travel and time’, and ‘productivity loss’. While all eight measures are relevant for the economic evaluation of OOH-PC, in this study, we narrow our discussion to three key measures where the out-of-hours aspect is crucial and distinguishes itself from within-hours care: ‘continuity of care’, ‘health promotion’, and ‘productivity loss’. The added value of each selected measure is discussed in the following paragraphs. Among the five outcome measures not highlighted here, such as ‘diagnosis and disease management’, ‘patient travel and time’, or ‘informal caregivers time’, we believe that the difference with regular primary care is limited. However, all identified outcome measures that are currently absent from published economic evaluations of OOH-PC offer valuable insights into clinical outcomes or well-being, while depicting costs for the healthcare payer, the patient, or society.

Table 4 shows a comprehensive summary of the effect measures for OOH-PC evaluations. It lists additional measures proposed from integrated care and those used in previous OOH-PC evaluations. The effect measures that we emphasise in the main text are coloured red. In the subsequent paragraphs, we elucidate and delve into the proposed supplementary effect measures, offering illustrative examples from related domains.

Table 4. Comprehensive summary of effect measures related to process, patient outcomes, and healthcare resource use for OOH-PC evaluation

Productivity losses averted (opportunity cost of seeking healthcare)

Economic evaluations must consider the opportunity cost of seeking care, which can affect personal and workforce time, resulting in productivity loss (Ray et al., 2015; Weinstein et al., 1996). From the patient’s perspective, productivity loss contributes to wage loss and causes undesirable experiences when seeking care (Handley & Hollander, 1999; NHS Primary Care Commissioning, 2012). Additionally, the difficulty of taking time off work and the lack of access to convenient care are two factors often cited as barriers to accessing regular-hour primary healthcare (Friedberg et al., 2010; NHS Primary Care Commissioning, 2012). In contrast, these two factors are facilitators of increased use of OOH-PC (Zhou et al., 2015). Therefore, OOH-PC offers patients a convenient avenue to access primary care outside of working hours, helping to avoid wage and time losses. This is also relevant for caregivers responsible for caring for the disabled, elderly, and young children. OOH-PC grants these individuals the ability to refrain from taking time away from work and sacrificing their caregiving obligations concerning their care recipients’ healthcare needs. This also applies to students and school-aged children. Research indicates that students who rely on public clinics often miss entire days of school per appointment (Kornguth, 1990). Furthermore, research findings indicate that children frequently use OOH-PC in European settings (Huibers et al., 2011), and its use may help reduce school absenteeism (Institute of Medicine, 1997). Productivity gains in this group can be achieved by minimising absenteeism among students and school-aged children. Therefore, it is important to include the productivity loss avoided in the evaluation of OOH-PC, as it represents an important positive gain for patients, caregivers, students, and school-aged children. This gain is an important influence in this domain, and it is crucial not to underestimate this potential advantage. Additionally, it is important to consider the productivity loss caused by using OOH-PC. Furthermore, opportunity costs are important due to the increased recognition of patient-centred care (Baker, 2001). Subsequently, this has led to an emphasis on innovative healthcare delivery options that reduce the time burden (Ray et al., 2015).

Productivity gains have been included in economic evaluations of telemedicine (Patel et al., 2023; Snoswell et al., 2020), which is a conceptually similar intervention to OOH-PC. Although not delivered specifically outside of working hours, telemedicine also provides convenient, patient-centred care that improves access to care and reduces time burden. Additionally, it seeks to reduce the inefficient use of higher-level facilities, including EDs (Sun et al., 2020). By providing the necessary services by electronic means, telemedicine serves individuals who have difficulties in making appointments or may not have the time, resources, or motivation to travel to traditional clinics (Zhang et al., 2021). Eliminating these logistical difficulties results in decreased patient productivity loss in the economic evaluation of telemedicine (Agha et al., 2002; Kubes et al., 2021). Estimation of productivity losses or gains in economic evaluations is complex and uses various measures. These include the human capital approach, friction cost approach, or multiplier approach, which can account for productivity in natural units, opportunity costs, and replacement costs (Jiang et al., 2022). Because there is no universally accepted measure, the choice of approach must depend on existing national guidelines while considering available information in a specific context.

Access to health promotion services and early intervention

Health prevention involves taking action to avoid the onset of disease and associated risk factors (Radhakrishnan, 2017). These actions include vaccinations, prophylaxis, education of people about behavioural and medical health risks, and disease detection (Radhakrishnan, 2017). Similarly, health promotion involves empowering people to take control of their health and its determinants (Radhakrishnan, 2017). Examples of these include dietary and nutritional interventions, interventions to mitigate social ills such as domestic violence, and interventions to promote sexual and reproductive health, such as family planning services (Radhakrishnan, 2017).

Reliable access to health-promotive and preventive services in primary healthcare is important for improving health outcomes and reducing the financial burden of treating diseases (Hostetter et al., 2020). Providing these services promotes timely diagnosis and treatment, increasing the chances of success (World Health Organization, 2018). An efficient and effective primary healthcare system must provide promotive and preventive services (Van Weel & Kidd, 2018). However, despite the benefits, many settings do not have optimal access to these recommended services (Borsky et al., 2018; Levine et al., 2019).

Having access to primary healthcare is a crucial factor that determines whether people receive promotive and preventive services (Xu, 2002; Friedberg et al., 2010). OOH-PC is efficient in improving access to primary care (Hong et al., 2020). Therefore, OOH-PC represents a suitable resource for those who would otherwise not seek care at a regular-hour health facility for the necessary services and check-ups. This is particularly so in settings or modalities where OOH-PC permits or is used for such services. Without such care and services, delayed diagnoses and treatment can lead to complicated disease management (World Health Organization, 2018). Given the profound role that OOH-PC can play in delivering promotive and preventive services, a robust economic assessment must consider incorporating these benefits. Their omission could produce a conservative estimate of the economic benefits of OOH-PC.

Improved access to preventive and promotive health services has been used in conceptually similar interventions. For example, a recent cost–benefit analysis (CBA) from Australia examined nurse-led primary healthcare facilities and explored how they impacted the provision of promotive and early interventions (KPMG, 2018). Similarly, economic evaluations of mobile and community clinics included the adoption of preventive and promotive services (Liu et al., n.d.; Oriol et al., 2009; Stillmank et al., 2019).

In the context of a societal evaluation, it has been proposed that the value of healthcare services should not be limited to their value to patients alone (Culyer et al., 2018). While the well-being of patients remains a focal point, it is imperative to incorporate the perspectives of healthcare providers to gain a comprehensive understanding of the practical implications of the delivery of healthcare services. This is of relevance for health promotion and preventive services, as providers often face heavy workloads (Smits et al., 2014; Royal College of General Practitioners, 2019). This can limit their capacity to offer non-urgent care on an unscheduled basis, as they must prioritise urgent care. Thus, in addition to the potential benefits to the patient, it is necessary to quantify the deterrent effects, if possible, and consider the viewpoint of service providers as relevant and appropriate measures of value regarding health-promoting and preventive services in the OOH-PC domain. However, striking a balance between the potentially opposing needs and perspectives of patients and providers is a complex but indispensable task.

Measuring the impact of an intervention on the acceptance of promotive or preventive services involves assessing whether the number of visits related to these services has changed since the implementation of that intervention (KPMG, 2018). However, establishing the long-term economic impact of preventive care and health promotion is challenging for several reasons. At the population level, it can be challenging to isolate the impact of individual interventions because the impacts are bundled or extend beyond healthcare. Additionally, some settings may encounter challenges with missing or inferior data, or practical/ethical/privacy problems when linking data from several sources or contacts (Colliers et al., 2016). In cases where data are accessible, the estimation of QALYs can be considered using clinically preventable burden scores, as has been demonstrated in previous research (Maciosek et al., 2017; Stillmank et al., 2019).

Continuity of care

Relationship continuity of care is the maintenance of continuous and sustained relationships between patients and healthcare professionals (Gulliford et al., 2006; Hill & Freeman, 2011). Research findings demonstrate that establishing a good, trust-based, and long-term relationship with a primary care physician of one’s choice can lead to improved health outcomes, better quality of care, and reduced healthcare expenses (Starfield et al., 2005). On the other hand, management continuity refers to care systems facilitated by integration, coordination, and the sharing of information between different providers (Gulliford et al., 2006; Hill & Freeman, 2011). Continuity of care in primary healthcare, both in relationship and in management, is beneficial for patients, clinicians, and health systems. It leads to increased patient satisfaction, improved care for chronic patients, increased use of preventive care, promoted adherence to medical advice, reduced dependency on hospitals, and reduced mortality (Gray et al., 2018; Sidaway-Lee et al., 2019).

Despite its benefits, continuity of care remains low in some primary care settings and is rarely measured (Sidaway-Lee et al., 2019). Measurement and comparison of continuity rates among providers can, in turn, improve continuity (Kontopantelis et al., 2013). Furthermore, ensuring patient-centredness and continuity of care are crucial attributes of a well-functioning primary care system. Therefore, it is important to establish whether an intervention demonstrates a reasonable level of continuity.

In the out-of-hours domain, relationship continuity may not be present. The feasibility of primary care physicians providing 24-hour care is influenced by several factors, including provider preferences, patient needs, existing market supply, and financial considerations (O’Malley et al., 2012). Therefore, if OOH-PC is not provided by the usual physician, a mechanism is necessary to facilitate the sharing of health information and systematic notification procedures to maintain information continuity between providers to prevent fragmentation of care (O’Malley et al., 2012). However, the exchange of patient information is of interest for all healthcare stakeholders and interventions, especially in the context of integrated care. Interaction is needed, for example, for a scheduled follow-up with the usual primary physician or when complications and/or exacerbations of previously treated conditions arise (O’Malley et al., 2012). The degree to which primary care interventions and modalities provide continuity of relationship and care should be monitored, assessed, and included in economic evaluations.

Incorporating continuity of care into economic evaluations is not common. Nonetheless, an Australian study assessed continuity of care as a qualitative aspect of its economic evaluation (KPMG, 2018). The study evaluated changes in continuity of care after the implementation of a primary care nurse-led clinic with services offered by a nurse and a collaborating GP, who visited the site bi-weekly (KPMG, 2018). This study revealed improved continuity of care by allowing the community to follow up on health-related issues before seeing a specialist and by acting as a link between the community and other health service providers in the wider region (KPMG, 2018).

Several measures of continuity of care exist, including the Usual Provider of Care index, the Bice-Boxerman Continuity of Care index, the Herfindahl Index, and the Sequential Continuity of Care Index (Pollack et al., 2016). Additionally, the measurement of coordination between professionals of different disciplines could use tools such as the relational coordination survey (Gittell, 2011).

Discussion and research implications

Various economic evaluation techniques can determine the value-for-money of OOH-PC while including the three proposed additional effects. Cost-effectiveness analyses (CEA), which measures effects in natural units, has the potential to individually incorporate the three proposed effect measures. Continuity of care, for instance, could be assessed using natural measures such as the change in the proportion of sequential patient visits at the same provider (Roos et al., 1998). It can also be assessed by changes in interprofessional team communication and relationship scores (Hustoft et al., 2018). The use of monetary CBA, which converts all health-related effects to monetary terms (Drummond et al., 2015), could incorporate all three additional effect measures. To monetise continuity of care, one can assign a value to its anticipated outcomes, such as avoided hospital use or prompt treatment initiations. However, it is important to avoid the potential issue of double counting when considering these outcomes. Moreover, CBA could potentially be more time-consuming than CEA and is criticised for assigning monetary values to health states (Tsiachristas et al., 2016). Cost–utility analysis can incorporate all additional measures. By considering health gains through health outcomes such as mortality, the inclusion of continuity of care is feasible (Tsiachristas et al., 2016). However, as with CBA, caution is needed to prevent double counting. A cost-consequence analysis (CCA) presents a range of outcomes alongside costs; therefore, it could integrate all the proposed additional effects. Due to its clarity, CCA is used to inform decision-making (Mauskopf et al., 1998) but is criticised for its inability to rank alternative interventions based on their effectiveness (Perkins et al., 2015). Multi-criteria decision analysis (MCDA) is a systematic comparison of different alternatives by considering multidimensional factors (Baran-Kooiker et al., 2018). MCDA could potentially incorporate all proposed effect measures for the evaluation of OOH-PC. However, MCDA presents the challenge of assigning weights to effects based on the preferences of stakeholders within a specific setting (Marsh et al., 2018). It is necessary to further explore the applicability of MCDA approaches to OOH-PC.

Productivity costs often have a strong impact on cost-effectiveness outcomes (Krol & Brouwer, 2014). Therefore, whether and how to include them in economic evaluation is a debate that has been ongoing for several years (Krol et al., 2013). Some argue that the inclusion of productivity costs raises equity concerns, as interventions aimed at the employed produce more favourable cost-effectiveness outputs compared to interventions aimed at the unemployed (Lensberg et al., 2013). However, excluding productivity costs due to equity concerns is contested because other cost types, such as medical costs, also discriminate across different population groups, such as between the young and the old (Krol et al., 2013). To accommodate equity concerns, it has been suggested to report productivity gains in non-monetary units such as days or hours gained or lost (Drummond et al., 2015). Additionally, equity concerns can be addressed by evaluating productivity gains or losses for the unemployed using shadow prices that consider the opportunity costs associated with unpaid work activities, including household work, shopping, and childcare (Drummond et al., 2015). On the other hand, there is a lack of standardisation and consensus regarding the methodology for measuring productivity costs (Jiang et al., 2022; Krol & Brouwer, 2014). Recently, recommendations considered the use of instruments that include both paid-work productivity losses and those related to unpaid work (Krol & Brouwer, 2014). These include the iMTA Productivity Cost Questionnaire (iPCQ) or the Valuation of Lost Productivity (Krol & Brouwer, 2014). The omission of productivity costs in economic evaluations may partly be due to some national health economic guidelines that prescribe a health system perspective (Jiang et al., 2022). However, many economic evaluations taken from a societal perspective still exclude productivity costs (Jiang et al., 2022; Krol et al., 2016). This suggests a potential bias in the selection of cost types, and decision-makers need to be mindful of their inclusion or exclusion whenever the perspective is (partially) societal (Krol et al., 2013). However, decision-makers should also be mindful of whether health-related quality of life (HRQoL) has already been factored in, as it accounts for the effects of productivity gains or losses on an individual (Jiang et al., 2022). Including both HRQoL and productivity gains or losses may result in duplicate counting (Jiang et al., 2022). In response to the debates surrounding the inclusion of productivity gains or losses, there is a suggestion to present two scenarios of cost-effectiveness results, one with and another without productivity (Pritchard & Sculpher, 2000).

The implementation of OOH-PC on a large scale may face challenges due to shortages in the health workforce (Velgan et al., 2023). These shortages already make it difficult to recruit healthcare providers to perform regular contractual hours, let alone out-of-hours (The Scottish Government, 2015). Because of these shortages, a trade-off between regular and out-of-hours is likely. The provision of OOH-PC can, on the one hand, attract healthcare providers for higher pay (Broadway et al., 2017; Longden et al., 2018), leaving a gap in regular-hour care. On the other hand, OOH-PC may not interest all providers due to, for example, the impact on their work–life balance (The Scottish Government, 2015). Whether OOH-PC will threaten the sustainability of regular-hour primary care practices remains uncertain. Therefore, the aim of improving access to regular-hour primary care while concurrently improving out-of-hour continuity of care through OOH-PC requires careful balancing and consideration in future research.

Due to the unscheduled nature of OOH-PC, there is a great diversity of care provided compared to regular-hour primary care (NHS: Health Education England, n.d.). As a result, there is a growing need for adapted training and career development to ensure that providers have the right skills to handle the increasingly challenging and complex environment (GP Training: Urgent and Unscheduled Care (Including Out-of-Hours), 2022). These skills include the ability to handle medical, surgical, and psychiatric emergencies out-of-hours, the ability to make appropriate referrals to hospitals and other professionals, to manage personal time and stress, and to maintain personal security and awareness of environmental security risks (Royal College of General Practitioners, 2019; GP Training: Urgent and Unscheduled Care (Including Out-of-Hours), 2022). Additionally, providers need to be informed about the prevailing governance approaches due to the existing links between OOH-PC, ambulance services, and EDs (Royal College of General Practitioners, 2019). Consequently, there is a recommendation for multidisciplinary teams for out-of-hours care (Royal College of General Practitioners, 2019). Therefore, it is essential that resource planning for the primary care workforce considers the degree of training needed to deliver OOH-PC effectively. However, additional training resources may not always be available, which can negatively affect OOH-PC provision.

While OOH-PC serves as a viable alternative to ED care, it has the potential to trigger a supply-induced surge in healthcare utilisation during non-office hours (Longden et al., 2018). This influx of patients outside of regular office hours may result in increased demand for healthcare resources, which can lead to additional healthcare expenses from the healthcare payers’ perspective (Morreel et al., 2022). Moreover, the provision of OOH-PC may not always be a feasible option for healthcare providers due to factors such as higher clinical indemnity insurance costs. In the United Kingdom, for instance, providers face elevated indemnity insurance expenses for out-of-hours services compared to their regular office hours counterparts (NHS England and the Department of Health, 2016).

During the identification of innovative measures for the evaluation of OOH-PC, methodological challenges were encountered. First, given our scope on economic evaluations of OOH-PC, we did not conduct a systematic search for studies that focused only on the effects of OOH-PC, nor for studies that examined economic evaluations of integrated care. However, we made a conscious effort to prioritise the scientific relevance of the sources that we utilised and attempted to mitigate potential biases by using recent systematic reviews and by conducting a thorough snowballing of the references included in the selected contributions. The second challenge concerns the identification of additional effect measures in the current multidimensional and multi-objective framework, which often involves engagement with different stakeholders to develop standardised measures. However, the present study constitutes a tool that can effectively facilitate the establishment, development, and advancement of economic evaluation mechanisms for OOH-PC in line with integrated care initiatives.

The use of additional effect measures can present certain obstacles. Determining the suitability of potential measures for a given situation necessitates an evaluation of their significance. This empirical evaluation can be difficult and time-consuming, as it relies on a comprehensive assessment of previous context-specific evidence regarding potential impacts.

Conclusion

In this paper, we discussed effect measures for conducting broad welfare-gain-driven economic evaluations of OOH-PC by drawing on experience from integrated care programmes. A focus on resource use measures can be too limiting in the OOH-PC domain, where a wide range of outcomes are relevant from the health system and patient perspectives. In this regard, we identified three relevant effects not yet considered in previous economic evaluations of OOH-PC. These are ‘productivity loss’, ‘health promotion and early intervention’, and ‘continuity of care’. This proposal of additional effects is neither comprehensive nor exhaustive but serves to highlight how to broaden the economic evaluation of OOH-PC by considering additional processes and patient outcomes related to the out-of-hours context. Determining what to include or exclude depends on the specific context, considering the evaluation perspective and the strength of existing evidence supporting the significance of an effect measure within that context.