Challenge of forecasting demand of medical resources and supplies during a pandemic: A comparative evaluation of three surge calculators for COVID-19

Abstract

Ever since the World Health Organization (WHO) declared the new coronavirus disease 2019 (COVID-19) as a pandemic, there has been a public health debate concerning medical resources and supplies including hospital beds, intensive care units (ICU), ventilators and protective personal equipment (PPE). Forecasting COVID-19 dissemination has played a key role in informing healthcare professionals and governments on how to manage overburdened healthcare systems. However, forecasting during the pandemic remained challenging and sometimes highly controversial. Here, we highlight this challenge by performing a comparative evaluation for the estimations obtained from three COVID-19 surge calculators under different social distancing approaches, taking Lebanon as a case study. Despite discrepancies in estimations, the three surge calculators used herein agree that there will be a relative shortage in the capacity of medical resources and a significant surge in PPE demand if the social distancing policy is removed. Our results underscore the importance of implementing containment interventions including social distancing in alleviating the demand for medical care during the COVID-19 pandemic in the absence of any medication or vaccine. The paper also highlights the value of employing several models in surge planning.

Introduction

The outbreak of the new coronavirus disease 2019 (COVID-19), caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), started in Wuhan City, China, and has rapidly spread all over the world. The newly identified coronavirus has 96% homology with the genetic sequence of β-coronaviruses; which include SARS-CoV and the Middle-East respiratory syndrome CoV (MERS-CoV) [1, 2]. Despite similarities, SARS-CoV infection was limited to specific geographic areas, while COVID-19 has been declared as a global pandemic by the WHO on 11 March 2020. As of 17 September 2020, there are 29 737 453 COVID-19 confirmed cases, 937 391 deaths and around 213 countries or territories affected with cases around the world [3].

COVID-19 has not only caused mortality, but has alsoput tremendous pressure on healthcare systems and led to shortages in personal protective equipment (PPE) [4, 5]. In the Middle East and North Africa Region (MENA) countries including Lebanon, the COVID-19 pandemic is leading to a health crisis and worsening the spike in demand for healthcare resources and supplies [6]. After detecting the first COVID-19 case in Lebanon, authorities took fast action and declared an emergency state in the country. Therefore, mitigation measures have been taken into effect early in March, while gradually being lifted as of 10 May. Yet, the activities in the airport were restarted at one-fifth capacity as of 1 July 2020 [7]. This fast action helped in the initial containment of the disease [8]. Unexpectedly, after the massive Beirut's port explosion on 4 August that caused hundreds of deaths, thousands of injuries and destroyed half of the city's medical centres, the country witnessed a surge in COVID-19 infections. This urged the Lebanese authorities to impose a partial lockdown which included the closing of some businesses, banning public gatherings and enforcing midnight curfew [7]. However, the two-week lockdown was prematurely eased due to the economic crisis [6]. At the time of writing this paper, despite the spike in COVID-19 cases (a total of 169 472 confirmed cases and 1379 deaths as of 27 December 2020) [9], mitigation policies such as the lockdown, travel ban and the stay-at-home have been lifted in Lebanon while keeping social distancing and personal protective measures.

To monitor the strain on healthcare systems in each country, it is essential to closely follow the demand for hospital beds, intensive care unit (ICU) beds and ventilators [10]. Therefore, a growing number of models have been established around the world to aid in forecasting COVID-19 deaths, cases and the demand for medical supply (including ventilators, hospital beds and ICU beds, the timing of patient surges and more) [11–13]. While simple models apply the user's inputs on the local population and current status of COVID-19, more sophisticated models permit the user to modify other parameters (e.g. social distancing changes) that affect the trends. Some of the models match projected cases to existing capacity to project when and where a caseload surge will surpass capacity [14]. Data-driven modelling approaches have also appeared for the COVID-19 pandemic, in which statistical and machine-learning models are used for projecting cases, hospitalisation, deaths and the impact of social distancing [15]. Unfortunately, forecasting what is most likely to occur in the upcoming weeks during the COVID-19 pandemic is not available for all countries including some European countries or few states in the USA [11]. Where available, different models are providing widely varying numbers of needed medical resources and/or supplies which often lead to an incorrect distribution of what is available due to inconsistency in numbers. For example, the Centers for Disease Control and Prevention has reported that COVID-19 outbreaks in parts of the USA have resulted in surges in hospitalisations and ICU patients [16]. However, providing accurate predictions of the healthcare system capacity peak demand is controversial due to the scarcity and/or unreliability of data in addition to challenges associated with forecasting the effects of the rapid changes in mitigation policies [17]. So far, the efforts to accurately model any emerging outbreak's trajectory for the upcoming days are limited due to variabilities in assumptions and parameters including social distancing [17–19]. Accordingly, the use of a single forecasting model may not precisely predict how the pandemic evolves [20].

Regardless of all the challenges, COVID-19 has put forecasting at the top of global public policymaking and developing effective preventive strategies [21, 22]. Since there is no ‘gold standard’ for predicting thresholds, the reasonable evaluation of the outputs of various forecasting models has remained an open question [23]. In this study, we highlight this challenge by comparing the projected demand for medical resources and supplies from three surge calculators for COVID-19 taking Lebanon as our case study. To this end, we adapted an available statistical model for estimating the daily impact of COVID-19 on hospital services based on the COVID-19 Hospital Impact Model for Epidemics (CHIME). The model was modified to incorporate longer projection periods and different social distancing policies (%). We then compared the hospital beds, ICU beds and PPE demand over 200 days projected by the three surge calculators (a) CHIME PPE calculator, (b) WHO COVID-19 Essential Supplies Forecasting Tool (COVID-19 ESFT) and (c) our own developed American University of Beirut Medical Center (COVID-19 AUBMC Surge needs) calculator. The results of these calculators differ depending on the parameters and assumptions implemented within to generate the forecasted data. Despite the discrepancies in estimations, the three surge calculators used herein consistently agree on trends demonstrating that social distancing policy can help reduce the demand for medical resources and supplies amid the COVID-19 pandemic.

Methods

Data collection

Case counts, population and hospital capacity data

We used the WHO and the Lebanese Ministry of Public Health (MoPH) websites to identify data on the estimated Lebanese population in 2020 (6 825 000) and confirmed COVID-19-infected individuals and death cases, respectively. Data on inpatient beds, ICU capacity and mechanical ventilators were obtained from the Lebanese MoPH website (Table 1). The data for the COVID-19 AUBMC calculator regarding PPE demand were collected based on historical and current data estimates from the American University of Beirut Medical Center (AUBMC), presuming these data as a nationwide reference. In most of the calculations, numbers derived from Lebanon were used where available.

Table 1. Input parameters (including display, hospital, spread/contact and severity) used for forecasting healthcare demand in Lebanon during the COVID-19 pandemic through the CHIME Model/calculator, COVID-19 AUBMC and WHO COVID-19 ESFT calculators

Statistical input parameters

CHIME model and modifications

Our estimations were based on the CHIME model that was initially developed by the Predictive Health at Penn Medicine in the USA [24], which permits healthcare systems to enter data about their population and modify the assumptions about the COVID-19 spread and behaviour [24]. The tool runs a modified susceptible-infected-recovered (SIR) model, a traditional epidemiological forecasting technique, to estimate the number of new COVID-19 hospital admissions per day. The SIR model calculates the theoretical number of people infected with an infectious disease over a period of time in a closed population with the Kermack−McKendrick model being the simplest of all SIR models [25, 26]. The CHIME model was built to help hospital systems to accurately project the needed resources during the COVID-19 pandemic, mainly hospital beds, ICU beds and ventilators [27]. The model accounts for four main parameter categories: (1) the hospital parameters, (2) spread and contact parameter, (3) severity parameters and (4) display parameters. The model allows integrating some information about social distancing policies that are emplaced by governments [27].

In this study, we have introduced some modifications to the CHIME application to make it compatible with the input parameters available for the projections in Lebanon.

The modifications were threefold:

1. 1. Automate the data projection process:

The application was updated allowing the user to enter the input parameters into an ‘Excel workbook’ and obtain the generated census compiled together in the same workbook. This alleviates the data processing part where the user needs to obtain each projection separately. It also enables faster simulations for different input parameters and most importantly different social distancing measures (Table 2).

1. 2. Allow projections over a longer period:

Table 2. The four different social distancing policy scenarios presumed, along with their subsequent mitigation dates, to estimate the capacity of hospital beds, ICU beds, ventilators and PPE demands during the COVID-19 pandemic in Lebanon over 200 days

The application initially allowed for a maximum 30-day projection period. Our contribution enables for longer periods such as 200 days, as seen herein. This provides analysts and healthcare workers (HCW) with longer forecasts, thus giving them more time to prepare for periods of peak demand.

The CHIME projects the daily and the cumulative number of hospitalised, ICU, ventilated and newly admitted COVID-19 cases, with social distancing policy percentage being the variable.

1. 3. Provide simultaneous known cases (per day) as output

The modification allows us to study the number of daily infected individuals after deducting the number of recovered cases. The number reflects the confirmed cases and does not consider possible infected individuals that are not confirmed yet. The aim behind this modification is to provide these numbers as inputs to the calculators, especially COVID-19 AUBMC and WHO COVID-19 ESFT calculators.

Our projections were based on three social distancing policy scenarios (0%, 30% and 50%) based on the National Health Strategic Preparedness and Response Plan for COVID-19 pandemic lock-down management and exit strategy implemented by the Lebanese Government [6]. The 0% social distancing scenario refers to when all restrictions and interventions are lifted without any action taken by the Government. The 30% social distancing scenario represents relaxing social distancing measures to obtain only a 30% reduction in contact rate through (1) re-opening of businesses with the public advocacy of social distancing and improved hygiene, (2) re-opening of the airport and borders, (3) partial opening of some public and private schools, (4) a nationwide overnight curfew between 9:00 p.m. and 5 a.m. (local time) and (5) banning events over 100 persons. The 50% social distancing scenario refers to a 50% reduction in contact rates where the following actions were taken: (1) imposing lockdown measure to stem the spread of the COVID-19 pandemic, (2) school and non-essential businesses closure, with individuals permitted to leave their homes wearing facemasks to buy food and medicine, (3) a nationwide travel ban, (4) a nationwide overnight curfew between 7:00 p.m. and 5 a.m. (local time), (5) and ban gatherings over 50 people, (6) and all individuals over the age of 65 are required to remain at home [28, 29].

We also simulated the strict social distancing policy (92%) based on the Wuhan-style containment [28]. The latter was performed to assess how strict ‘lock-down’ could help better contain the COVID-19 outbreak and maintain the demand for hospital beds, ICU beds, ventilators and PPE. The Wuhan-style represents the strict social distancing measures taken through the lockdown of Wuhan City, Hubei province in China during the COVID-19 pandemic [28]. This style reflects a 92% reduction in contact rates for 12 consecutive weeks to fully contain the disease through suspending flights, trains and blocking roads in addition to the rigorous implementation of public health measures [28, 30, 31].

Although Lebanon started re-opening gradually with limited capacity as of 4 May 2020, the country went again into a complete four-day lockdown starting 14 May. Therefore, we ran our estimations on 14 May 2020 (day 0) assuming the four social distancing policy scenarios. The other social distancing policy scenarios and the mitigation dates below were chosen as the per five-step re-opening plan implemented by the Lebanese Government [6] (Table 2).

PPE calculators

We have used the WHO COVID-19 ESFT calculator (WHO 2020a) and the COVID-19 AUBMC surge needs calculator, to estimate daily and total hospital beds, ICU beds, ventilators and PPE demand. The results obtained from the CHIME PPE calculator were then compared to those derived from the WHO COVID-19 ESFT and COVID-19 AUBMC surge needs calculators. As the COVID-19 AUBMC surge needs calculator was developed by our team, we chose the CHIME PPE calculator because it is compatible with the CHIME model, and the WHO COVID-19 ESFT calculator because it is widely used as a worldwide reference to estimate medical resources and supplies to respond to the current COVID-19 pandemic. Also, the CHIME and COVID-19 ESFT calculators are freely available online for use by governments, stakeholders and healthcare centres.

The CHIME PPE calculator

The CHIME PPE calculator was generated to work in parallel with CHIME-generated projections. The calculator uses forecasted patient censuses to output daily and cumulative projections for each type of PPE (including N95 masks, surgical masks, gloves, gown and eye disposable protection) quantities per day and computes the cumulative PPE predictions [32]. This tool also permits users to input their custom scenarios (standard, crisis, contingency and custom), tailored to the specific situations relevant to their hospital or healthcare system [32]. In our study, we chose the values for the standard scenario assuming we do not have exact publicly available data estimates of the exact number of staff and HCW in Lebanon. As social distancing policies and gradual lifting of restrictions are always the keep factors being debated in the intervention strategies, our estimations were based on the four social distancing policy scenarios mentioned above (Table 2). For each scenario, the projected hospitalised, ICU and ventilated censuses are inserted into the CHIME PPE calculator in addition to the daily admissions. These data are obtained from the Penn CHIME application model. Based on the provided data the calculator outputs the PPE predictions per day in addition to the total cumulative predictions over the chosen period of 200 days. Since other calculators do not forecast the daily PPE demand, we compared the average daily PPE demand from this calculator with the predictions of other calculators.

WHO COVID-19 ESTF calculator

The 2020 WHO COVID-19 ESTF calculator is developed to aid countries in estimating potential needs and supplies to respond to the COVID-19 pandemic [33]. The tool is not meant to be used as an epidemiological model, yet it has simple exponential growth and SIR case forecast options built-in [34]. The calculator is a supply forecasting tool that helps in estimating potential requirements for essential supplies including PPE (e.g. surgical masks, gloves, gowns, goggles, respirators and face shields), biomedical equipment for case management (e.g. mechanical ventilators and oxygen concentrators), drugs for supportive treatment, hygiene and IPC commodities, diagnostics and consumable medical supplies. The calculator also estimates the weekly number of COVID-19 patients classified according to severity as follows: mild, moderate, severe and critical. Severe and critical cases are admitted to ICU and require oxygen and ventilation, respectively. Inpatient beds in this calculator refer to ICU beds occupied by the critical and severe COVID-19 cases per week and not day. Therefore, the estimated peak for inpatient and ICU beds will be the same throughout the chosen period.

Although the tool is suited for projections over a short period (12 weeks), it offers an option to enter data manually and make projections over longer periods. The WHO COVID-19-ESFT does not quantify or account for resources already available locally or those pending delivery. The calculator projects the PPE quantity per person per day for inpatient care, cases in isolation, screening and laboratory and the total daily costs (USD) of items over 28 weeks. Then, it adds the total quantity for each per day. In this study, we use the default input parameters set by the WHO COVID-19 ESFT calculator for Lebanon, including the population estimate, patients' case sensitivity, HCW and staff. To maintain consistency across the different calculators, we manually input the cumulative projected COVID-19 cases as obtained from the CHIME model application. The data are compiled in a weekly form (up to 28 weeks) to be compatible with the COVID-19 ESTF calculator. This approach guarantees that all calculators are using the same numbers for infections and the focus would be on the discrepancies/agreements between the models on the estimates of resource demand.

The COVID-19 AUBMC surge needs calculator

The COVID-19 AUBMC calculator (https://www.aub.edu.lb/fm/vmp/Pages/calculators.aspx) was developed by our team based on AUBMC and MoPH data and is implemented as an Excel file that predicts the average total of PPE needs (e.g., gloves, surgical face masks, face shields and N95 masks) per day. The estimations are based on the average number of patients admitted and tested individuals per day using actual PPE demand data collected from AUBMC. Note that we assume AUBMC policies of PPE use (such as wearing surgical masks over N95 masks, replacing surgical masks per patient and replacing N95 masks per shift) are representative of policies at the national level. The average number of admitted patients is calculated from the Penn CHIME model application projections over the whole pandemic period (200 days) using the hospitalisation percentage of simultaneously infected individuals. The average number of tested individuals is calculated by dividing the total number of tests done as indicated in (https://www.worldometers.info/coronavirus/?#countries) by the number of pandemic days. The PPE estimations in this calculator are based on average values as compared to daily/weekly projections used in the other two calculators. This calculator also projects the total number of ventilators, hospital beds and ICU beds occupied during the COVID-19 pandemic using the same inputs of the CHIME application. The calculations of the beds and ventilators are designed to estimate the availability during peak periods of the pandemic. The calculator takes as input the currently available ventilators and beds dedicated to COVID-19 patients as declared by MoPH in addition to the peak simultaneous infections and admissions as projected by the Penn CHIME model application. This information helps in forecasting possible shortages in beds or ventilators and the dates when these shortages would occur (Table 3).

Table 3. Data assumptions used to estimate peak daily hospital beds, ventilators and the average daily PPE demand using the COVID-19 AUBMC surge needs calculator

The CHIME application is based on the SIR model famous across pandemic forecasts. This application is used to provide the common forecast data used across all the calculators presented in this paper. The AUBMC calculator estimates the average PPE needs per day for admitted and tested patients as well as the total PPE needs across the chosen pandemic period. The estimate considers the equipment needed for three categories of patients: (1) on admission, (2) in the operating room and (3) ICU patients. The equipment needed per category per patient is estimated based on data collected by AUBMC and protocols followed by AUBMC medical staff. These estimates are assumed to be representative of the national level. The inpatient beds, ICU beds and ventilators needed are calculated by taking the peak simultaneous active cases and estimating the number of patients from these cases who will need to be admitted or need ventilators using the historical national level percentage of patients who get admitted (ICU or regular) and the percentage of those in ICU who would need a ventilator. The hospitalised and ICU admitted percentages can be changed by the user to account for different scenarios. For additional information about the AUBMC calculator, the readers are invited to visit the tool available at the following link (https://www.aub.edu.lb/fm/vmp/Pages/calculators.aspx) which includes detailed documentation about the inputs and formulas used to calculate the PPE, beds and ventilator needs.

Results

Based on MoPH data, from 29 March 2020, Lebanon is estimated to have 2308 ICU beds and 11 794 inpatient (hospital) beds. We assume that 32% of ICU beds and 20% of hospital beds at the national level are currently occupied by non-COVID-19 patients (Table 4). These assumptions were based on calculating the total number of beds occupied per day for the last 6 months, yet taking a percentage below AUBMC occupancy for the national level. However, this occupancy number is, in reality, higher due to the casualties from the devastating port explosion in Beirut on 4 August 2020. Also, there has been a surge in cases nationally. One reason behind this surge is that many people have been unable to follow precautionary measures, such as social distancing, during the relief efforts in Beirut. Also, some of the major hospitals have been partially or heavily damaged as a result of the huge blast at Beirut's port [35, 36]. MoPH reported that almost all COVID-19 beds are full and hospitals are running out of space for new patients by the time of writing this paper. This incident highlights the challenge of planning during a pandemic, particularly when coupled with unexpected disasters.

Table 4. Estimated total and available numbers of hospital beds, ICU beds and ventilators for the COVID-19 and non-COVID-19 patients in Lebanon

In one scenario and based on our data, assuming strict social distancing policy or the Wuhan-style (92%), the modified CHIME model estimates that out of the remaining 9435 hospital beds, a peak of 67 inpatient beds is needed by the COVID-19 cases or 0.7% of the available hospital beds. Of the remaining available 1569 ICU beds and 687 ventilators, the model estimates a peak of 15 (1%) ICU beds and 10 ventilators (1.5%) are needed by COVID-19 patients respectively, with a decreasing pattern in the estimated COVID-19 infected patients over the projected period. Assuming the same scenario, the WHO COVID-19 ESFT calculator estimates a weekly peak of 11 inpatient beds (0.1%), 17 ICU beds (1.1%) and 9 ventilators (1.3%) are occupied. The COVID-19 AUBMC calculator estimates a different peak of 71 inpatient beds (0.8%), 18 ICU beds (1.1%) and 11 ventilators (1.6%) (Figs 1a, 2 and 3). Upon relaxing social distancing measures to 50%, the CHIME model estimates the same peak capacity for the available inpatient beds, ICU beds and ventilators. Still, the COVID-19 AUBMC calculator differentially estimates that the peak of daily occupied inpatient beds increases to 94 (1%), while ICU beds and ventilators increase to 23, representing 1.47% and 3.35% of the available ICU beds and ventilators, respectively. On the other hand, the COVID-19 ESFT estimates an increase in the weekly peak demand of inpatient bed capacity to 1279 (13.75%), the ICU beds to 58 (3.7%) and the ventilators to 16 (1.31%) (Figs 1b, 2 and 3).

Fig. 1. Forecasted peak of the daily inpatient beds, ICU beds and ventilators by the three surge calculator during the COVID-19 pandemic in Lebanon over 200 days assuming the four social distancing policy scenarios (a, b, c and d).

Fig. 2. Variation in the forecasted peak capacity for inpatient beds (a), ICU beds (b) and ventilators (c) assuming the four social distancing policy using the three surge calculators over 200 days in Lebanon during the COVID-19 pandemic. All calculators show a significant increase in the number of occupied beds and ventilators upon relaxing social distancing measures.

Fig. 3. Variation in the projected percentage capacity of occupied hospital beds and ventilators upon changing the social distancing policy scenarios (%), using the three surge calculators during the COVID-19 pandemic in Lebanon. The three calculators forecast a sharp increase in the percentage of the occupied inpatient beds, ICU beds and ventilators by the COVID-19 patients when social distancing measures are relaxed to 0%.

When the social distancing policy is reduced to 30%, all the surge calculators variably show that the capacity of the daily peak of hospital beds and ventilators occupied by COVID-19 patients significantly increases. The CHIME estimates a daily peak of 4016 inpatient beds (43%), 912 ICU beds (58%) and 602 ventilators (88%) to be occupied. With a higher occupancy rate, the COVID-19 AUBMC calculator estimates a peak of 4896 inpatient beds (52%), 1224 ICU beds (78%) and 1224 ventilators. The latter indicates a shortage of 537 ventilators for COVID-19 patients in this scenario. On the contrary, the WHO COVID-19 ESFT calculator estimates a marked increase in the weekly peak of inpatient beds (2665) and ventilators (266) but a shortage of 1096 ICU beds (Figs 1c, 2 and 3). To evaluate to which extent removing social distancing practices strain hospital capacity, we ran our estimations assuming 0% social distancing. Our results show that in addition to the variability in projections, all calculators estimated that there will be a shortage in inpatient and ICU bed capacity during the 200 days. Yet, only the WHO COVID-19 ESFT calculator estimates that ventilators available can accommodate the critical COVID-19 patients needing ventilation. The CHIME model estimates that the daily peak capacity significantly increases to 113 356 inpatient beds, 25 528 ICU beds and 17 003 ventilators. Similarly, this increase was marked by the COVID-19 AUBMC calculator that estimated a peak daily capacity of 124 658 inpatient beds, 33 696 ICU beds and 36 834 ventilators. The WHO COVID-19 ESFT calculator forecasted a daily peak capacity of 11 876 inpatient beds, 11 876 ICU beds and 657 ventilators (Figs 1d, 2 and 3).

We also compared the projected demand for different PPE types (including N95, surgical masks, gloves, gowns, disposable eye protection/goggles and face shield) by the three surge calculators over 200 days assuming the same social distancing policy scenarios (%). The CHIME calculator estimates a minimum average daily demand of various PPE types when the Wuhan style (92%) policy is applied. The projected outputs include an average daily demand of 689 N95 masks, 1239 surgical masks and 13 086 gloves, gowns and disposable eye protection. When compared to the CHIME projections, the COVID-19 AUBMC calculator estimates a slightly more daily average demand of 817 N95 masks, however, it estimates an average daily peak demand of 11 516 surgical masks, 57 842 gloves and 11 811 gowns and 11 429 face shields. Interestingly, the WHO COVID-19 ESFT calculator estimates a significantly higher average daily demand values for all PPE types. The estimated average daily demand includes 75 082 surgical masks, 159 488 gloves, 42 331 gowns, 36 941 goggles and 160 face shields (Fig. 4a). The estimated average daily demand for gloves in the WHO COVID-19 ESFT calculator refers to the sum of gloves used for surgery, examination and heavy-duty. When social distancing policy changes to 50%, the forecasted average daily demand for all PPE types increases gradually to 12 023 N95 masks, 216 411 surgical masks, 228 434 gloves, gowns and disposable eye protection. This increase in the projected average daily PPE demand was also observed with the COVID-19 AUBMC and the WHO COVID-19 ESFT calculators. The COVID-19 AUBMC estimates 883 N95 masks, 13 201 surgical masks, 72 317 gloves, 14 274 gowns and 11 796 face shields. The WHO COVID-19 ESFT estimates an average daily demand of 269 432 surgical masks, 926 610 gloves, 78 503 gowns, 71 158 goggles and 280 face shields (Fig. 4b). The projected average daily demand for all PPE types by the three surge calculators was still revealing a spike in values with the 30% social distancing policy. The CHIME calculated a total average daily demand of 84 761 N95 masks, 1 525 695 surgical masks, 1 610 456 gloves, 1 610 456 gowns and 1 610 456 disposable eye protection. AUBMC estimates 1716 N95 masks, 34 705 surgical masks, 257 035 gloves, 45 709 gowns and 16 472 face shields. A significant surge in the average daily demand was also obtained with the COVID-19 ESFT calculator with an estimation of 2 786 405 surgical masks, 210 525 007 gloves, 535 175 gowns, 510 030 goggles and 8201 face shields (Fig. 4c). On the other hand, the variability in projections was more emphasised when we applied the 0% social distancing policy. The CHIME projected estimated an average daily demand of 2 701 804 N95, 48632478 surgical masks and 51 334 282 for gloves, gowns and disposable eye protection. AUBMC estimated an average daily demand of 34 517 N95 masks, 881 405 surgical masks, 7 530 079 gloves, 1 283 394 gowns and 200 594 face shields. This trend was also obtained with COVID-19 ESFT, estimating an average daily need of 26 385 273 surgical masks, 45 233 595 gloves, 1 982 045 gowns, 1 248 759 goggles and 15 433 face shields (Fig. 4d).

Fig. 4. Comparing the forecasted average daily demand for all PPE types by the three surge calculators during the COVID-19 pandemic in Lebanon for 200 days, assuming the four social distancing policy scenarios (a, b, c and d). The surge calculators forecast a significant spike in the average daily PPE demand as social distancing measures are relaxed.

We then compared the change in the forecasted PPE fold-demand from other scenarios with the Wuhan style scenario (92%) since we have the least contact rate, minimum COVID-19 cases and minimal projected hospital utilisation and PPE demands. Therefore, when we changed social policy to 50%, the CHIME estimated an average of a 17-fold increase in the daily demand for all the PPE types. However, the COVID-19 AUBMC and the WHO COVID-19 ESFT reveal a minimum of a 1- and 4-fold increase in the average PPE demand per day. This difference in PPE demand projections becomes more apparent as social distancing policy decreases to 30% where the WHO COVID-19 ESFT and the CHIME estimate up to 46 and 100-fold increase in the average daily demand for all PPE types, respectively. Although the COVID-19 AUBMC calculator projected a substantial increase in the average daily demand for all PPE types, this pattern in fold increase was not observed. The calculator, however, estimates only a 4-fold increase in the average daily demand for all PPE types. The difference in estimations was more obvious with the relaxed social distancing scenario (0%). While the CHIME estimates a significant increase in the average daily demand to 3000 folds for all PPE types, the COVID-19 AUBMC and the WHO COVID-19 ESFT forecast around 100- and 200-fold increase in the average daily demand for all PPE types, respectively (Fig. 5).

Fig. 5. The fold change in the forecasted average daily demand for all PPE types by the three surge calculators at the 50%, 30% and 0% social distancing scenarios compared to the Wuhan style scenario (92%). All surge calculators used herein show a sharp increase and an excess in the average daily PPE demand upon relaxing social distancing policy measures during the COVID-19 pandemic in Lebanon up to 200 days.

Discussion

In this study, our results indicate that there is a discrepancy in the forecasted data between the three surge calculators despite the use of identical input data. Yet, the three surge calculators used consistently show that relaxing social distancing policy and mitigation measures can dramatically overwhelm hospital capacity and lead to a dramatic surge in the daily PPE demands during the COVID-19 pandemic. These results are congruent with recommendations for implementing initiatives to ‘flatten the curve’ and avoid healthcare systems from being overwhelmed through the exponential growth of the disease [37].

Since prompting social distancing policy measures have helped in reducing the pandemic's spread as shown by COVID-19 mathematical models [38, 39], our comparison was more based on testing the efficacy of this approach on the healthcare infrastructure capacity and PPE demand using the three aforementioned surge calculators.

When assuming the strict social distancing measures (Wuhan-style), the three surge calculators projected an unpronounced daily demand for hospital beds, ventilators and PPE during the pandemic. The variability in estimations between the calculators revealed the least margins of error given this scenario. When relaxing the social distancing measures to 50%, the calculators forecasted that the available hospital beds, ICU beds and ventilators are enough to accommodate the projected COVID-19 cases, yet with a very noticeable increase in the average daily PPE demand. By decreasing the social distancing policy to 30%, the calculators estimate that there is a substantial surge in the demand for PPE and hospital utilisation. At this stage, the difference becomes more apparent between the forecasted data. However, when the simulated social distancing measures are completely removed, all calculators show to some extent that the COVID-19 pandemic will put unprecedented strain on inpatient beds, ICU beds, ventilators and there will be a drastic increase in the average daily PPE demands. This underscores the urgency of implementing social distancing to help in limiting COVID-19 community transmission by reducing the circle of social contacts and the contact rate [40]. Our results are similar to those of the IHME team and Murray who forecasted that the healthcare system's capacity will be stressed in the USA and Europe during the COVID-19 pandemic and proposed some measures to increase the supply of key products and services [11, 12]. Interestingly, the data presented in this study indicate that the output predictions of all used surge calculators vary very widely upon relaxing the social distancing policy measures leading to a rise in the margins of agreement.

The discrepancy in the obtained results could be related to several reasons. For PPE, some of the items are subdivided into different categories such as the types of gloves, gowns and masks used by patients and staff as in the WHO COVID-19 ESFT calculator. Also, the WHO COVID-19 ESFT calculator gives a more detailed quantification for PPE, for example, the estimation for gloves refers to the sum of those used for surgery, examination and heavy duties. In the CHIME and COVID-19 AUBMC calculators, some of these categories are missing, which indicates that the PPE types are not being included in the calculated demand. In the COVID-19 AUBMC calculator, the N95 masks are considered to be used only in intubations. However, the WHO COVID-19 ESFT calculator does not even give estimates for N95 masks. Moreover, the CHIME and the WHO COVID-19 ESFT calculators include PPE needs for testing staff and HCW, which adds to the demand in its estimations. Considering the need for PPE in healthcare departments related to non-COVID-19 patients and used for additional precaution could also increase the calculated demand. Varying results across the three calculators are also evident for the inpatient beds, ICU beds and ventilators needed during the peak demand period. This can be related to the way each calculator assesses the severity of the cases. According to the COVID-19 AUBMC calculator, each ICU patient is assumed to require ventilation. However, the WHO COVID-19 ESFT calculator assumes that only severe and critical COVID-19 cases are admitted to the hospital and assumes that only the critical cases in ICU require ventilation, while the rest need oxygen tubes. As for the CHIME, it classifies the hospital admitted COVID-19 into three categories including hospitalised, ICU and ventilated. The percentages of beds occupied by non-COVID-19 patients (Table 4) are not used in the COVID-19 AUBMC surge needs calculator, and were only considered to estimate the available ICU and inpatient beds for COVID-19 patients. Therefore, those assumptions do not pose a limitation on the comparison in terms of relative numbers between the models, but can only affect the analysis of when an overload would occur. Also, our study is relative because the availability of beds was not monitored closely, and Lebanon has been provisioning additional beds especially after the Port explosion where several field hospitals were established. Since there is no absolute truth that we can compare to assess the accuracy, we only resort to the relative comparison. Nevertheless, this wide variability in estimations can add more uncertainty to the forecasted hospital utilisation and PPE demand and highlights the importance of maintaining social distancing in the absence of pharmaceutical intervention for COVID-19.

To be able to evaluate the surge calculators objectively, the results produced should be compared with their real known values. However, parameters used in models for forecasting the dissemination of infectious diseases are prone to uncertainties and limitations [41]. Besides, enhancing the current model forecasting abilities is directly proportional to the accuracy of the data provided [41]. In our case, we input the cumulative projected COVID-19 cases as obtained from the modified CHIME model application into all calculators to maintain consistency. Yet, in addition to using already built models, we still lack accuracy as we do not have tangible data in Lebanon on PPE consumption and the capacity of healthcare system infrastructure. Also, we are not very sure how much our input data are up-to-date and reliable especially that the healthcare capacity is subjected to change throughout an outbreak [41]. Aiming for a relatively long forecasting period (200 days) directly affects the uncertainty of predictions. While the CHIME application developers do not provide a quantitative measure of the uncertainty across the forecast period, they do mention that the projections beyond 30 days will hold higher uncertainties [42]. This is also evident in a recent study which shows that uncertainties increase when forecasting beyond 30 days [43]. The uncertainty is a property of the SIR model used by the CHIME tool developers to calculate the infection projections. These numbers are used as inputs for all surge calculators being compared. Accordingly, the effect of the uncertainty in the projections weighs equally on all calculators being evaluated and reduces the impact on the outcomes, which are relative in nature.

If we were to assess the accuracy of the CHIME forecasts and in turn the accuracy of the presented calculators, we would have to compare the forecasts with the actual national-level data. However, this is hindered by three factors: (1) the limitation of the CHIME application where it can only account for one social distancing scenario per forecasting period, (2) the fluctuating social distancing measures taken by the Lebanese government over the pandemic period which renders the true infection data incomparable with the CHIME forecasts and (3) the lack of actual national-level data on the consumptions and occupancies. It is important to note that the scope of this paper is to evaluate inconsistencies across surge calculators for the same infection forecasts. Future work is planned to evaluate the calculators over shorter periods where a consistent social distancing scenario is maintained and where some national-level data can be obtained. These data will most likely be related to the levels of occupancies of ICU and hospital beds. The results of this study were based on showing the differences in estimations done by three surge calculators for COVID-19 and a modified version of the CHIME model to measure needs for different health system resources based on the total of predicted simultaneously active cases. These calculations were carried out based on three important assets receiving significant attention worldwide: inpatient beds, ICU beds, ventilators and PPE [44, 45]. Other items and human resources required in the diagnostic and treatment chain can be forecasted including staff and HCW in the frontline of COVID-19 response, and therapeutics for supportive treatment. We believe that one of the major limitations for forecasting COVID-19 is based on the limited evidence since neither the magnitude nor duration of the COVID-19 wave is known with certainty. Another limitation is that we did not factor in PPE re-use measures such as sterilisation of used N95 masks. Therefore, the adequate management of medical resources (including PPE, beds, ventilators and healthcare providers) is highly recommended at this stage as some countries have started to experience a resurgence of COVID-19 cases as the pandemic continues to accelerate. Experience from several countries including China, South Korea and Singapore in addition to mathematical modelling has revealed that the pandemic can be contained even in the exponential growth phase using a combination of interventions [40, 46]. The latter mainly includes social and physical distancing, public awareness and wearing masks [40].

In conclusion, the surge calculators used here, regardless of the variability in outputs, can be powerful tools for measuring the impact of social distancing policy through highlighting the dangers of scaling down non-pharmaceutical public and social health measures in the absence of any vaccine or therapy against COVID-19. Characterising forecasting uncertainty can be improved by some promising avenues including methodological advances in model comparison and averaging [41]. Thus, the use of more than one model is recommended to generate more accurate and better predictions of the pandemic's evolution [47]. In other words, policymakers can use these calculators interconnected with each other based on the available data for each country to understand the pandemic from all its angles to be able to generate policymaking frameworks. This urges the need for a clear methodology that allows policymakers to decide which model is more applicable or adaptable for their context [20] and underscores the necessity of enabling calculators to be adopted to local policies and behaviours beyond social distancing. Although gaps in the present data streams provide a challenge for the current epidemic forecasting, recent breakthroughs in this field afford the possibility for refining future predictive models [41]. Since the most essential piece of the puzzle in forecasting is data or the quality of data source, we suggest that should the data be more accurate, one can provide pandemic forecasting with fewer constraints.