Using social media data to assess the impact of COVID-19 on mental health in China

Abstract

Background

The outbreak and rapid spread of coronavirus disease 2019 (COVID-19) not only caused an adverse impact on physical health, but also brought about mental health problems among the public.

Methods

To assess the causal impact of COVID-19 on psychological changes in China, we constructed a city-level panel data set based on the expressed sentiment in the contents of 13 million geotagged tweets on Sina Weibo, the Chinese largest microblog platform.

Results

Applying a difference-in-differences approach, we found a significant deterioration in mental health status after the occurrence of COVID-19. We also observed that this psychological effect faded out over time during our study period and was more pronounced among women, teenagers and older adults. The mental health impact was more likely to be observed in cities with low levels of initial mental health status, economic development, medical resources and social security.

Conclusions

Our findings may assist in the understanding of mental health impact of COVID-19 and yield useful insights into how to make effective psychological interventions in this kind of sudden public health event.

Introduction

The epidemic of coronavirus disease 2019 (COVID-19) has become a severe public health crisis (Sohrabi et al., 2020). In addition to the adverse impact on physical health, the outbreak and rapid spread of COVID-19 have also brought about mental health problems among the public, such as anxiety and depression (Holmes et al., 2020; Liu et al., 2020; Wang et al., 2020a). To capture the psychological problems during the COVID-19 epidemic, online questionnaires and surveys are widely used in ongoing studies (Gao et al., 2020; Hao et al., 2020; Rajkumar, 2020; Wang et al., 2020b, c). Researchers detect the symptoms of mental illness and identify risk factors by asking participants to answer well-designed questions and report their characteristics. The challenge of these traditional methods is that it is difficult to monitor the mental health condition in real time and understand its dynamic changes (Areán, Ly, & Andersson, 2016; Gruebner et al., 2017). The large-scale and real-time data generated by the widespread use of social media provide an approach to overcome these problems. By applying natural language processing (NLP), the expressed sentiment of tweets posted on the online social media platforms could be extracted from the text (Conway & O'Connor, 2016; Gohil, Vuik, & Darzi, 2018). This is an effective indicator to reflect psychological response and has been increasingly used for measuring the mental health status (Gruebner et al., 2016; Liu, Zhu, Yu, Rasin, & Young, 2017; Wongkoblap, Vadillo, & Curcin, 2017).

In this study, we investigate how the COVID-19 epidemic affected mental health across China's cities using social media data from Sina Weibo, the largest microblog platform in China. The data included around 13 million geotagged tweets in mainland China between 1 January 2020 and 1 March 2020 from active Weibo users. For each tweet, we conducted the sentiment analysis to extract the expressed sentiment using the open-source NLP technique from Baidu (Tian et al., 2020a). Then we measured the daily mental health status for a city by calculating the median sentiment value based on tweets in that city on each day (Zheng, Wang, Sun, Zhang, & Kahn, 2019), which ranges from 0 to 1 with 0 indicating a strongly negative emotion and 1 indicating a strongly positive emotion. To quantify the causal effect of COVID-19 epidemic on mental health, we employed a difference-in-differences (DiD) approach (Dimick & Ryan, 2014; Donald & Lang, 2007; He, Pan, & Tanaka, 2020). The treatment group was defined as cities that have reported the first COVID-19 case. Following the definition, our analyses included 324 treated cities and 35 control cities. Specifically, the COVID-19 was first detected in Wuhan city in December 2019, but the pathogen was unknown and the severity was underestimated at first (Li, Wang, Xue, Zhao, & Zhu, 2020; Tian et al., 2020b; Yu et al., 2020). Therefore, the situation in Wuhan is different from other cities in treatment group and we excluded it from our data. We controlled daily air pollution and weather conditions since these factors could also affect the expressed sentiment on Weibo tweets (Zheng et al., 2019).

Our study has the following strengths and contributions. First, the scale of our data is large, which are collected based on a 20-million-level active user pool in Sina Weibo (Hu, Huang, Chen, & Mao, 2020). All geotagged tweets posted by these active users during our study period were selected and used to construct a national panel data set, covering 359 cities in China. Second, the DiD approach helps us to infer the causal relationship between COVID-19 epidemic and mental health. For example, since the occurrence of COVID-19 in China almost coincided with the Chinese Spring Festival (25 January 2020), it is hard to distinguish the effect of the national holiday from the impact of COVID-19 epidemic just by before-after comparison (Li et al., 2020; Su et al., 2020). In our DiD strategy, cities without COVID-19 cases can serve as the counterfactual and various confounding factors can be controlled in the model. Therefore, we could plausibly identify the causal impact of COVID-19. Third, our comprehensive data set allows us to examine whether COVID-19 disproportionately affects the mental health among different segments of the population, categorized by gender and age, and investigate whether the psychological effect varies across different types of cities. Relying on these strengths, our findings may assist the policymakers to understand the impact of COVID-19 on mental health in detail using social media data and provide useful implications for the psychological interventions when facing this kind of public health crisis.

Materials and methods

Data

Social media data

Sina Weibo (https://www.weibo.com/), the Chinese equivalent of Twitter, is the largest microblog platform in China. Large-scale data access is difficult for Weibo because of the limitation of its application programing interface (Shen et al., 2020). To solve this problem, our Weibo data were obtained based on a pool of 20 million active users (Hu et al., 2020), which was selected from over 250 million Weibo users generated by snowball sampling. We collected all geotagged tweets of these active users between 1 January 2020 and 1 March 2020. Geotagged tweets mean that the users share their location information based on the exact latitude and longitude when they post these tweets. Then, 13 million geotagged tweets in mainland China during our study period were selected, including the gender and age information of their users.

Using these data, we conducted our sentiment analysis by applying the SKEP model (Tian et al., 2020a) from Baidu Senta (an open-source python library) published in 2020, which integrated sentiment knowledge into pre-trained models and achieved new state-of-the-art results on most of the test data sets. For each tweet, the sentiment analysis could return two probabilities representing the intensity of the positive and negative emotions based on the text, and the sum of these two probabilities is 1. In this study, we used the positive probability as a measurement of the user's mental health status at the time when the tweet was posted. The daily mental health status for a city is measured by calculating the median positive probability for that city on each day (Zheng et al., 2019). This city-level mental health status ranges from 0 to 1 with 0 indicating a strongly negative emotion and 1 indicating a strongly positive emotion. We also calculated the mean value of the positive probabilities and used it to measure city-level mental health status in our robustness check.

COVID-19 epidemic data

In this paper, the treatment group was defined as cities that have reported the first COVID-19 case. We collected the date of the first confirmed case in each city from the official websites of local health commissions. COVID-19 was first detected in Wuhan city in December 2019, but the pathogen was unknown at first and human-to-human transmission was not verified. Therefore, the situation in Wuhan is different from other cities in the treatment group and we excluded Wuhan from our data. Finally, our data included 324 treated cities and 35 control cities. The geographical distribution of these 359 cities and the cumulative number of treated cities by 1 March 2020 are presented in Fig. 1 and online Supplementary Fig. S1.

Fig. 1. Geographical distribution of 359 cities. As of 1 March 2020, 324 cities (Wuhan was excluded) have reported COVID-19 cases and the rest is the control group, including 35 cities.

Air pollution and weather data

Since air pollution and weather conditions could affect the expressed sentiment on social media (Zheng et al., 2019), these confounders should be controlled in our analyses. In China, the air quality index (AQI) is a composite measure of air pollution, constructed by the concentrations of PM_2.5, PM₁₀, SO₂, CO, O₃ and NO₂ (Zhong, Yu, & Zhu, 2019). A lower AQI means better air quality. We collected daily city-level AQI data from the Ministry of Ecology and Environment in China (https://datacenter.mee.gov.cn/). City-level weather data including daily mean temperature, wind speed, rainfall and cloud, were obtained from an online platform called Huiju Data (http://hz.zc12369.com/), which collects data from China Meteorological Administration.

Socio-economic data

To explore the heterogeneity across cities, we collected the cities' socio-economic status from the 2019 China City Statistical Yearbook (National Bureau of Statistics of China, 2019). These data contain city-level statistics reflecting economic development, medical resources and social security level.

Summary statistics

The summary statistics of different variables between 1 January 2020 and 1 March 2020 are reported in online Supplementary Table S1. The average city-level mental health status was 0.6397, with a standard deviation of 0.0684. We observed a decline in the mental health status of treated cities after reporting COVID-19 cases.

Models

We used a DiD model to identify the impact of COVID-19 epidemic on mental health in China. This model could estimate the relative change in mental health status between the treated and control cities, specified as follows:

(1)$$Y_{it} = \alpha + \beta \cdot {\rm COVID}\_19_{it} + \gamma \cdot X_{it} + \mu _i + \pi _i + \varepsilon _{it}\;\;\;\;$$

where Y _it represents the mental health status in city i on date t measured by the social media data. ${\rm COVID}\_19_{it}$ denotes whether the COVID-19 epidemic has occurred in city i on date t, and takes the value 1 if the city has reported the first COVID-19 case and 0 otherwise. X _it are the control variables, including AQI, mean temperature, mean temperature squared, rainfall, wind speed and cloud. μ _i indicate city fixed effects, which are a set of city-specific dummy variables. By introducing the city fixed effects, we can control for time-invariant confounders specific to each city, such as geographical conditions and short-term economic level. π _i indicate the date fixed effects, which are a set of dummy variables accounting for shocks that are common to all cities on a given day, such as the Chinese Spring Festival Spring and nationwide policies. In this specification, both location and time-fixed effects are included in the regression, so the coefficient β estimates the difference in mental health status between the treatment cities and the control cities before and after the occurrence of the COVID-19 epidemic. We expected β to be negative, as both the coronavirus itself and counter-COVID-19 measures such as lockdown could harm the mental health (Fu et al., 2020; Pfefferbaum & North, 2020).

The underlying assumption for the DiD estimator is that treatment and control cities would have parallel trends in mental health status in the absence of the COVID-19 event. Even if the results show that mental health status declines in treated city after the occurrence of COVID-19, the results may not be driven by the epidemic, but by systematic differences in treatment and control cities. For example, if treatment cities have a decreasing trend in mental health status and the control cities not, this could also drive the results. Although we cannot observe what would happen to mental health in the treated cities if the COVID-19 epidemic did not occur, we can still examine the parallel trends in mental health for both groups before the COVID-19 epidemic and investigate whether the two groups are comparable. To achieve this goal, we adopted an event study approach using the following relative time model (Burtch, Carnahan, & Greenwood, 2018; Greenwood & Agarwal, 2016; Liu & Bharadwaj, 2020):

(2)$$ \eqalign{ Y_{it} = \alpha & + \mathop \sum \limits_{m = k, m\ne -1}^M \beta ^k\cdot {\rm COVID}\_19_{it, k} + \gamma \cdot X_{it} + \mu _i + \pi _i \cr & + \varepsilon _{it}\;\;\;\; } $$

where ${\rm COVID}\_19_{it, k}$ are a set of dummy variables, which indicate the treatment status at different periods (weeks). Here, 7 days (1 week) are put into one bin (bin m ∈ M), so the high volatility of the daily mental health level could not affect the trend test (He et al., 2020). We omit the dummy for m = −1 (1 week before the event), so the coefficient β ^k measures the difference in mental health status between the treatment and control cities in period k relative to the difference 1 week before the treatment. This specification could not only test the parallel trend assumption, but also examine whether the impact of COVID-19 epidemic fades out over time. If the pre-treatment trends are parallel, the coefficient β ^k would be not significantly different from zero when k ≤ −2. The psychological effect of COVID-19 would fade out over time during our study period if we observe that β ^k is negative at first and then becomes not significantly different from zero in subsequent periods when k ≥ 0. In all analyses, the standard errors were clustered at the city level.

Results

The impact of COVID-19 epidemic on mental health

We estimated the relative change in mental health status between the treated and control cities by using equation (1). The results reported in Table 1 indicate that the occurrence of COVID-19 had a significantly negative impact on mental health. After reporting the first COVID-19 case, the mental health status measured by the median sentiment value of Weibo tweets in treated cities declined by 0.0097 relative to cities without COVID-19 cases when controlling air pollution, weather conditions and a set of fixed effects (in column 2). We also observed that the inclusion of air pollution and weather variables made the R ² of our DiD model become higher, which increased the fit performance of the regression. This finding is consistent with our expectation which expects that both the coronavirus itself and subsequent transmission control measures such as lockdown could aggravate the mental health status (Fu et al., 2020; Pfefferbaum & North, 2020).

Table 1. Effect of COVID-19 on mental health

Note. Due to some missing values of air pollution and weather data, the numbers of observations in the two columns are not the same. Standard errors are clustered at the city level and shown in parentheses. *p < 0.05; **p < 0.01; ***p < 0.001.

We conducted some additional analyses to validate the robustness of our main finding. We first excluded cities in Hubei province, the worst-hit region in China during the epidemic. Similar results suggest that the psychological effect of COVID-19 is not only driven by these cities (online Supplementary Table S2). We conducted further robustness check by replacing the dependent variable with the mean value (instead of the median value) of the expressed sentiment on Weibo tweets. The finding is still consistent (online Supplementary Table S3). After the occurrence of COVID-19, our Weibo data may contain more tweets from those people who are more sensitive to COVID-19 since they participate more in social media during the epidemic, which may affect our results. To address this issue, we conducted a tweet-level analysis and controlled user-fixed effects. The dependent variable is the expressed sentiment of each tweet, and the independent variable is a binary variable, which is equal to 1 when the tweet was posted after the city reported the first COVID-19 case, and 0 otherwise. We still observed similar results when controlling the user-fixed effects (online Supplementary Table S4).

Although our results are consistent across various robustness checks, it is still possible that the decrease in the mental health status may be driven by some unobserved differences between the treatment and control groups. If this were true, the psychological effect of COVID-19 would be statistically significant with any ordering of COVID-19 occurrence in treatment cities. Thus, we carried out a random implementation model to determine how likely it was that a random occurrence of COVID-19 would yield an aggregate effect size comparable to our true estimates (Greenwood & Agarwal, 2016; Greenwood & Wattal, 2017; Liu & Bharadwaj, 2020). First, we randomly assigned the pseudo-presence of COVID-19 to our treated cities, and then estimated the effect of the random occurrence of COVID-19 using equation (1) to obtain the coefficient for the pseudo-treated (denoted as β _pseudo). This procedure was repeated 1000 times and then we calculated the mean and standard deviation of β _pseudo. The Z-score was used to examine the difference between our original estimate β (reported in Table 1) and the mean of β _pseudo. In addition, we also replicated the whole procedure on all cities (instead of only the treatment group). The results show that the mean of β _pseudo is close to 0 and significantly different from the true estimate β (online Supplementary Table S4). Therefore, our original estimation is not spurious, and the causal claim is strengthened as well.

Test for pre-treatment parallel trends

To test whether the parallel trends assumption in our DiD model is violated, we adopted an event study approach and fitted a relative time model [see equation (2)] (Greenwood & Agarwal, 2016; He et al., 2020; Liu & Bharadwaj, 2020). This model could measure the difference in mental health status between treated and control cities in each period relative to the difference 1 week before the treatment. The estimated coefficients and their 95% confidence intervals are plotted in Fig. 2. We find that the estimated coefficients are not significantly different from 0 before the occurrence of COVID-19, suggesting that there is no systematic difference in trends between treated and control cities before the treatment. This implies that parallel trends assumption of our DiD model would be reasonable in the absence of the COVID-19 epidemic.

Fig. 2. Effect of COVID-19 on mental health over time. The estimated coefficients from equation (2) and their 95% confidence intervals (error bars) are shown. The dummy variable indicating 1 week before the occurrence of COVID-19 is omitted from the regression. Thus, the difference in mental health status between treated and control cities 1 week before the treatment is set to be zero and serves as the reference point. The estimation signifies the difference in mental health status in each period relative to the difference 1 week before the treatment.

In addition to the test for pre-treatment parallel trends, this relative time model could also examine whether the impact of COVID-19 epidemic on mental health status among the public changes over time. We expect that the estimated coefficients are negative and statistically significant at first after the occurrence of the epidemic, and then become not significantly different from 0 in subsequent periods because the psychological effect is likely to fade out when the epidemic tends to be stable during our study period. All results shown in Fig. 2 are consistent with our expectation except for the estimated coefficient in 1 week after the occurrence of COVID-19 epidemic. The unexpected result is probably due to the temporary ‘pulling together’ or ‘honeymoon period’ phenomenon (Gordon, Bresin, Dombeck, Routledge, & Wonderlich, 2011; Madianos & Evi, 2010; Matsubayashi, Sawada, & Ueda, 2013). That is, to fight with COVID-19, social connectedness, community cohesion and mutual support are enhanced, mitigating the negative psychological impact of the epidemic. More future studies are needed to explore the underlying mechanisms. Besides, we also obtained similar results when using the mean sentiment value as the dependent variable in equation (2) (online Supplementary Fig. S2).

Heterogeneity across different subpopulations and cities

To investigate the heterogeneous effects of COVID-19 epidemic on mental health, we conducted two types of heterogeneity analyses. In the first analysis, we examined whether different subpopulations are disproportionately affected by the occurrence of COVID-19. To do so, we divided the tweets data into subgroups according to the self-reported gender and age information of the Weibo users. Then we calculated the daily city-level mental health status for each subgroup by the same method mentioned before and estimated equation (1) separately. The disproportionate effects on mental health among different gender and age groups are shown in Fig. 3. We find that women are more susceptible to the psychological impact of COVID-19. After the occurrence of this epidemic, the negative effect on mental health is more pronounced among teenagers (younger than 18 years old) and older adults (older than 45 years old). Collectively, these results imply that, by and large, the adverse psychological outcomes caused by COVID-19 are more likely to be observed among the vulnerable groups.

Fig. 3. Heterogeneous effects of COVID-19 on mental health across different subpopulations. Each row means a separate regression using equation (1) on the corresponding subsample. We use the gender and age information of Weibo users to separate our data. The estimated effects of COVID-19 and their 95% confidence intervals (error bars) are plotted.

In the second heterogeneity analysis, we investigated whether the psychological effect of COVID-19 varies across different types of cities. We first collected socio-economic statistics reported in the 2019 China City Statistical Yearbook (National Bureau of Statistics of China, 2019) for the cities in our data, such as regional GDP and the number of hospitals. For the initial mental health status, we measured it by using the median sentiment value of tweets posted in each city during the first week of our study period. Then our data were partitioned into high and low based on the median value for each factor. For example, if the regional GDP in a city is lower than the median GDP, it falls into a low GDP group, otherwise a high GDP group. The psychological effect was estimated separately using equation (1) based on data in each subgroup. We expect that the deterioration of mental health after occurrence of COVID-19 is more likely to be observed in cities with low levels of economic development, medical resources and social security, since these areas own poor financial, material and human support in the fight against this epidemic and the provision of mental health service. Our conjecture is confirmed in Fig. 4a–c: the negative effect is more notable in the low group. In Fig. 4d, we find that cities with poor initial mental health status are more susceptible to the psychological impact of COVID-19, so more related measures should be taken in these areas after the occurrence of epidemic.

Fig. 4. Heterogeneous effects of COVID-19 on mental health across cities. These heterogeneity analyses are divided into four categories: economic development (a), medical resources (b), social security (c) and initial mental health status (d). Data are partitioned into High and Low based on the median value for each factor. Each row means a separate regression using equation (1) on the corresponding subsample. The estimated effects of COVID-19 and their 95% confidence intervals (error bars) are plotted.

Discussion

In addition to the physical harm, the outbreak and rapid spread of COVID-19 has caused some additional effects, such as the improvement in air quality (He et al., 2020) and the changes in mental health (Pfefferbaum & North, 2020). To fully understand the influence of this unprecedented event, we need to quantify these additional effects and this paper is an essential component. Our findings in this study could contribute to answering three research questions related to mental health impact of COVID-19.

First, does COVID-19 has a causal effect on the psychological changes reflected on social media in China? Applying a DiD approach on a comprehensive panel data set, our analyses reveal a deterioration in mental health status caused by the occurrence of COVID-19 among users on Sina Weibo, the Chinese equivalent of Twitter. This finding is robust in a set of robustness checks. However, the mental health measure is derived from the people who post tweets on social media. Although this group contains a large number of people, we acknowledge that it is not randomly drawn from the full population. Little children and people who are very old are less likely to use Sina Weibo (Wong, Merchant, & Moreno, 2014; Zheng et al., 2019), and these individuals in fact may be more vulnerable to the psychological effect of COVID-19 (Jiao et al., 2020; Yang et al., 2020). Therefore, our results may underestimate the overall adverse effect of COVID-19 epidemic on the mental health status of a representative sample of the full population. In addition, due to the fear of legal consequences, from the accusation of spreading rumors, the self-censorship of sensitive conversation in Weibo could exist among the general public, especially during the early stage of COVID-19. This phenomenon may lead to bias in our data. Moreover, although Weibo is the largest microblog platform in China, it cannot be neglected that some social media users may employ a virtual private network to access overseas media information. Thus, the heavy reliance on one single data source, that is Weibo, may not be an ideal case. Further studies considering the diversity of data sources are needed to validate our results and take a more all-rounded look at the mental health impact of COVID-19.

Second, does the psychological impact of COVID-19 fade out as the epidemic tends to be stable over time? The results of our relative time model show that the effect of COVID-19 on mental health is likely to fade out during our study period. But, our results do not allow us to draw any conclusion that the psychological effect will disappear in the long term although the epidemic in China has been almost controlled. The end of this COVID-19 epidemic could not mean the disappearance of its effect on mental health among the public. The socio-economic effects caused by COVID-19, such as economic recession and social inequalities, are also harmful to our mental health status in the post-epidemic era, which might last for a long period (Kathirvel, 2020). Besides, a group of people may have difficulties in adjusting back to normal life when the epidemic is over, such as the students (Lee, 2020). For example, during the COVID-19, students have to adapt themselves to online study. However, if the schools are reopened, they have to readjust to the traditional classes. The frequent shifts in lifestyle could bring about further psychological problems. Moreover, subsequent vaccine problems could also cause mental health changes among the public. Therefore, the assessment of psychological impact in post COVID-19 stage may be complex and need further rigorous analysis.

Third, does the effect of COVID-19 on mental health vary across different population groups and cities? Our first heterogeneity analysis shows that the psychological effect is more pronounced among women, teenagers (younger than 18 years old) and older adults (older than 45 years old). Thus, we should pay more attention to these vulnerable people when providing mental health services. Nevertheless, we are unable to capture the heterogeneous effects on little children and people who are very old, due to the limitation of the age distribution of Weibo users (Wong et al., 2014; Zheng et al., 2019). Traditional questionnaires and surveys may be better methods to investigate the psychological impact on these population groups. Besides, it would be interesting to further categorize the population into caregiver group who is living with the elderly, or non-caregiver group who is living separately with their elder families if the data are accessible. Future studies could examine whether the COVID-19 epidemic would pose further stress to this group of caregivers when the mortality rate is so high among the elderly (Jordan, Adab, & Cheng, 2020). The results of the second heterogeneity analysis imply that mental health impact of COVID-19 is more likely to be observed in cities with low levels of initial mental health status, economic development, medical resources and social security. Therefore, people with poor mental health status before COVID-19 and those living in underdeveloped areas that lack financial, material and human support could suffer from more serious mental health problems. These findings may help the government to grasp the point in decision making. For example, when allocating public resources and providing mental health support, giving priority to these areas at high risk may make the inputs produce more benefits. Additionally, the heterogeneity analysis also reminds us of the important role of the economic state, medical resources and social security in mitigating the negative psychological effect.

Conclusion

We conclude this paper by pointing out several directions for future research. First, we only focus on the text of tweets. However, some tweets contain other types of valuable data, such as pictures and videos, which provide rich information (Pittman & Reich, 2016). More further studies are needed to extract sentiment from them and take advantage of these data to measure the psychological response more accurately. Besides, bad mental health status could lead to subsequent severe consequences, such as suicide behavior (Sher, 2020). This suggests the need to collect related data to quantify the causal impact of COVID-19 on these adverse outcomes. In addition, the outbreak of COVID-19 simultaneously brought about infodemic (Zarocostas, 2020). The rapid spread of misinformation through social media platforms may also affect mental health, and assessing this phenomenon is a meaningful task (Casigliani et al., 2020). We believe that our findings in this study, together with future research, will assist in the understanding of mental health impact of COVID-19 and yield useful insights into how to make effective psychological interventions in this kind of sudden public health event.