Understanding drivers of loneliness: machine learning insights from the HILDA survey

Abstract

Background:

Loneliness has emerged as a pervasive public health challenge. Understanding loneliness and its associated risk factors is crucial for developing interventions to address this issue effectively. This study aimed to investigate loneliness among adults living in Australia, comparing different age cohorts.

Method:

This study used 10,815, 11,234, 14,670 and 15,049 records with loneliness measurements taken at 2006, 2010, 2014 and 2018, respectively, from the Household, Income and Labour Dynamics in Australia (HILDA) survey. A supervised machine learning algorithm, CatBoost, was employed to predict loneliness. Model predictions were explained using SHapley Additive exPlanations (SHAP) and partial dependence plots across five age-based subgroups to capture life stage variations.

Results:

Mental well-being, having a life partner, social connectedness and social fulfilment were the most important predictors of loneliness at the whole-population level. Among young adults, the level of friendship fulfilment, financial satisfaction and health status were relatively strong predictors of loneliness, while loneliness in older adults was more strongly associated with spare time fulfilment, community satisfaction and the loss of loved ones. Youth who reported that they did not have a lot of friends were predicted to have a 46.5% (95% CI: 45.9%–47.2%) chance of experiencing loneliness. Seniors have a 44.9% (95% CI: 43.9%–45.8%) chance of experiencing loneliness if they were almost always not fulfilled in their spare time.

Implications:

This study underscores the need to recognise the heterogeneity of loneliness across the lifecourse and the importance of both targeted strategies and efforts to improve broader social cohesion.

Introduction

Despite decades of peace and prosperity in industrialised nations post-World War II, a concerning paradox has emerged: loneliness is increasingly recognised as a significant public health issue. Economic prosperity often masks the profound social and emotional challenges individuals face, with economic policies contributing to those challenges – particularly by increasing inequality, disrupting social connections and exacerbating isolation (Occhipinti et al., 2024). Loneliness, a facet of mental well-being, has reached epidemic levels, according to former US Surgeon General Vivek Murthy (2017). In developed countries, approximately a third of the population contends with loneliness, with 1 in 12 severely affected (Cacioppo and Cacioppo, 2018). This trend is prevalent in Australia, where loneliness was a substantial concern even before the COVID-19 pandemic (Lim et al., 2020), affecting approximately one in five Australians (Welfare, 2023). Loneliness, defined as distress due to inadequate social relationships (Hawkley and Cacioppo, 2010; Peplau and Perlman, 1982; Badcock et al., 2022), has far-reaching consequences beyond subjective feelings. It is associated with deteriorating mental well-being (Wang et al., 2018) and suicidal ideation (McClelland et al., 2020; Stravynski and Boyer, 2001), as well as increased risks of dementia, Alzheimer’s disease (Wilson et al., 2007; Salinas et al., 2022), cardiovascular diseases (Hawkley and Cacioppo, 2003) and stroke (Valtorta et al., 2016). Long-term loneliness is linked to a 26% higher risk of mortality (Holt-Lunstad et al., 2015). Additionally, loneliness is a well-established social determinant of depression throughout the lifecourse, with a sense of social sufficiency and the need for belongingness proposed as factors that modify the strength of this relationship, though the mechanism is likely to be complex (Erzen and Çikrikci, 2018). Furthermore, evidence suggests a bidirectional relationship, with depression acting as a risk factor for the development of loneliness, while concurrently, loneliness functions as a precursor to the onset of depressive symptoms (Sbarra et al., 2023). Scholarly attention has increasingly focused on other key influences on loneliness, particularly social connection. While some studies highlight social connection’s mitigating role (Victor and Yang, 2012; Guthmuller, 2022; Franssen et al., 2020; Ejlskov et al., 2018; von et al., 2020; Hawkley and Kocherginsky, 2018), others find no evidence of association (Dahlberg et al., 2022; Nyqvist et al., 2023). However, research predominantly concentrates on specific demographics, particularly the elderly and populations from North America and Europe. Loneliness research in Australia is an emerging field, often concentrating on older age groups (Ogrin et al., 2021; Steed et al., 2007; Engel et al., 2021), specific cohorts such as those with disability, cardiovascular disease or dementia (Freak-Poli et al., 2022; Moyle et al., 2011; Bishop et al., 2024), and typically employing more traditional analytic methods. Responding to the global urgency highlighted by the World Health Organization commission to address loneliness, our study employs advanced machine learning techniques to predict loneliness among adults living in Australia. By investigating complex, non-linear relationships across different age groups, our approach examines whether new insights can be derived to inform strategies for addressing the challenge of loneliness across the lifecourse.

Methods

Data

The data for our study came from the Household, Income and Labour Dynamics in Australia (HILDA) survey – a nationally representative longitudinal study tracking the demographic, economic, social well-being and health of Australian households over time (Watson and Wooden, 2021). Data were collected via a combination of in-person interviews and self-completion questionnaires. This analysis focused on adults living in Australia (aged 18 and above) and extracted data from four waves spanning 2006, 2010, 2014 and 2018, with sample sizes of 10,815, 11,234, 14,670 and 15,049 individuals (Department of Social Services Melbourne Institute: Applied Economic & Social Research, 2023), respectively. These time points were specifically chosen due to the availability of social connection and participation measurements, aligning with the key interests of our study. Merging these datasets resulted in a final dataset comprising 51,768 entries. There were 9,643 entries excluded from the original data due to missing loneliness measurements. To model loneliness across life stages, the data were divided into five age-based subgroups: youth (18–30, n = 12,277), young adults (31–40, n = 8,774), middle adults (41–60, n = 18,187), late adults (61–75, n = 9,064) and seniors (76+, n = 3,466).

Outcome measure and features

The primary outcome measure, loneliness, was originally assessed using a 7-point scale to gauge agreement with the statement: “I often feel very lonely,” with responses ranging from 1 (completely disagree) to 7 (completely agree). To simplify interpretation, this study re-coded the loneliness variable into two categories: “not lonely” for responses lower than 4 and “lonely” for 4 and above. The considered features associated with loneliness encompass two main perspectives: fundamental and social integration, with the latter comprising both micro and macro levels. Fundamental features included age, gender, highest education achieved, perceived health status, mental and emotional well-being, frequency of physical activity, current labour force status, job satisfaction and satisfaction with financial situation.

The micro level of social integration considered individual-level social relationships and fulfilment. This included having a life partner, social connection and experiencing recent significant personal losses. Social fulfilment measures, including friendship fulfilment and spare time fulfilment. The macro aspect measures broader community connection levels, including community satisfaction and community participation. Community participation involved attending events, volunteering and club membership. Additionally, the study wave of data (four time points) was incorporated as a feature to explore potential associations with loneliness. See Supplementary Materials Appendix A.1 Table A.1 for data summary and Appendix A.2 for the methods used to construct derived features.

Categorical boosting

To predict loneliness, we employed the supervised machine learning algorithm CatBoost, which utilises gradient boosting to combine multiple weak learners, like decision trees, to address residuals from previous trees. Unlike other boosting algorithms, CatBoost efficiently manages categorical features, automatically encoding them during training (Dorogush et al., 2018). Its ordered boosting technique effectively tackles data leakage and overfitting issues (Prokhorenkova et al., 2018). Recent studies have demonstrated CatBoost’s superiority over other gradient boosting methods like XGBoost and LightGBM (Sahin, 2022; Bentéjac et al., 2021). Additionally, our non-parametric tree-based model offers greater flexibility in handling complex data and relationships compared to linear models. An accessible, health-focused introduction to gradient boosting methods is provided elsewhere (Zhang et al., 2019).

Interpretation tools

To enhance the interpretability of our machine learning models, we employed the SHAP (SHapley Additive exPlanations) framework (Lundberg and Lee, 2017), a model-agnostic approach that provides explanations for model outputs by assigning SHAP values to features, elucidating their contributions to predictions. Additionally, to visualise complex relationships captured by our models, we utilised partial dependence plots (PDP) (Hastie et al., 2009), which illustrate how predicted probabilities of loneliness vary with changes in specific features. See Supplementary Materials Appendix A.3 and Appendix A.4 for detailed explanations and methodologies regarding the SHAP and PDP.

Modelling strategies

Figure A.1 (see Supplementary Materials Appendix A.5) illustrates the modelling workflow for analysing the full population. To construct models with good generalisation, we limited the potential for overfitting by performing cross-validation. We split the data into training and testing sets (70/30 ratio) using stratification to mirror the original dataset’s loneliness distribution. A recursive feature elimination based on SHAP values was employed for feature selection, retaining 10 features per model. Hyperparameters, including learning rate and L2 regularisation coefficient, were tuned using grid search and 5-fold cross-validation. The CatBoost built-in early stopping mechanism determined the optimal number of trees. Statistical modelling was performed in Python version 3.8.

Results

Exploratory data analyses

There was a monotonic increase in loneliness prevalence from 2006 (29.7% [95% CI: 28.8%–30.5%]) to 2018 (31.0% [95% CI: 30.3%–31.7%]) (Table A.2, Supplementary Materials Appendix A.6). Across age groups, prevalence formed a “W” shape, with individuals aged 31–40 and 61–75 experiencing less loneliness than those aged 18–30 and 76 and above. Seniors had the highest prevalence, with a noticeable decline since 2006, while other groups generally showed an increasing trend (Figure A.2, Supplementary Materials Appendix A.6). Loneliness rates among females consistently exceeded those of males, but the gender gap has gradually narrowed over time (2018: 29.7% [95% CI: 28.7%–30.8%], female 32.1% [95% CI: 31.1%–33.1%]; 2006: male 27.8% [95% CI: 26.5%–29.0%], female 31.3% [95% CI: 30.1%–32.5%]) (Figure A.3, Supplementary Materials Appendix A.6).

A data summary, stratified by loneliness status, is presented in Table A.1 in Supplementary Materials Appendix A.1. New composite features, “cp” and “connect,” were derived from original features related to community participation and social connectedness, respectively (Supplementary Materials Appendix A.2). Wilcoxon rank-sum tests and Pearson’s χ² tests were used to assess differences in numerical and categorical features between individuals experiencing loneliness and those who were not. Results indicated significant associations between loneliness and all features except for the time points of data collection (“Wave”).

Statistical modelling results

Figure A.4 (Supplementary Materials Appendix A.6) shows how each feature contributes to the predicted probability of loneliness for a given observation based on SHAP values. Figure A.6 (see Supplementary Materials A.4) presents PDP (in the first column) and SHAP scatter plots (in the second columns) for the top five most important features. To summarise the contribution of these features across all samples, we present beeswarm plots of SHAP values in Figure A.5 (see Supplementary Materials Appendix A.6) for the entire population and each age cohort. These features were ordered based on descending feature importance, calculated as the mean absolute value of SHAP. Each dot for a specific feature on the plot represents an observation from the data, and its corresponding value on the x-axis indicates the magnitude and direction of its contribution to the predicted probability. The top five most influential features contributing to loneliness for the entire population were mental well-being (“mental”), having a life partner (“ptnr”), friendship fulfilment (“sful1”), social connectedness (“connect”) and spare time fulfilment (“sful2”).

Mental well-being emerged as the most predictive feature of loneliness for both the overall population and age cohorts. It contributed to a maximum increase of over 40% and up to a 20% reduction in the probability of loneliness (see Figure A.5a, Supplementary Materials Appendix A.6). A non-linear relationship was revealed by the PDP in Figure A.6a (see Supplementary Materials Appendix A.6), where a sharp reduction in loneliness was evident as the mental well-being score increased from 50, levelling off at 90. The average chance of experiencing loneliness was estimated be 14.7% [95% CI: 14.6%–14.8%] if the population’s mental well-being score were improved to 90, representing a halving compared to the prevalence. Adults with poor mental well-being and less social connection were at a particularly high risk, with average predicted probabilities of loneliness reaching over 69% (Figure A.7, Supplementary Materials Appendix A.6).

Except for the youth age group, social connectedness was consistently a crucial predictor of loneliness, with a stronger emphasis as life stages progress (see Figure A.5, Supplementary Materials Appendix A.6). A distinct downward trajectory is evident in the SHAP and PDP plots (see Figure A.6c and Figure A.6d, Supplementary Materials Appendix A.6), where a higher social connectedness index was associated with lower predicted probabilities of loneliness. More specifically, the chance of loneliness was predicted to be 37% [95% CI: 36.6%–37.3%] on average if the population’s social connectedness was low as 2 on the index. Assuming other features remain constant, the interaction between age and social connectedness in Figure A.8 (see Supplementary Materials Appendix A.6) reveals that among Australians who were less socially connected, individuals aged 45–75 generally experienced greater loneliness than others. Figure A.9 (see Supplementary Materials Appendix A.6) illustrates that if social connectedness and friendship fulfilment were improved jointly, loneliness could more effectively be reduced. Furthermore, adults with a social connectedness index in the highest 2.5th percentile were estimated to experience greater loneliness compared to some who were relatively less connected. This may be because those who were coping with loneliness were in the process of actively seeking social connections. Interestingly, we found that this group consists of older, single females who experienced the death of someone important in the last 12 months.

As shown in Figure A.5 (see Supplementary Materials A.4), distinct separation and clusters of SHAP values were observed between individuals with a life partner and those without, indicating a strong influence on loneliness. Notably, adults living in Australia without a life partner are more likely to experience loneliness, with the average probability as high as (37.7% 95% CI: 37.4%–38.0%), marking an 11-percentage-point increase compared to those with a life partner (26.6% [95% CI: 26.4%–27.1%]) (see Figure A.6e, Supplementary Materials Appendix A.6). Interestingly, across different life stages, having a life partner was more important among adults aged 31–40 and 76+ compared to others. Social fulfilment emerged as one of the top risk factors. Interestingly, friendship fulfilment played a more pronounced role among younger cohorts, while spare time fulfilment showed a stronger influence among older cohorts (see Figure A.5, Supplementary Materials Appendix A.6). Specifically, youth (aged 18–30) who strongly disagreed with the statement ‘I seem to have a lot of friends’ were predicted to be over two times lonelier compared to those who strongly agreed (46.5% [95% CI: 45.9%–47.2%]; 22.0% [95% CI: 21.4%–22.5%]). Youth who were less mentally well and not fulfilled in their friendships were at a high risk of loneliness, with predicted loneliness reaching as high as 78%. Conversely, this probability could be reduced to below 15% through the joint improvement of mental well-being and social fulfilment (see Figure A.10). For the elderly (aged 76+), those who were almost always not fulfilled in their spare time were predicted to have a (44.9% 95% CI: 43.9%–45.8%) chance of experiencing loneliness, reflecting a 13-percentage-point increase compared to the cohort prevalence. This prediction increased to more than 51% if they also did not feel part of the community (see Figure A.11, Supplementary Materials Appendix A.6).

In contrast to the micro level of social integration, community participation was not found to be as influential in predicting loneliness compared to other factors and hence was not selected in the final models. Results for other features, including experiencing the death of someone important, sex, employment and age, are provided in Supplementary Materials Appendix A.6. Table A.3 in Supplementary Materials Appendix A.7 presents key performance metrics of the fitted models, including accuracy, precision, recall, F1-score and the area under the curve (AUC) of the receiver operating characteristic. The performance of our model was acceptable in predicting loneliness with an out-of-sample AUC of 0.80 in the full data set and 0.77–0.84 across the different age groups.

Discussion

Our study employed predictive machine learning models to examine loneliness risk factors among adults living in Australia, highlighting the importance of social integration and mental health. Insights were provided for both the entire population and specific age groups. For the overall population, key factors were identified: mental well-being, having a life partner, social connectedness and social fulfilment – encompassing friendship and spare time fulfilment. Moreover, heterogeneity in loneliness was observed across different life stages, suggesting tailored approaches may be necessary. Specifically, among young adults, loneliness correlated more strongly with friendship fulfilment, satisfaction with financial situation and health, whereas among older adults, spare time fulfilment, community satisfaction and experiencing loss of loved ones were more influential. Recognising both the homogeneity and heterogeneity of loneliness is crucial for effective interventions. This finding of heterogeneity in drivers of loneliness reflects distinct social and emotional needs at different stages of life and is consistent with international studies. Loneliness in youth is often linked to identity formation and peer group dynamics. A longitudinal study in the Southeastern United States found that adolescents who perceived higher levels of cumulative support from family, peer and teacher relationships exhibit greater socioemotional functioning, sense of belonging and decreased feelings of loneliness (Cavanaugh and Buehler, 2016). Similarly, a study involving 14,077 adolescents from 156 schools in England from 2006 to 2014 found that loneliness in youth is associated with peer relationships and social inequality, with authors suggesting that comparison in terms of living conditions contributes to loneliness among young people (Qualter et al., 2021). They also found that loneliness becomes more intense among older adolescents, suggesting that loneliness emerging during adolescence is likely to be carried into early adulthood (Qualter et al., 2021). Our findings are also consistent with the literature on drivers of loneliness in seniors, which is primarily associated with adjustment to life transitions such as retirement and bereavement. A 28-year prospective study in Finland found that loss of a partner, reduced social engagement, increased physical disabilities and increased feelings of low mood were related to enhanced feelings of loneliness (Aartsen and Jylhä, 2011).

Mental well-being consistently emerged as the most influential feature in our models, highlighting its significant relationship with loneliness across all populations. This finding aligns with existing research, which has shown a strong association between loneliness and mental health (Beutel et al., 2017; Mansour et al., 2021; Richard et al., 2017). Despite its protective effect against loneliness, the effect of improvement in mental well-being does not follow a linear pattern, with slower progress observed in the lower range (i.e. less than 40). Mendelian randomisation analysis (Sbarra et al., 2023) and prospective cohort studies (Nuyen et al., 2020; Mann et al., 2022) indicate that the relationship between loneliness and mental health is bidirectional. Poor mental well-being may contribute to loneliness through social withdrawal and an unmet need for social support, while, conversely, loneliness may exacerbate existing mental health issues; however, the precise mechanism underlying observed associations between loneliness and mental health is likely to be complex and requires urgent clarification (largely via longitudinal studies and utilisation of appropriate statistical techniques).

Our findings regarding social integration are generally consistent with current research. For example, our models showed that people without a life partner experienced significantly greater loneliness (Steed et al., 2007; Beutel et al., 2017; Botha and Bower, 2023; Dahlberg et al., 2015). We further found that this companionship with a life partner is more pivotal among middle-aged adults and the elderly compared to others. Some studies have highlighted that individuals with frequent social connections and more friendships are less lonely (Botha and Bower, 2023). Our model results also underscore the significant impact of social connection and social fulfilment on loneliness. Greater social connections with family, relatives, friends and neighbours, as well as fulfilling friendships and spare time activities, were found to be protective against loneliness. Furthermore, there would be more effective protection against loneliness when social connectedness and social fulfilment are improved jointly. While our analyses showed weaker associations between community participation and loneliness, macro factors may play an important indirect role in loneliness. By fostering civil society, investing in social infrastructure and ensuring robust social protections, the vulnerability to loneliness may be mitigated. Such measures also create fertile grounds for enhancing social integration through opportunities to establish, expand and nurture personal relationships and mental well-being (Swader and Moraru, 2022). Given the distinct life stage-related challenges of loneliness, interventions must be designed to address specific needs: enhancing social skills to enable successful reconnection to peers, family and school community for young people; promoting community engagement, social support and physical and financial accessibility for seniors; and providing mental health support across the lifecourse (Eccles and Qualter, 2021; Katulis et al., 2023; Gustafsson et al., 2017).

Efforts are already underway globally to address the loneliness epidemic. For instance, the US Surgeon General has outlined a framework emphasising the strengthening of social infrastructure and reducing disparities in social connection, with the aim of mitigating loneliness (Department of Health, Human Services, 2023; Murthy, 2020). In the UK, the Loneliness Commission was established to ensure that reducing loneliness remains an enduring parliamentary priority. They have also published the world’s first loneliness reduction strategy and created a Know Your Neighbourhood Fund to invest in empowering communities to alleviate chronic loneliness in disadvantaged areas and other initiatives (McDaid et al., 2022; Department for Culture, Media and Sport, 2023). Intervention research is being undertaken in Australia demonstrating the promising effects of targeting the development and maintenance of social group memberships in improving mental health, well-being and social connectedness and reducing loneliness (Haslam et al., 2016; Haslam et al., 2019). However, despite research and advocacy highlighting the need for a systemic response, the Australian Government has yet to establish a national strategy (Gregory, 2024; Suicide Prevention Australia, 2022; Ending Loneliness Together, 2022).

Our findings underscore calls in the Australian context to develop targeted interventions to address loneliness. Assuming that mental ill-health is a cause of loneliness, population-based and health services interventions should focus on improving national mental health and social community connections while considering the heterogeneity across life stages. Population-based mental health initiatives could focus on delivering an appropriate balance of universal and indicated interventions (Skinner et al., 2023). National mental health services initiatives could focus on increasing equitable and early access for young people to quality mental health care and enhancing technology-based coordination of care (Crosland et al., 2024). Creating supportive environments in workplaces and communities that prioritise mental well-being and promote community-based support networks and peer support groups is essential. To achieve these goals, collaboration between governments, businesses and community groups is important to ensure a coordinated and comprehensive approach to fostering social connections and mental health support.

Modelling has also shown good impacts from interventions focused on fostering social connectedness, from which people can build quality friendships, facilitate employment opportunities and provide mental guidance and counselling for those experiencing health issues (Occhipinti et al., 2021; Occhipinti et al., 2021). By implementing such initiatives, younger individuals may benefit from increased social support networks and enhanced well-being. On the other hand, interventions for loneliness among older adults could focus on facilitating regular social groups and events in the community, encouraging participation in community building and providing care and counselling services aimed at supporting those experiencing grief. However, these interventions require the allocation of resources to foster a more connected community. This includes building a supportive infrastructure that encourages investments in community facilities, mental health services, employment programmes and social support networks tailored to the diverse needs of different age groups. Investing in social capital infrastructure to foster social production (unpaid activities that contribute to civil society and strengthen the social fabric of communities) (Occhipinti et al., 2023) could be a strategic approach to combating loneliness, particularly among older adults. This approach underlines the importance of social integration and could guide policy interventions that prioritise social cohesion and the creation of supportive environments conducive to mental health and interpersonal relationships. Additionally, initiatives aimed at reducing loneliness should be integrated into broader public health strategies to ensure sustained support and impact. By prioritising these efforts and investing in the necessary infrastructure, policymakers and communities can work together to create environments that promote social cohesion, mental well-being and overall resilience against loneliness among adults living in Australia.

A key limitation of this study is the simplification of the 7-point loneliness scale into a binary feature, focusing on the likelihood of “often feeling very lonely” rather than assessing the severity of loneliness. Furthermore, the measure of loneliness used in the HILDA survey (and our analyses) is direct, asking participants specifically about loneliness, and is therefore open to potentially significant social-desirability bias (Mund et al., 2023). Another limitation is that our model predicts loneliness using data for the included features collected in the same study wave only so that our analyses are effectively cross-sectional, restricting our ability to infer causality and precluding the establishment of definitive cause-effect relationships.

Conclusion

In conclusion, our study highlights the complex interplay of various factors potentially contributing to loneliness among adults living in Australia across different age groups. Through the utilisation of predictive machine learning models, we identified common risk factors, including mental well-being, social connectedness, social fulfilment and having a life partner. Our findings contribute to the growing literature highlighting the importance of addressing loneliness as a multifaceted issue that requires targeted interventions tailored to the specific needs of different age groups. By recognising the heterogeneity of loneliness and prioritising efforts to foster social cohesion and support networks, policymakers and communities can work towards creating environments that promote mental well-being and resilience against loneliness among adults living in Australia.