1 - Conducting Multi-nation Research

Summary

This chapter presents considerations when conducting multi-nation research, with a particular focus on research conducted with sexual minority and gender diverse (SMGD) communities. Specifically, this chapter provide a nonexhaustive list of best practices from idea development, including forming multi-nation research teams, to practical challenges of the research process, such as questions of cross-cultural reliability and validity of chosen methods, and challenges of recruitment and data collection. Additionally, reference is made to specific methodological aspects, including missing data analysis and common analytical procedures, such as multilevel modeling. Examples and aspects of reflective practices, such as reflections on positionality, the impact of cultural and ethical aspects on the research process, and sources of bias and how research teams may make efforts to overcome them, are also presented.

Over the past decade, multi-nation studies have become more prevalent in the social sciences, represented by a growing body of literature that reflects, promotes, and consolidates best practices and insights for this research approach (Forscher et al., 2023; Hearn, 2015; Jarke et al., 2022; Moshontz et al., 2018; Puthillam et al., 2024; Syed & Kathawalla, 2022). Several global changes have driven this, including increased opportunities for networking among scientists and global challenges such as the COVID-19 pandemic. Moreover, professional boards have called for researchers to move toward globalization of research efforts (IARR, 2023; Puthillam et al., 2024).

In the present chapter, we first follow one of the best practices that we recommend by offering our own author positionality. We then highlight reasons for conducting multi-nation studies.

Drawing on experience completing the Lived Experiences of Sexual Minority and Gender Diverse Individuals Multi-Nation Project (hereafter SMGD-MN) that was co-led by Drs. Randall and Lannutti (for more information see the Introduction), we reflect on the question: What makes a well-designed and well-executed multi-nation study? In doing so, we present questions, reflections, dilemmas, and considerations with references to examples and recommended practices of multi-nation research. Moreover, we aim to showcase the specific challenges researchers may encounter when conducting multi-nation research and how they may be overcome. We will use the term “multi-nation” here as this is a broad yet technical term, that reflects the multiplicity of research teams and samples involved from many countries.¹ This chapter will consider the cultural factors that may play a role in research and reflect on their significance.

It is also important to note that, while many steps may be valid for qualitative and mixed-method designs, the focus here is specifically on self-report quantitative studies; however, we will refer to the many lessons multi-nation quantitative studies may learn from qualitative approaches. Finally, this chapter will summarize general, topic-independent aspects of multi-nation studies. However, we will build on experiences and practices within research about SMGD individuals and their relationships wherever possible. Moreover, many considerations and suggestions in this chapter are in line with and drawn from the observations our multi-nation research team has made during the research process outlined in this edited book. We will reflect on these experiences in the first sections of the chapter. However, the thoughts presented here are not prescriptive; they rather summarize considerations the authors distilled from the literature on multi-nation research and their own research experiences.

Authors’ Positionalities

The first author self-identifies as a middle aged, white, middle-class, heterosexual male, living in Central-Europe, experienced in working in international research teams and as a couple and family therapist providing support for SMGD clients. The second author identifies as a young, white, middle-class, heterosexual female, living in Central-Europe. The third author identifies as a young, Latino, middle-class, heterosexual cisgender male living in the United States. The fourth author identifies as a young, white, middle-class, heterosexual female, living in Central-Europe. The last author identifies as a middle-aged, Asian, heterosexual male living in India. His research primarily focuses on cross-cultural understanding of relationship dynamics of individuals across Western and non-Western cultural contexts.

Reasons to Conduct Multi-nation Studies

While there are a multitude of reasons to conduct multi-nation studies, we would like to highlight three important reasons that are particularly salient for relationship science. First, multi-nation research may counterbalance the lack of voices of underrepresented populations and minorities in social science research (Curran et al., 2023; Pollitt et al., 2023; Randall et al., 2023; Totenhagen et al., 2023a). A common critique of psychological theories and research is that they primarily represent WEIRD (White, Educated, Industrialized, Rich, and Democratic) respondents (Henrich et al., 2010). Extending the scope of research collaborations may help increase the representation of underrepresented populations, nations, and languages in empirical research (Curran & Randall, 2023). The need for inclusion may contribute to more creative theory formation, unique aspects in questioning, new methodological procedures (Moshontz et al., 2018), and important implications for advocacy and policy.

Second, as part of psychology’s “replication crisis,” researchers face critiques about replicability, generalizability, strategy selection, inferential reproducibility, and computational reproducibility of previous results (Forscher et al., 2023). To answer these challenges, researchers have proposed several procedures to address low power, unreliable results, and publication bias (Cuccolo et al., 2021). One solution is to build “big team science” by forming large scientific networks and research collaborations. These collaborations afford the possibility to pool resources from research groups and reach out to potential respondents from all across the globe (i.e., crowdsourcing) (Cuccolo et al., 2021). Crowdsourcing data collection has the advantage of the natural heterogeneity of research contexts, and including many labs around the world may help eliminate biases that might be present in laboratories led by a single principal investigator (Moshontz et al., 2018). Moreover, by reaching out to more heterogeneous samples, researchers can increase the diversity of respondent samples. Variations are always present in large, crowdsourced samples. However, cultural differences and their diversity are systematic and influenced by specific cultural contexts (Matsumoto & Juang, 2013). In multi-nation research, these differences can be accounted for and analyzed.

Third, there is an increasing need for research on cultural variations to examine social relationships, especially in underrepresented populations such as sexual minorities and gender diverse (SMGD) individuals (Flatt et al., 2022; Izienicki, 2021; Walch et al., 2020). Multi-nation research can detect and test systematic, culture-based variations in the phenomena under study, such as gender relations (Hearn, 2015), structural stigma against sexual minorities (Pachankis & Bränström, 2018), and moderating effects of social support and dyadic coping on symptoms of distress for SMGD individuals in a relationship (present book); while, at the same time, it can elucidate context-independent associations that may generalize across nations and cultures. Furthermore, not only can the inclusion of SMGD individuals and communities increase the diversity of the samples, it can also give voice to their experiences and needs. This inclusivity, in turn, may raise respect and understanding for these individuals and communities worldwide.

The SMGD-MN Research Project

Overview of the SMGD-MN Project Development

The initiative was launched by Ashley K. Randall and Pamela J. Lannutti (co-PIs) and was extended to a network of relationship science scholars in various countries. The inspiration for the project was the recognition that a growing body of research has focused on the lived experience of those who identify as either SM and GD or both; however, much of this research has focused on a single dimension of identity within a specific geographical area (i.e., within the United States and other Western cultures only, with mostly white samples in those cultures). Therefore, the goal of this project was to examine the lived experience of SMGD individuals in a multi-nation study in the hope that, by collecting demographic data and data related to individual and relational well-being, we can understand better the lives of SMGD individuals living around the world. The research project also aimed to refine and develop theoretical thinking about SMGD people’s lived experiences and relationships by increasing their representation from multiple nations. Furthermore, it also aimed to offer new findings that focus on aspects of support and coping processes in the lives of SMGD individuals, as well as contextualizing and explaining the implications of these data for clinicians and policymakers alike.

Team Composition

The research team members were first approached based on a previous cooperation experience with a large-scale research project on COVID-19-related stress and dyadic coping (Randall et al., 2022). The invitation, sent out in October 2020, focused on collecting national samples of SMGD individuals and studying their experiences in the respective countries. The international team of authors was finalized in December 2020 with a “country team” for each participating nation.

The teams used the Slack platform to discuss research process-related topics. The main advantage of Slack is that it allows researchers to send messages in different project channels and to structure those conversations into new discussion threads, while also having the information accessible in one place. Between December 2020 and the end of March 2021, the research teams discussed topics of interest, the theoretical framing, and topics related to data collection and measurement.

In April 2021, the “SMGD-MN Phase 1: Lived Experiences of SGM Individuals” project was registered in OSF Registries under the reference ID TSJ8V (https://doi.org/10.17605/OSF.IO/TSJ8V). The original study plan included 18 countries (Austria, Belgium, Brazil, Germany, India, Indonesia, Ireland, Italy, Malaysia, Netherlands, Nigeria, Portugal, Switzerland, Tanzania, Thailand, Türkiye, United Kingdom, and United States). However, due to challenges associated with data collection, teams in Belgium, Ireland, Nigeria, South Africa, Tanzania, Thailand, and the United Kingdom were unable to complete data collection.² The remaining 11 countries represent four continents and provide rich data from several cultural regions of the world; however, we acknowledge the need for continuous expansion of the scientific discovery of research in this (and related) domains.

Study Development and Implementation

Screening of the available measures revealed the need to translate several scales from English so that all measures could be administered in the native language of the respective country. In doing so, appropriate translation and back-translation procedures were followed (e.g., Beaton et al., 2000), yet specific procedures for this work were determined by each country’s team. If needed, the measures were translated into the native language by the teams for each nation. The country teams organized the translation process and represented variations of the standard-to-and-back-translation procedure. For example, since there were three German-speaking study sites, measures were translated by one study site and double-checked by another. In the third study site, discrepancies were resolved through discussion. In Brazil, two bi-lingual (Portuguese–English) researchers independently translated the instruments, and a third researcher compiled the two versions. Two bi-lingual (Portuguese–English) researchers jointly compared the compiled version with the original instruments in English and framed the final Brazilian versions of the instruments. In Thailand, the research team applied a committee approach: The translation panel included five members from various health-related disciplines performing collaborative consensus efforts in the translation. Once the measures were finalized, each team created their unique country codebook that contained the compilation of study measures and scoring information.

Once each country’s codebook was completed, the PIs along with the international research team formed a plan for obtaining Institutional Review Board (IRB) approval. First, IRB approval was obtained by PI Randall from Arizona State University (ASU) in April 2021. According to the national and university regulations, each country’s PI could decide if they would default to ASU’s IRB – in this case a letter from the local IRB official was required – or apply for their own institutional IRB. Most of the country teams applied for their respective university IRB approval, while Indonesia, Portugal, and Thailand teams’ universities accepted the IRB approval of the ASU.

SMGD-MN Data Collection

Country teams were autonomous in data recruitment and collection. Most teams followed a composite approach and posted recruitment calls to various university and community listservs and social media sites (e.g., Facebook, Twitter/X, Instagram). Many teams approached LGBTIQA+ organizations and stakeholders (e.g., community leaders), who, in turn, served as survey multipliers on social media and their organizations’ listservs or newsletters. The involvement of organizations and stakeholders also ensured that potential participants had trust in the study and, presumably, were more willing to answer the call. Furthermore, country teams decided whether they offered compensation for study participation. Data were collected in late 2021 and early 2022.

During data collection, a data management team was formed to ensure uniform data management practices. Data screening procedures were built on the experiences with the previous multi-nation study on COVID-19-related stress and dyadic coping (Randall et al., 2022). At the end of data collection and after an initial data screening by the country teams to ensure that the data collection survey worked correctly, the resulting datasets were further screened for indicators of careless responding (Brühlmann et al., 2020; Curran, 2016). Three indicators of careless responding were calculated for each country dataset: Percentage of missing responses, long string index, and person-total correlation. The potential invalid responses were flagged and removed from the dataset.

In sum, the SMGD-MN study is an excellent example of the possibilities (and challenges) of conducting multi-nation research. The original research question was valid and worth studying because cultural and national variations likely existed, yet the diversity of datasets was managed and analyzed following a uniform protocol by a core team, ensuring the highest possible comparability of multi-nation datasets. Building upon the experiences with the SMGD-MN study, the following sections will present several considerations for conducting multi-nation research.

Considerations for Multi-nation Research Studies

Research Questions and Methods

Multi-nation research can address multiple types of research questions including method validation, indigenous cultural questions, and cross-cultural comparative studies (Matsumoto & Juang, 2013). Method validation studies establish the psychometric equivalence of an existing measure before using it in a new cultural context. Indigenous cultural studies adopt the mono-cultural framework and focus on an in-depth analysis of cultural systems. Finally, comparing a specific variable across different cultural contexts is the common method in cross-cultural comparative research (Berry et al., 2002; Matsumoto & Juang, 2013). Similarly, research traditions may be also identified as (i) studies that intensively examine psychological phenomena within a single cultural context, known as the emic approach, and (ii) studies that compare broad patterns of behaviors across multiple cultures, known as the etic approach (c.f., Berry, 1989).

The emic perspective focuses on culture-specific psychological components that are not applicable to everyone. On the other hand, the etic approach is based on the premise that multiple psychological constructs have a universal appeal and are equally applicable to everyone across cultures. For example, Worthen et al. (2017) compared the attitudes toward SMGD individuals across a sample of university students from Italy, Spain, and the United States. Results from 1,311 individuals found that political beliefs, feminism, and religiosity influenced attitudes toward SMGD individuals similarly across cultural contexts. Although cross-cultural psychology literature has been populated by studies involving these two approaches, many theorists argue in favor of the integration of the emic–etic approach and promote studies with a focus on universal psychology that applies to a broad range of cultures (Berry et al., 2002; Cheung et al., 2011). Multi-nation studies, as is the case with the SMGD-MN study, may be particularly well-suited to take an emic–etic approach because multiple investigations within nations and across nations may result from the larger study.

van de Vijver and Matsumoto (2011) identified three approaches to selecting comparative research methods based on the nature of the research questions. First, contextual studies examine the sociocultural factors such as socioeconomic conditions, education, and health services available for SMGD individuals across countries that may fully or partially explain national differences. Contextual studies will help understand biases and inequivalence ingrained in the cultural factors and assess their role in explaining the observed cultural differences (Matsumoto & Juang, 2013). Second, exploratory studies attempt to closely examine the similarities and differences of cultural contexts (e.g., whether the COVID pandemic exacerbated social discrimination among sexual and gender minorities worldwide). In contrast, hypothesis testing directly tests the inferences of a theory across contexts (e.g., whether the sexual minority stress theory works similarly across individualistic and collectivistic cultures). Third, structure-oriented studies examine the similarities and differences in a construct (e.g., whether homophobia is conceptualized similarly across cultures), their structure (e.g., can minority stress be assessed involving the same dimensions across cultures), and their association with other constructs (e.g., whether minority stress and stigma have the same relationship across cultures), while level-oriented studies compare the scores on a given variable across cultural contexts (e.g., the mean scores on minority stress across cultural contexts).

Team Formation

Successful multi-nation research requires forming enduring networks of researchers and teams from overlapping disciplines, professional organizations, and research areas (e.g., Forscher et al., 2023; Jarke et al., 2022). As was the case with the SMGD-MN study, it is recommended that a first invitation to potential collaborators should be clear and comprehensive about the aims and conditions of the study. The conditions may include the possible outcomes, authorship roles, deadlines, and the platforms used for communication and data management. Transparent sharing of the broader context (e.g., professional debates and future grant application possibilities) may also strengthen future team members’ involvement.

Cuccolo and colleagues (2021) review the possible contributions of crowdsourcing research within cross-cultural studies that can easily be generalized to multi-nation research. Researcher crowdsourcing brings together the intellectual and material resources of many researchers and labs, often from considerably diverse geographic locations. Researchers may need to formalize their collaboration through a written agreement to delegate responsibilities and negotiate the terms of data acquisition, analysis, and reporting. For example, the CRediT taxonomy (Brand et al., 2015) or APA guidelines (American Psychological Association, 2023b) are tools that can be used to formalize authorship decisions in written agreements. New initiatives such as Psychological Science Accelerator aim to build research collaborations where the organizational background (e.g., website and listserv, boards, and committees) provides stability for designing new projects and creates recruitment networks for future projects (Moshontz et al., 2018).

Studies discussing team formation emphasize the importance of involving a diverse research community (Abboud et al., 2022; Gabb & Singh, 2015; Wagaman et al., 2018). For example, Gabb and Singh (2015) intentionally composed their research team to enlist multiple perspectives on romantic couple diversity at all stages of the research process, including the intersection of race, culture, class, generation, and sexual orientation. Researchers also emphasize the importance of including team members that share identities with research participants to align the goals of researchers with the goals of the communities they study (e.g., community-based participatory research; Pollitt et al., 2023; Ricks et al., 2022). The SMGD-MN study is a good example of a diverse group of researchers, some of whom are SMGD and some who are not, to study an underrepresented group. We believe the SMGD-MN study has benefited from the diverse professional and personal experiences of our research team, especially in forming research questions, data analysis, and results interpretation.

Reflexivity and Positionality

Importantly, researchers emphasize the need for reflexivity, or the recursive practice of reflecting on the influence of one’s identities on the research process (for more information see Curran & Randall, 2021). The practice of reflexivity is gaining acceptance in social sciences, especially where culturally-embedded topics, such as SMGD experiences, are studied (Du Preez, 2008). Reflexive research practice often includes a reflection on the authors’ positionality (e.g., identity, privilege, and power) and its influence on methodological and theoretical decisions (Curran & Randall, 2021). Jafar (2018) argues that a more reflexive research practice in quantitative research may increase the validity of results by better defining the boundaries within which the results should be interpreted.

A growing number of research teams provide members’ positionality statements (Randall et al., 2022), as exemplified in this volume. Positionality statements acknowledge the relevant background, context, and situatedness of researcher positions and identities (Curran & Randall, 2021), that include but are not limited to current geographical location, sexual orientation, gender identity, and personal experiences related to the topic. Notably, some researchers may hesitate to disclose identities that could have a negative consequence for their professional and personal lives (e.g., gender, sexual orientation), especially if they live in countries with high levels of stigma (Rosser et al., 2021). This was the case in the SMGD-MN study. Multi-nation research teams should openly discuss this issue and create an environment where members feel comfortable reconciling their personal perspectives with the need to protect their identities.

There are several aspects throughout the research process (including data analysis, interpretation, and write-up) where researchers may reflect on their practice and can make decisions toward being more inclusive and responsive to diversity and equity concerns (Roberts et al., 2020; Tajima, 2021). Specifically, Andrews and colleagues (2019) suggest five points on how to treat different positions and biases fairly in the research process. In the context of SMGD-related research, these aspects can be reformulated in the following ways:

1. 1. Examine your own biases concerning SMGD-communities.

2. 2. Commit to digging deeper into your data to respond to the message it covers about the participants from SMGD-communities.

3. 3. Recognize how the research process impacts SMGD-communities.

4. 4. Engage with SMGD-communities as research partners – direct research efforts toward problems and issues deemed relevant by members of SMGD communities.

5. 5. Guard against the implicit or explicit assumption that living with white, heterosexual, socioeconomically privileged identities, to name a few, is the default experience.

With all these aspects in mind, how, then, should researchers pursue reflexive practice in multi-nation research? Formal positionality statements may be necessary; however, they represent the minimum requirement for reflexive practice. A growing body of literature presents the challenges in virtual communication in geographically distributed (international) teams (Morrison-Smith & Ruiz, 2020; Ward & Given, 2019). While most of the studies focus on possible technological solutions, little is known about best practices. In qualitative research, it is a common practice to make field notes and memos during the research process that can be transmitted into a virtual collaborative network (Antonio et al., 2020). Multi-nation teams valuing reflexive practice should use consistent methods in their projects, that include small, individual team discussions to allay any concerns with this process. Having such open conversations can help foster a more diverse and inclusive research process.

Cultural Reflectivity

Culture is an umbrella term that refers to the distinctive customs, values, beliefs, knowledge, art, and language of a society or a community (VandenBos, 2015). Cross-cultural research attempts to understand social and psychological phenomena from a cultural perspective and provide a better understanding of the role of culture. In this section, we provide an overview of various theoretical concepts that can help reflect the role of cultural concepts in multi-nation research teams.

First, teams may reflect their stance toward three general orientations, whether a theory or phenomenon is culturally bound or universal across all humans (Matsumoto & Juang, 2013). Absolutism assumes that psychological processes and phenomena are the same across all cultural contexts. For example, constructs like ‘intelligence’ or ‘depression’ are assumed to be the same everywhere despite individual differences. The second perspective, known as relativism, assumes that all human behavior is culturally patterned. In other words, relativism posits that culture influences every aspect of human behavior. The third perspective, universalism, falls between absolutism and relativism, and assumes that basic psychological processes are common to all humans, but culture influences the development and display of those characteristics, such as the expression of emotions (Matsumoto et al., 2008). Thus, culture allows and partially accounts for the variations and varieties of human behavior.

Second, individualism and collectivism are two of the most frequently researched cross-cultural constructs that may affect the teams’ work throughout the research process. The individualism-collectivism dimension has been used to describe cultural variations in attitudes, values, group norms, family relationships, and so on (Hofstede, 1980; 2011; Triandis, 2004). Individualism refers to the cultural predisposition in which the ties between individuals are loose – everyone is expected to look after themselves and their immediate family and is allowed to make decisions about one’s life in matters of career, marriage, and so on. In contrast, collectivism refers to the degree to which people in a society are integrated into groups. On the collectivist side, we find cultures in which people from birth onward are integrated into strong, cohesive in-groups, often extended families (Bhugra, 2004).

Teamwork in multi-nation studies may include the reflection of basic beliefs and assumptions regarding SMGD people-related cultural dimensions. However, little multi-nation research has documented the reflective process on potential cultural biases in the researchers and the research teams. For example, in a recent cross-cultural comparison of the role of internalized heterosexism and psychological intimate partner violence perpetration in lesbians and bisexual women in Türkiye and Denmark, Ummak and colleagues (2022) reflected in detail on the social and cultural differences between the two countries, such as individualism-collectivism, and how these differences might have impacted the unique experiences of their respondents. At the same time, Ummak and colleagues (2022) did not consider how their diverse cultural background was represented in the research process. While the cultural aspects are crucial, the lack of the authors’ comments on their positionality highlights the need for a more transparent research practice in multi-nation research (c.f., Jafar, 2018; Manohar et al., 2017).

Ethical Aspects of the Research Process

Ethics features prominently in many aspects of multi-nation research, ranging from practical challenges like study design and IRB review to the dissemination of findings via open science practices (Paxton & Tullett, 2019). Concerning IRB review, collaborators from many nations may apply for multiple local/national IRB approvals. However, researchers encounter obstacles due to varying standards and processes across the affiliated institutions (Stein et al., 2015). Alternatively, the team may apply for joint IRB approval (Barchi et al., 2014), although this still requires approval from local IRBs. Thus, multi-nation teams should be prepared to coordinate various levels of IRB review to ensure they are meeting the various standards and ethical guidelines across nations.

During the application process, team members should also reflect on how they will treat differences in data security rights and the broader legal environments in different countries and regions (Barchi et al., 2014).

Researchers are responsible for the safety and confidentiality of SMGD participants’ data (American Psychological Association, 2017), and this is especially true in countries with discriminatory norms against SMGD communities. Risks to participants’ confidentiality should be detailed explicitly within a culturally responsive informed consent process, and researchers must inform participants of the measures taken to protect sensitive information. Researchers may also choose to involve community members in the conceptualization and measurement of social identities (e.g., race, ethnicity, sexual identity, gender identity, etc.) to attune the perspectives of researchers with the communities they study – a practice common in community-based participatory research (Ricks et al., 2022). Online data collection methods may also provide security to the participants, because online distance assessment increases the sense of anonymity.

Another systemic aspect of ethical research practice concerns access to the research infrastructure. Namely, researchers based in nations with more expansive scientific infrastructure may have better access to cutting-edge resources than researchers in the “periphery” (Lindemann & Häberlein, 2023). Some authors call attention to the potential drawbacks of research conducted by siloed laboratories in large institutions. In this model, “the current incentives, infrastructure, and institutions in academic science have all developed under the assumption that science is conducted by solo Principal Investigators and their dependent trainees, an assumption that creates barriers to sustainable big team science” (Forscher et al., 2023, p. 607). In contrast, multi-nation researchers working in large teams may also reflect on the potential risks of big team science and how to avoid them. This includes being aware of institutionalization and the resulting conservative, power-oriented practice, such as unequal access to financial resources or the imposition of a centralized perspective (Hearn, 2015).

Open science is a further practice that is supported by ethical considerations, specifically the principles of integrity and trustworthiness (Lindemann & Häberlein, 2023). As Moshontz et al. (2018) argue: “Once materials and data are ready to share within a collaborating team, they are also ready to share with the broader community of fellow researchers and consumers of science” (p. 504). Researchers have a growing number of available procedures to substantiate the validity of their research, and that may ultimately contribute to a “credibility revolution” (Vazire et al., 2022). Preregistration (Nosek et al., 2018, 2022), transparency, and registered reports (Chambers, 2019; Hardwicke & Ioannidis, 2018), for example, the use of the OSF platform (Paxton & Tullett, 2019), are options that help avoid questionable publication practices like p-hacking, hypothesizing after the results are known (or HARKing), and selective reporting by employing preregistration. These research pathways also help develop alternative explanations and support robustness checks, cross-validation, and internal replications (Nosek et al., 2022).

Moreover, ethical considerations also apply to data sharing. Specifically, significant contributions to the research outcomes may come from less empowered populations with greater privacy risks (D’Ignazio & Klein, 2020), such as SMGD respondents (e.g., Matson et al., 2019). Therefore, sharing only de-identified data, while upholding the “do no harm” ethical principle (Tajima, 2021), is vital for ethical data management (Meyer, 2018).

Recruitment and Data Collection

Recruitment and data collection present many challenges for all researchers, especially when working traditionally underrepresented populations across nations. For one, researchers must consider the target sample size, particularly in research with hard-to-reach populations, such as SMGD people. While it might be relatively easy to recruit hundreds of participants in a nation with relatively progressive SMGD rights, it might be more difficult to reach the target sample size in high-stigma countries. Second, research teams need to consider the technicalities of recruitment. A local team will have more direct access to the research process (e.g., they can receive feedback from participants, quickly make necessary adjustments) and may be able to build rapport with study participants’ more quickly than a team of foreign researchers. Third, researchers leading multi-nation projects need to be aware of challenges researchers might encounter with respect to study recruitment and incentivizing participants due to financial constraints.

Researcher and participant crowdsourcing, as well as community-based participatory research, might offer useful solutions for these challenges. Community-based participatory research (CBPR; Ricks et al., 2022) is a framework built upon equitable collaboration among scientific researchers, community members, and other stakeholders to improve community health, reduce health disparities, and improve health equity. Being an insider (e.g., sharing an identity with participants or belonging to their community) may open doors and provide practical access to marginalized groups (Gabb & Singh, 2015).

Reflections on Sources of Bias

One of the most important criteria in designing and evaluating cross-cultural research is the presence/absence of equivalence and bias (Matsumato & Juang, 2013). Equivalence is a condition of similarity in conceptual meaning and empirical methods between cultures to make cultural comparisons. Conceptual equivalence refers to the similarity of the meaning of the phenomena under investigation (e.g., whether the construal of dyadic coping – mutual and supportive coping behavior among couples – is similarly understood among couples from different cultures and minority groups; c.f., Totenhagen et al., 2023b). Measurement equivalence refers to the level of comparability of measurement outcomes (e.g., whether the minority stress scale operates similarly across various cultures).

Bias refers to differences in the concepts and methods that do not have the same meaning within and across cultures. Bias threatens the equivalence of measurement outcomes across cultures, and only when instruments are free from bias are measurement outcomes equivalent and provide the same meaning within and across cultures. Matsumoto and Juang (2013) categorized biases into five major types: (1) Conceptual, (2) method, (3) measurement, (4) response, and (5) interpretational.

Conceptual or construct bias refers to having non-equivalence in the meaning of the overall theoretical framework across the cultural contexts studied (e.g., overlapping definitions, differential appropriateness of the behaviors associated with the construct, and incomplete coverage of the relevant aspects/facets of the construct).

Method bias includes sampling, linguistic, and procedural biases. Sampling bias concerns conducting research with a sample that does not truly represent the population or comparing non-cultural demographic variables. Linguistic bias occurs when items on questionnaires or instructions are not semantically equivalent across languages. Biases can also occur for the unintended and unreflected upon differences in procedures used to collect data in different cultural contexts, called procedural bias.

Measurement bias refers to the degree to which the measure used to collect data in different cultures is reliable and valid (Matsumoto & Juang, 2013). Contrary to the common assumption, linguistic equivalence will not ensure measurement equivalence as words might have different meanings and emphases depending on the cultural context, that is, conceptual inequivalence (Kanth et al., 2021, 2023). Thus, validating the measures across cultures and demonstrating measurement invariance across cultures is imperative before using them for cross-cultural research.

Responding in a particular way to items or scales can lead to response bias (e.g., socially desirable responding, the tendency to agree rather than disagree with items, and extreme response bias), that, in turn, can reflect cultural variation.

Finally, interpretational bias can occur while analyzing data and interpreting findings. One of the most common forms of interpretation bias is the overreliance on statistical significance. Effect size estimates are strongly recommended in cross-cultural research (Matsumoto & Juang, 2013). Although we briefly discuss these types of bias here, a more comprehensive presentation of these biases and how to handle them are discussed by van de Vijver and Matsumoto (2011).

Questions of Cross-Cultural Reliability and Validity

Because more than 70 thousand questionnaires are available in English (c.f., American Psychological Association, 2023a), their cross-cultural adaptation is frequently required before using them in a different linguistic and cultural context. This was frequently the case in the SMGD-MN study. Researchers should not only aim to develop the target version as equivalent to the original questionnaire as possible but adapt it to the new cultural context (Bowden & Fox-Rushby, 2003). Beaton and colleagues (2000) offer a five-stage guideline to this process from forward translations to pretesting. Beaton and colleagues (2000) argue for two independent translations (provided by an informed and a naive translator) to highlight ambiguous or challenging phrases in the original translation. The two translations should then be synthetized and back-translated, preferably by two independent native speakers. Then an expert committee, including all former translators, reviews all the translations and reaches a consensus on any discrepancy. The final step is a field test with the intended target population. This recommendation was updated by Borsa, Damásio, and Bandeira (2012), who detailed the role of the original authors through the adaptation process and elaborated on statistical procedures as the last step. Researchers of multi-nation studies might rely on already existing adapted versions of the proposed questionnaires. However, the research team should be prepared to follow a complete adaptation process, at least for some of their questionnaires. Conducting a multinational study using only pre-existing adopted questionnaires in all participating languages would limit the research’s scope due to the varying availability of such questionnaires depending on the targeted languages.

Despite the common practice of cross-cultural adaptation of research questionnaires, and existing guidelines, studies might lack rigor in their validation of research instruments (Arafat et al., 2016). This increases measurement error across scales and makes multi-national comparisons less reliable (Byrne & Watkins, 2003; Davidov, 2008; Stevanovic et al., 2017). Even when questionnaires are previously validated, the researchers should ensure that the instruments measure the same construct in all sampled groups to validated group comparisons (Milfont & Fischer, 2010). Measurement invariance tests provide additional insight to questions concerning cross-cultural generalizability and possible biases detailed in previous sections (Jeong & Lee, 2019).

Vandenberg and Lance (2000) summarized a stepwise procedure to test measurement invariance in a multigroup confirmatory factor-analytic (CFA) framework. The CFA begins with an omnibus test to establish any differences between the groups. In the next steps they suggest imposing increasingly restrictive models beginning with a least restricted model of configural invariance, where the pattern of free and fixed factor loadings is assumed to be identical across groups. This ensures that the scales measure the same concept without any bias. The second step is metric invariance that assumes equal loadings of items on their respective factors between groups. The third step is scalar invariance that assumes equivalence of intercepts across groups and is required for valid mean comparisons across groups (Byrne & Watkins, 2003; Davidov, 2008; Stevanovic et al., 2017). However, less strict measurement invariance conditions might be valid for group comparisons if a reasonable number of parameters are constrained and there is adequate justification (Steenkamp & Baumgartner, 1998; Vandenberg & Lance, 2000).

Missing Data Analysis

Missing data are often an inevitability in social science research (Graham, 2008). The best practices for handling missing data in multi-nation relationship research are no different than most other studies in social science research, with some caveats. Data collected from participants in many different nations, cultures, and languages have a nested or clustered data structure. Observations collected from members of the same nation, culture, or sub-culture are likely to be systematically related due to similarities in social norms, beliefs, behaviors, and response tendencies. Within a single nation, data may also have a clustered structure based on the nature of data collection. For example, data collected from romantic dyads are assumed to be clustered at the level of the dyad (Kenny et al., 2006); partners are assumed to be interconnected, and therefore their responses on psychological measures will likely covary (e.g., shared political affiliation). Nested and/or clustered data is not independently distributed, and missing data procedures (and inferential models, in general) will be biased when this within-cluster interdependence is ignored (Enders, 2022; Hox et al., 2017).

Multi-nation research with SMGD individuals may include data that are nested at both the nation- and couple-level (see chapter 8 in Enders, 2022). For example, an individual’s self-reported sexual satisfaction may depend on both their partners’ behavior (e.g., couple-level factors) and the cultural-norms regarding sexual satisfaction in their country (e.g., nation-level factors; Fisher et al., 2015). Missingness of these data may be related to couple- or nation-level factors – for example, individuals from countries with more conservative sexual values may be less likely to report their sexual satisfaction. Data that are missing as a function of other observed data (e.g., partners’ behavior, nation-level norms, etc.) are considered missing at random (MAR). When missingness is caused by data that are unobserved, these missing data are assumed to be missing not at random (MNAR). Methods such as selection models and pattern mixture models have been developed for MNAR data, but these approaches require researchers to make strict, unverifiable assumptions about the data that may lead to more biased estimates than analyses designed for MAR processes (see chapter of Enders, 2022 for further reading on MNAR processes). Thus, the MAR assumption underlies most data imputation and missing data handling procedures for nested data and will therefore be the focus in the present chapter (Enders, 2022).

Broadly, there are four general frameworks that can be invoked when conducting missing data analysis: Listwise deletion, maximum likelihood estimation, Bayesian estimation (and imputation), and multiple imputation (Enders, 2022). Listwise deletion removes all data from a participant if data from any variable are missing. While this approach is the default in most software packages, it can only be recommended if (1) data are missing completely at random (MCAR) and (2) analyses are sufficiently powered despite the loss of data. Both assumptions are rarely satisfied, especially with nested data, so better alternatives are warranted. Maximum likelihood estimation, Bayesian estimation, and multiple imputation each seek to remedy the problem of MAR processes and are highly preferred to listwise deletion; however, no approach is unequivocally superior.

In maximum likelihood estimation, missing values are not excluded from the analysis (i.e., listwise deleted), nor are missing values imputed – instead, partially missing records are retained within the estimation procedure to help identify the model parameters that best fit the data (i.e., the parameters with the maximal likelihood, given the data). Within a Bayesian estimation framework, missing values are imputed by way of Markov Chain Monte Carlo (MCMC) estimation, that is a set of iterative algorithms used to derive stable, conditional distributions for each model parameter, given (1) prior information about parameters and (2) the observed data. In a Bayesian framework, missing values are imputed using MCMC estimation, and missing values are estimated just like any other parameter in the model. Missing values are imputed in the Bayesian framework only for the purpose of arriving at more accurate estimates for model parameters – imputed datasets are not analyzed on their own in secondary analysis. Conversely, multiple imputation focuses on just that: Missing values are inputted in the dataset and this process is repeated multiple times to derive a set of completed, imputed datasets. Next, analyses are conducted on each of these imputed datasets, and the estimates pooled together to arrive at a single set of results that account for uncertainty across imputed datasets (i.e., “Rubin’s Rules”; Little & Rubin, 2020).

In sum, listwise deletion (i.e., exclusion of incomplete records) is not the recommended course of action for handling missing data. Instead, researchers are encouraged to think critically about the missing data mechanism that is causing missingness (e.g., MAR, MNAR). Are rates of missingness higher for data collected from a specific country, by a specific research team, or from a particular group, at a particular time? These are questions that should be considered prior to selecting an imputation procedure, and efforts should be taken to include variables that predict missingness in models that leverage maximum likelihood or Bayesian estimation to handle incomplete records. No single approach will be ideal in all circumstances, but there are many options for handling missingness with nested data, including options for handling categorical outcomes, longitudinal data, mediation and indirect effects, and models with interaction effects (Enders, 2022). Enders (2022) and Little and Rubin (2020) describe these techniques in detail, with supplementary guides for applying these techniques in popular software such as Mplus, R, SPSS, and SAS.

Common Analytical Procedures

Data collected from SMGD individuals across nations (and possibly within-couples) present unique analytical challenges for multi-nation relationship research. As previously described, these data are nested or clustered, such that observations from individuals in the same nation, or from the same romantic couple, are non-independent (Hox et al., 2017). Stated differently, variation in the data is partially explainable by cluster membership (in this case, dyad or country membership). The intraclass correlation coefficient (ICC) is a common index of covariation explained by cluster membership:

$\rho =\frac{{\tau }_0^2}{{\tau }_0^2+{\sigma }^2},$

where ρ is the ICC, ${\tau }_0^2$ is within-cluster variance, and σ² is between-cluster variance, or the remaining variance not explained by cluster membership. The denominator of the ICC represented the total variance in the data, and the numerator refers only to variance explained by cluster membership. Thus, the ICC can range from .00 to 1.00.

If data were nested at three levels (e.g., individuals nested within couples and nations), then separate ICCs can be computed at the couple- and nation-level:

${\rho }_{\text{Couple}}=\frac{{\tau }_j^2+{\tau }_k^2}{{\tau }_j^2+{\tau }_k^2+{\sigma }^2},$

${\rho }_{\text{Nation}}=\frac{{\tau }_k^2}{{\tau }_j^2+{\tau }_k^2+{\sigma }^2},$

where ${\tau }_j^2$ is the within-couple variance and ${\tau }_k^2$ is within-nation variance. Notice, that Level-3 ICC (e.g., nation-level ICC) can never exceed Level-2 ICC (e.g., couple-level ICC), because it is assumed that both members of a single couple reside in the same nation (for exceptions, see cross-classified models in Hox et al., 2017).

An ICC of zero would suggest that cluster membership has no bearing on variability in the data and can therefore be ignored when selecting an analytical procedure. In practice, ICCs are rarely non-zero when data possess a hierarchical structure and even an ICC as low as .01 can lead to a .20 Type-I error rate – four times higher than the accepted α = .05 (Hedges & Hedberg, 2007; Huang, 2018). Failing to account for clustering in the data leads to underestimated standard errors, that inflates Type-I error rates (Lai & Kwok, 2015).

Multilevel Modeling

The multilevel model (MLM) is the most common method for handling nested data structures with a non-zero ICC (Hox et al., 2017), and applications specific to dyadic analysis have risen in popularity in concert with the uptake of MLM (Kenny et al., 2006). MLM, also called mixed-effects modeling, specifies fixed and random components of a statistical model. The fixed component of the model captures the average effect across all clusters, whereas the random component captures variation in effects between clusters (Hox et al., 2017). Furthermore, predictors in an MLM model can be decomposed into within- and between-level parts – that distinguish between effects at the person-level (e.g., relationship satisfaction of a single person) and at the cluster-level (e.g., average relationship satisfaction of an entire nation). MLM is used extensively within multi-nation relationship research (e.g., Hilpert et al., 2016; Randall et al., 2022). For example, Pachankis and Bränström (2018) present an exemplary study for testing cross-level effects using multinational data. Pachankis and Bränström (2018) study the influence of country-level factors (i.e., structural stigma against sexual minorities) on the association between person-level factors such as concealment of one’s sexual orientation and life satisfaction. Pachankis and Bränström (2018) also provide examples of cross-level tests of mediation and moderation effects. In SMGD research, a recent study applied MLM to survey data collected remotely over a 30-day span to identify associations between sexual minority stress and substance use (Dyar et al., 2022).

Concluding Remarks

Multi-nation studies contribute to a more reliable and inclusive psychological science and, specifically, to research on SMGD individuals across the globe (Elmer et al., 2022). Multi-nation research has the potential to elucidate the strengths, resources, and vulnerabilities of SMGD people from many cultures. Insights generated from multi-nation research, such as the SMGD-MN study highlighted in this volume, can inform practitioners and policymakers concerned with advancing the welfare of individuals, couples, and families in clinical and nonclinical domains (Martínez et al., 2023). This chapter provides a nonexhaustive overview of best practices for conducting multi-nation research, with a special focus on research with SMGD communities. Researchers are encouraged to build on this framework to make multi-nation research accessible and scalable for collaborators of many cultures and languages.