Replication Studies in Second Language Acquisition Research: Definitions, Issues, Resources, and Future Directions

Abstract

A long-standing concern in the field of second language acquisition is that replication studies are not only infrequent but also poorly designed, reported, and labeled. This special issue responds to an urgent need for action by showcasing eleven high-quality replication studies. In doing so, this collection highlights exemplary standards in replication study design and reporting. This introduction to the special issue provides readers with a point of reference for what replication research is, including why replication studies are needed, issues about originality and innovation in replication research, how replication studies can be designed and conducted, and recent advances and resources to support future replication efforts in the field. The introduction concludes with an overview of each study in the special issue, highlighting its main components and discussing how the replication strengthens the field and advances knowledge and understanding about the topic.

A long-standing concern in the field of second language acquisition (SLA) is that replication studies are infrequent (Polio & Gass, 1997; Porte, 2012; Santos, 1989; Tarone, Swain, & Fathman, 1976). This situation threatens the validity and reliability of SLA research as well as the growth of the discipline because confirmed trends in research cannot be distinguished from chance findings and artifacts of a specific research method (Mackey, 2012; Porte, 2013; Porte & McManus, 2019; Valdman, 1993). In addition, replication studies are often poorly designed, reported, and labeled (Marsden, Morgan-Short, Thompson, & Abugaber, 2018; McManus, 2022a; Polio, 2012; Yamashita & Neiriz, 2024), echoing broader concerns about low study quality, transparency, and the need for methodological reform (e.g., Plonsky, 2013, 2024; Riazi & Farsani, 2023; Sudina, 2021). The consequences of this chronic replication problem are severe because a lack of replication (a) harms the credibility of the field’s key claims and theories about SLA, (b) weakens attempts to inform language learning policy and pedagogy, and (c) impedes scientific progress.

This special issue responds to an urgent need for action by showcasing eleven high-quality replication studies. In addition to demonstrating how and why replication is essential to the growth and credibility of the discipline, this collection exemplifies what high-quality replication studies look like and how they can be designed, carried out, and reported. Before looking into the detail of the special issue’s eleven replication studies, this introduction provides a point of reference for what replication research is, including why replication studies are needed, issues about originality and innovation in replication research, how replication studies can be designed and conducted, and recent advances and resources to support future replication efforts in the field.

What is replication research?

Replication is a research method used to confirm, consolidate, and extend knowledge and understanding in empirical fields of study (Brandt et al., 2014; Isager et al., 2023; Porte & McManus, 2019). A replication study works toward this goal by repeating a study’s research design and methods with or without changes, collecting new data, and systematically comparing the previous study’s results with those from the new study (Marsden et al., 2018; McManus, 2022b; Nosek & Errington, 2020). As a result, a replication study is a new empirical study with new data that purposefully addresses the same or similar research question and/or hypothesis as a previous study (McManus, 2023; National Academies of Sciences, Engineering, and Medicine, 2019; Schmidt, 2009). Because one aim of carrying out a replication study is to determine the extent to which repeating a previous study’s design and methods leads to comparable patterns of results and conclusions, replication provides a systematic framework for reconsidering, refining, extending, and sometimes limiting prior research findings. This is why replication has long been considered an essential part of the research process (King & Mackey, 2016; Peterson & Panofsky, 2021; Porte, 2012). Replication studies are key to scientific progress precisely because they systematically revisit prior claims to confirm their stability and detect potential anomalies. It is through this essential process of confirming results, detecting anomalies, and understanding their nature that new lines of research can emerge.

Given these features of the replication research process, it is troubling that replication studies in SLA are still claimed to be infrequent and poorly designed, reported, and disseminated (Marsden et al., 2018; Polio & Gass, 1997; Porte, 2012). One outcome of this situation is that calls for replication are now relatively common across the social sciences (Language Teaching Review Panel, 2008; Plucker & Makel, 2021; Zwaan, Etz, Lucas, & Donnellan, 2018), with journal special issues highlighting the specific ways that replication studies support disciplinary growth (Godfroid & Andringa, 2023; Tschichold, 2023). In SLA, field-specific resources now exist to support the conduct of replication studies (Marsden et al., 2018; McManus, 2023, 2024b; Porte & McManus, 2019).

When conducting a replication study, the aim is to design and report a new empirical study that repeats a previous piece of research in some way, with systematic comparison throughout (Brown, 2012; Porte & McManus, 2019). By conducting a close replication study, in which almost all elements of the initial study’s design and methods are retained without modification (Brandt et al., 2014; LeBel, McCarthy, Earp, Elson, & Vanpaemel, 2018; McManus, 2022b), the researcher’s aim is to understand how an intentional, carefully selected, well-motivated, and principled modification to the initial study’s method impacts the results, if at all. For example, a replication that modifies a prior study’s pedagogical intervention while retaining all other aspects of the research design and methods can be used to understand how that intentional modification influenced the study’s findings and conclusions (e.g., McManus & Marsden, 2018). In this way, replication research allows the stability of a study’s claims to be examined in a principled manner.

Even though determining the stability of previous results is recognized as an essential first step in developing any research program (Gass, Loewen, & Plonsky, 2021; Isager et al., 2023; Mackey, 2012; Schmidt, 2009), this feature is frequently absent in how SLA research is conducted and reported. In other words, systematically revisiting a prior study to confirm its results before extending that line of research is rarely practiced (but see Ellis, Hafeez, Martin, Chen, Boland, & Sagarra, 2014; Morgan-Short et al., 2018). Integrating replication into the research process (i.e., how we develop research questions, design studies, generate claims, and assess theories) is required to support the growth and longevity of the discipline in a principled and systematic way. For example, even though it is well understood that the amounts and types of prior knowledge and experiences that learners bring to learning a new language can substantially influence the trajectories of new language learning (Duff & Byrnes, 2019; Ellis, 2019; Lantolf, Poehner, & Swain, 2018), no single study can examine all contextual influences on learning outcomes (e.g., L2 proficiency, L1 background, prior immersion and instructed experiences, social networks). This is because each study can only ever provide a snapshot of learners’ knowledge and use of language at that one point in time. Replication is needed to determine the stability of important and influential claims as well as to investigate variables unaccounted for in previous research. The more stable a claim turns out to be through replication, the more confident we can be that what has been observed and reported is a confirmed trend (rather than a chance finding, for instance). By not probing the stability of claims made in prior research through replication, we continue to build a discipline “on the basis of scanty evidence” (Polio & Gass, 1997, p. 500).

Originality and innovation in replication research

Unfortunately, claims that replication studies lack originality and/or innovation are not only common but they also misguided. Furthermore, these criticisms serve only to harm the discipline. Does repeating aspects of a prior study’s research design and methods mean that replication studies lack originality and innovation (Martin & Clarke, 2017; Zwaan et al., 2018)? Regrettably, the field’s practices reinforce this view. For example, some journals (e.g., Applied Psycholinguistics) distinguish between “original research article” and “replication research article,” which, while likely unintentional, suggests that replication studies are not to be considered original. While concerns about originality are thought to have limited the conduct of replication studies in SLA research (Marsden et al., 2018; Porte, 2012; but see McManus, 2022a), it is misleading and incorrect to claim that replication studies cannot make original contributions.

Originality refers to studies designed to detect problems “in the existing archetype to the extent that it provokes other researchers and their understanding to further study and analyze so as to answer the questions raised” (Alajami, 2020, p. 4; see also Dean, Hender, Rodgers, & Santanen, 2006; Gordon, 2007). In other words, originality describes work designed to advance knowledge and understanding by investigating questions and/or problems and/or detecting anomalies in prior work. In addition, original research brings together new and existing information, including, but not limited to, new research questions/hypotheses, previously-reported methods, and new results (Dirk, 1999; Guetzkow, Lamont, & Mallard, 2004; Shibayama & Wang, 2020). Based on this understanding, it is difficult to see how a replication study that meets these criteria could be claimed to lack originality. As a reminder, a replication study repeats a previous study’s research design and methods with or without modification, collects new data, and systematically compares the earlier study’s findings with those from the new research (Mackey, 2012; Nosek & Errington, 2020; Polio & Gass, 1997; Porte & McManus, 2019). In this way, a key aim of replication research is to confirm, consolidate, and extend knowledge and understanding within empirical fields of study by confronting existing claims and ideas using new evidence (Brandt et al., 2014; Isager et al., 2023; Long, 1993; Santos, 1989). Thus, claims that replication research is unoriginal are misguided and not consistent with contemporary understandings of originality.

Innovation is also vital to the growth and strengthening of the discipline, but it, too, is misunderstood as describing something completely new, untested, and radical in nature with incremental innovation not counting (Kahn, 2018; Nind, Wiles, Bengry-Howell, & Crow, 2013; Wiles, Crow, & Pain, 2011). Clearly, setting this as a target for a research study would be both impossible to achieve and undesirable, rendering most SLA research lacking innovation. Rather, innovation exists along a continuum, is not a binary phenomenon, and is defined by small, gradual changes to current approaches (Ledgerwood, 2014; Long, 1993). This view of innovation is consistent with contemporary understandings of innovation as a cumulative process in which new lines of research emerge by building on established knowledge. In the next section, I review some of the ways that innovation and originality apply to the design and conduct of replication studies in SLA, highlighting how we can better support scientific progress by using replication to systematically revisit prior claims to confirm their stability and detect potential anomalies. I conclude this section by stressing that we must eliminate outdated notions and misconceptions of replication as lacking originality and innovation. These attitudes are unhelpful, misguided, and harmful to the discipline.

Approaches to replication

Because a replication study can come in different forms, several categorizations have been developed to convey the amount of intentional change implemented in the replication (Brandt et al., 2014; LeBel et al., 2018; Marsden et al., 2018; Porte & McManus, 2019). In an exact replication, for example, the selected study’s entire design, methods, and procedure are followed without alteration. In a close replication, one major variable is intentionally modified, while an approximate replication intentionally modifies no more than two major variables (for approaches to variable selection and modification in replication studies, see Brandt et al., 2014; LeBel et al., 2018; Porte & McManus, 2019). In a conceptual replication, however, almost all aspects of the previous study can be modified because the aim here involves “repetition of a test of a hypothesis or a result of earlier research with different methods” (Schmidt, 2009, p. 91). It is important to note, however, that drawing meaningful comparisons between studies (i.e., initial and replication) is difficult when several major modifications are integrated into the replication. This makes conceptual replications an inappropriate choice for revisiting prior studies to confirm results. At the same time, exact replication in the social sciences is probably one of the most difficult, if not impossible, types of replication study to conduct (Stroebe & Strack, 2014). Even though it might be possible to locate and use the previous study’s materials and analyses, keeping the data sample the same is very difficult when working with human subjects (Nosek & Errington, 2020). Of course, there could be similarities (e.g., L2 English learners in Austria and Germany), but several factors make the samples different (e.g., contexts, times, individual backgrounds).

In terms of using replication to revisit prior studies to confirm results and conclusions, close and approximate replication studies that include the least intentional modification are the most desirable because they allow for the greatest amount of systematic comparison (see Figure 1 for an overview of replication approaches).

Figure 1. Overview of approaches to replication.

The amount of modification between the initial study and the replication should always be reported in the study’s title, abstract, and main text (Appelbaum et al., 2018; McManus, 2022a, 2024), with labels such as close replication or approximate replication as well as summary tables to document between-study differences and similarities. This means that a replication should be clearly identified as a replication study from the outset. Furthermore, it is important for authors to identify, categorize, and report elements of the replication that are (a) the same as, (b) close to, or (c) different from the initial study (Brandt et al., 2014; LeBel et al., 2018; Marsden et al., 2018; McManus, 2023). In a close replication study, for instance, in which one variable is intentionally modified and all other differences are minimal, a minor modification refers to elements of the replication that are very close to those in the initial study, whereas a major modification refers to variables that are different between the studies. In addition, a major modification will represent the research focus of the replication, whereas a minor modification should not. In a close replication investigating whether web-based instruction influences the learning outcomes reported in a classroom-based study, for example, modifying the modality of instruction from in-person to web-based would represent an element of the initial study’s design that is different in the replication. In this example, the modality of the instruction (from in-person to web-based) can be categorized as a major variable modification. Communicating and justifying all between-study differences can also serve several benefits. For example, readers can judge if a modification appears justified or not and also where the replication is on the continuum from close to conceptual. Reporting all between study differences also allows future work to evaluate the boundary conditions on a particular effect and to develop guidance for evaluating magnitudes of change in replication research.

Supporting future replication efforts in the field

Supporting and improving the conduct and dissemination of replication studies is vital to the growth of the discipline. Doing so requires the field to develop and apply standards and expectations in design and reporting so that replication studies can meet their aims to confront and revisit existing understanding with new evidence. As a result, replication reporting must include the following components, as a minimum: a clear rationale for why the replication was carried out, full and transparent descriptions of differences and similarities in the design, methods, and results, and a framework for determining and evaluating the replicability of effect. Critically, however, these components are neither encouraged nor discussed in general reporting guidelines for empirical research in SLA, including research methodology textbooks (e.g., Mackey & Gass, 2022; Rose, McKinley, & Baffoe-Djan, 2020) and journal reporting guidelines (e.g., Language Learning, Studies in Second Language Acquisition; see McManus, 2024a). This is because the aims of most empirical studies (or extension studies) are different from those of replication studies. For one, extension studies aim to extend a current line of research in new ways (e.g., contexts, methods, populations), whereas a replication study aims to critically revisit one study to better understand its findings and impact on the discipline. This difference in aims impacts how the study is designed and reported, which explains why the information reported in replication studies is often piecemeal, incomplete, and difficult to understand and evaluate. There is thus a compelling need for disciplines to develop resources to support the design, conduct, and reporting of replication studies (e.g., Language Teaching Review Panel, 2008; Marsden et al., 2018; McManus, 2024a; Porte, 2012; Porte & McManus, 2019). In the next sections, I focus on three areas of current and future investment central to the growth of the discipline and the role of replication studies in that growth: reporting, design, and evaluation.

Replication reporting

In SLA research, improvements to the quality and transparency of reporting are needed (see Plonsky, 2013, 2024), and this situation has been noted for replication studies as well (Brown, 2012; Marsden et al., 2018; Polio & Gass, 1997; Porte & McManus, 2019). The task is to develop and implement reporting strategies to maximize the transparency and openness of a replication project, including, but not limited to, transparent study labeling, definition, and description of the replication approach, description and justification of between-study modifications, comparative reporting in the replication study’s design and methods, and evaluation of the initial study’s results in light of those reported in the replication. One resource contributing to innovation in this area includes a new standard for replication studies in applied linguistics (McManus, 2024a, see also 2023; Porte & McManus, 2019). This standard outlines basic expectations in replication reporting so that outcomes can be better understood and evaluated. Table 1, adapted from McManus (2024a), outlines the minimum requirements in replication reporting for each section of a publication. The replication studies included in this special issue closely align with this standard for replication reporting.

Table 1. Reporting expectations for replication studies in SLA research (adapted from McManus 2024a)

Replication design and study selection

In terms of supporting the design of high-quality replication studies, Brandt et al. (2014) and Isager et al. (2023) are two innovations that review and propose new ways for approaching the design of replication studies to maximize impact.

First, Brandt et al.’s (2014) “Replication Recipe” guides researchers through a series of questions and decisions for conducting a close replication study (see also McManus, 2023; Porte & McManus, 2019). In addition to spelling out a “convincing close replication par excellence” (e.g., defining the effects and methods that will be replicated a priori, following the initial study’s design and methods as closely as possible), Brandt et al. (2014) developed a 36-question guide that researchers can use to design a close replication study. The 36 questions are grouped into the following topics: nature of the effect to be replicated, design of the replication study, differences and similarities between the initial and replication study, analysis and evaluation of the replication study, registering the replication, and reporting the replication study. This guide also exists as a preregistration template on the Open Science Foundation website (OSF, https://osf.io/4jd46), which can be used by researchers to preregister a replication study. For example, the Replication Recipe begins with several questions about the aims of the proposed replication study, including what the effect is to be replicated, why this effect needs to be replicated, what is the effect size, confidence interval, and sample size of the initial effect, and where was the initial study was conducted (e.g., data collection location, region, country). Together, these questions guide researchers through several important aspects of doing replication research by focusing on what finding(s) the study seeks to replicate, why the replication study is needed/important, what the (statistical) parameters of the initial finding were, and contextual features of the result.

In terms of designing the replication study, researchers are provided with several questions that promote reflection and planning before proceeding with the replication, including whether the initial study’s materials are available, where the replication will take place, what the target sample size is, and with justification. Indeed, addressing these questions before carrying out the replication study is important because it helps researchers consider the feasibility of conducting the replication study. For example, not having access to the initial study’s materials can make conducting a replication difficult (but not impossible). Similarly, planning the target sample size helps determine to what extent the required number of participants can be accessed and recruited.

Perhaps one of the most useful elements of the Replication Recipe is documenting differences between the initial and replication studies, an essential component of designing, conducting, and reporting replication studies. The Replication Recipe includes nine questions, each with a response of exact, close, or different; for example:

• • The similarities/differences in the instructions are: exact, close, or different

• • The similarities/differences in the measures are: exact, close, or different

• • The similarities/differences in the stimuli are: exact, close, or different

• • The similarities/differences in the procedure are: exact, close, or different

• • The similarities/differences in the location are: exact, close, or different

• • The similarities/differences between participant populations are: exact, close, or different

Using these questions, researchers can determine to what extent the replication constitutes a close replication attempt (compared with an approximate or conceptual replication). In cases where “different” is selected for most responses, this means that the type of replication study being planned is conceptual (e.g., different instructions, measures, stimuli, location, participant populations).

In addition, the importance of deciding which specific study to replicate cannot be overlooked. To this end, Isager et al. (2023) proposed a decision model for replication selection under resource and knowledge constraints, which is useful for helping researchers think about the reasons for selecting a study for replication. This is because there are often multiple potential candidate studies in a topic/line of research that could be replicated, but constraints will necessarily prevent replication of all of them (e.g., time, resources). The question comes down to how one decides among them.

The literature has proposed several justifications for why a study might be selected for replication, including statistical (e.g., inconsistent results, low statistical power, low precision), theoretical (e.g., initial finding is theoretically important), methodological concerns (e.g., threats to validity), academic impact (e.g., frequently cited study), public/societal impact (e.g., findings used in popular press, education, policy; see Mackey, 2012; Nosek & Lakens, 2013; Porte, 2012, 2013). In terms of a consensus, it is broadly agreed that the claims/studies most in need of replication are those in which (a) the study’s design and methods are sound, and (b) the study is important and/or valuable, but (c) the evidence is ambiguous (Brandt et al., 2014; Nosek & Errington, 2020; Porte & McManus, 2019). Thus, claims that are important/valuable with unambiguous evidence are less likely to need corroboration through replication.

Of course, a key question here is how to determine whether a study’s design and methods are sound, if the study is valuable and/or important, and if the evidence underlying the claim is ambiguous or not. Even though there is no universally accepted means to answer these questions, there are steps researchers can consider. For instance, the academic, social, and theoretical impact of a study can inform decisions about the importance of the study (e.g., if the article is cited frequently in the field, and if the study’s findings are discussed in textbooks). In addition, closely reviewing the analysis and results can provide information about the quality of the evidence provided (e.g., do the findings come with small effect sizes, wide confidence intervals, and/or from small samples?). On this backdrop, Isager et al.(2023) recommended that researchers prioritize four factors when making decisions about what to replicate:

• • Value/importance of the claim

• • Uncertainty of the evidence underlying the claim

• • Ability of the replication study to reduce uncertainty about the claim

• • Costs and feasibility of executing a replication study

Replication evaluation

In addition to questions about replication reporting and design, an important consideration in replication research is determining how to evaluate the replicability of an effect. Or, otherwise said: How can results be compared to determine whether the replication has arrived at the same sets of results as the initial study? Marsden et al. (2018) reported that replications in SLA research mostly used the following methods: narrative comparison, mentioning the findings of the initial study, and dichotomous interpretation using p-values. These represent a good starting point, but researchers would benefit from considering additional methods for evaluating the replicability of an effect. One study that represents a particular innovation in this regard that can strengthen replication evaluation and reporting is Errington et al. (2021).

In this study, Errington et al. (2021) developed and examined several criteria to assess the replicability of preclinical research in cancer biology:

• • Effect in the initial and replication studies patterns in the same direction and is statistically significant (p < 0.05)

• • Initial effect size falls within the 95% confidence interval of the replication effect size

• • Replication effect size falls within the 95% confidence interval of the initial effect size

• • Replication effect size falls within the 95% prediction interval of the initial effect size

• • Meta-analysis combining original and replication effect sizes is statistically significant (p < 0.05)

Overall, Errington et al. (2021) showed replication rates as follows: 47% for same direction and statistical significance, 25% for the original effect size being inside the 95% confidence interval (CI) of the replication, 48% for the replication effect size being inside the 95% CI of the original, 61% for the replication effect size being inside the 95% prediction interval, and 63% for a criterion based on a meta-analytic combination of the data from the initial experiment and the replication. In addition, Errington et al. (2021) found that null results reported in the initial study were more likely to replicate than positive results in the initial study. Errington et al.’s findings and their recommendations for evaluating replication are useful because they illustrate the many varied ways that results can be evaluated. They also suggest that the most stringent tests of replicability will include more than one assessment. Implementing even just one of these methods would be a solid start toward moving the field beyond narrative comparisons only.

The special issue

To date, replication studies in SLA research have been rare and of varying quality with a preference for conceptual over approximate and close replication designs. In addition, the transparency and completeness of replication reporting have—in many instances—been low. This special issue responds to these limitations in replication design and reporting with eleven replication studies. Thus, this special issue aims to showcase and highlight exemplary standards in replication study design and reporting to benefit the field. Indeed, each study can be regarded as a role model usable by researchers and students in conducting their own replication studies and teaching others about replication design and reporting.

In compiling the special issue, a call for proposals was launched in December 2022, in which prospective contributors were invited to submit short proposals no longer than four pages that stated (i) the study to be replicated with justifications, (ii) the type of replication study to be completed, (iii) modifications to be implemented in the replication with justifications, (iv) design of the replication study, (v) data analysis plan, and (vi) potential impact. Forty-seven proposals were received, and each was independently evaluated by two reviewers (Kevin McManus and an SSLA editorial board member) using the criteria provided in the call. Following this process, nineteen proposals were selected for potential inclusion, subject to favorable evaluation following external peer review. At that point, authors were provided with recommendations for reporting replication studies as presented in Appelbaum et al. (2018) and Porte and McManus (2019), including:

• • Report the type of replication in the title and abstract, as follows: exact, close, approximate, or conceptual.

• • Report whether the replication has conditions, materials, and/or procedures that were not part of the initial study.

• • Present justifications and descriptions for all additions and variable modifications.

• • Summarize all changes between the initial and replication studies in a table.

• • Clearly label results from additional or different analyses.

• • State the criteria for deciding whether the initial study’s results were replicated. Examples of criteria include statistical significance testing, effect sizes, confidence intervals, and Bayes factors in Bayesian methods.

• • Include in the discussion a section titled “Future replication research” where suggestions for future replication studies building on the current replication study are made.

This process culminated in a special issue of eleven replication studies involving a variety of topics, populations, replication designs, motivations for replications, and initial studies that integrate open science practices (e.g., open data, open materials, preregistration). In the remainder of this introduction, I provide an overview of each study in this special issue, highlighting its main components and discussing how the replication study strengthens the field and advances knowledge and understanding about the topic.

Barlow et al. (2024) examined the ways that “native” and “non-native” listeners judge “foreign-accented” speech, contributing to an established line of SLA research suggesting that statements produced by “foreign-accented” speakers are less likely to be judged as true than those produced by “native speakers” (e.g., Hanzlíková & Skarnitzl, 2017; Lev-Ari & Keysar, 2010). While some studies suggest that these results might be attributable to processing difficulty, accent-based prejudice, or both, findings are mixed, at best, resulting in very little clarity and consensus on this topic. To address this limitation in the field, Barlow et al. (2024) carried out a close replication study of Boduch-Grabka and Lev-Ari (2021), a study that isolated the impact of processing difficulty on veracity judgments by exposing listeners to “native” or “non-native” accents prior to completing a veracity judgment task. The close replication retained all aspects of the initial study’s design, with the addition of an explicit bias task to independently examine the contributions of processing difficulty and accent-based prejudice to listeners’ judgments of veracity. The initial study’s audio materials were accessed from a public repository (Open Science Foundation, https://osf.io), with additional materials provided by the authors via an email request from Barlow et al. Overall, the initial study’s findings were not replicated: “we did not reproduce the effect of accent on veracity judgments that have been previously reported (Boduch-Grabka & Lev-Ari, 2021; Lev-Ari & Keysar, 2010) or the Boduch-Grabka and Lev-Ari (2021) finding that prior exposure to Polish-accented English speech improves veracity judgments of Polish-accented statements.” Furthermore, results from the explicit bias task did not appear to shed further light on listeners’ judgments. In terms of why Barlow et al.’s findings did not replicate the initial study, the authors suggest that methodological differences, including “insufficient methodological detail provided by the original researchers,” between the studies may explain the different patterning of results (e.g., time frame differences, social, cultural, and political effects, participant recruitment, potential inclusion criteria differences). Going forward, Barlow et al. (2024) recommend that previous studies reporting an accent-based reduction in veracity judgments “may not be robust to replication” and that future replication studies are needed to determine the factors that influence the presence/absence of such effects in listening.

Gudmestad, Edmonds, Henderson, and Lindqvist (2024) examined the extent to which L2 learners’ interpretation of verbal moods in L2 Spanish varied as a function of L2 proficiency and L1 background through a close replication of Kanwit and Geeslin (2014), an important study that “initiated L2 research on the interpretation of variable structures and because it demonstrated that the traditional interpretations of verbal moods in adverbial clauses in Spanish are variable for learners and NSs.” To examine the role of L1 background in the replication, Gudmestad et al. (2024) recruited Swedish-speaking and French-speaking learners of L2 Spanish who completed the same verbal mood interpretation task and grammar test (to assess L2 proficiency) as the initial study, whereas Kanwit and Geeslin (2014) recruited English-speaking learners only. To assess proficiency effects, participants were grouped by (i) institutional course level and (ii) performance on the grammar test, leading to two groups Level 1 and Level 2. In terms of evaluating the replicability of the findings, Gudemestad et al. used three criteria: crosstabulations of responses, statistical significance in the replication, and the direction of effect estimates for statistically significant results in the replication. Overall, the findings broadly confirmed those reported in the initial study. However, some differences emerged from the variable modification (L1 background). For example, while French-speaking learners’ patterns of interpretation of verbal mood patterns did not shift substantially between proficiency levels for interpretation, Swedish-speaking learners showed patterns of interpretation more consistent with the initial study. As discussed in the replication, this pattern of results (among others) suggests caution in the initial study’s proposed developmental stages for mood because L1 background was found to be an important variable not examined in the initial study.

Hamada, Shimizu, Hoshino, Takaki, and Ushiro (2024) conducted a secondary meta-analysis of Jeon and Yamashita (2022), a frequently cited and pedagogically important study in the field, to (a) ascertain whether the initial study’s metanalytic findings can be reproduced and (b) explore the extension of their findings to the L2 SVR model using a meta-analytic structural equation modeling (MASEM) approach. Furthermore, reproducing meta-analysis is widely agreed to be important (but seldom undertaken) to account for and explore unresolved questions from prior meta-analytic works (Plonsky, 2012; see also Boers, Bryfonski, Faez, & McKay, 2021). Hamada et al. (2024) worked toward this goal by using the same data set and meta-analytic calculations and procedures from the initial study. By adding MASEM (as an extension of the initial study), this secondary meta-analysis has the potential to provide a more nuanced, comprehensive, and robust account of effects on L2 reading outcomes than explored in the initial study (e.g., metalinguistic skills, language comprehension skills, word decoding skills). This is because the initial study used a correlation-based analysis that limits conclusions to a single effect, whereas MASEM allows for more than just one effect to be investigated simultaneously while also managing latent variables. In terms of evaluating the comparability of the results, Hamada et al. compared the average correlation coefficients and their statistical significance as well as standardized mean differences between the studies. Overall, some previously reported effects on reading comprehension were confirmed (e.g., decoding, orthographic knowledge), while others were not (e.g., L2 phonological awareness, L1 reading comprehension). The authors discuss the interpretation of the different sets of results as a potential consequence of data recoding.

Liu and Lu’s (2024) close replication study of Römer and Berger (2019) addressed long-standing concerns with data sparsity in learner corpora and the resulting claims about group-level L2 developmental trajectories. Data sparsity, Liu and Lu discuss, refers to the small-scale nature of most existing learner corpora, which can compromise statistical analyses. The issue of data sparsity can also be found in Römer and Berger (2019), a theoretically and empirically important study in the field showing that the associations between verbs and verb argument constructions (VACs) used by German and Spanish learners of English move closer to L1 usage norms as the learners’ proficiency increases. In that study, however, which involved correlation analyses of the first version of the EF-Cambridge Open Language Database (Alexopoulou, Geertzen, Korhonen, & Meurers, 2015) and the British National Corpus (BNC Consortium, 2007), a large number of combinations between verbs and VACs in the BNC were absent in the learner corpus. This issue was addressed with log transformation, an approach with several known limitations, especially for counting data with many zeros. Liu and Lu’s replication study addressed this concern by retaining the initial study’s overall design and methods, making one variable modification: using the EFCAMDAT2 corpus, an updated and larger version of the corpus used in the initial study. Also, to address analytical concerns, Liu and Lu (2024) carried out both correlational analyses (as in the initial study) as well as zero-inflated negative binomial regression analyses, a statistical approach that accounts for count data with overdispersion and excess zeros (Winter, Perlman, & Majid, 2018). Liu and Lu’s findings replicated those reported in the initial study. For example, both studies reported that learners produced a wider variety of verbs in specific construction types (e.g., V about N) as their L2 proficiency increased. While not discussed in the initial study, Liu and Lu’s (2024) analysis also indicated an influence of writing topics on the learners’ use of verbs. In addition, Liu and Lu (2024) found that the use of zero-inflated negative binomial (ZINB) models to account for count data with overdispersion and excess zeros corroborated the correlation analyses, showing that (i) L1 verb–VAC frequency was significantly positively associated with L2 verb–VAC frequency and (ii) that increased overlap between the verb–VAC combinations used by L1 users and higher proficiency L2 learners. Taken together, this replication corroborated the initial study’s findings while showcasing the advantages of ZINB models in analyzing corpus frequency data with overdispersion and excess zeros.

Huensch’s (2024) preregistered close replication study of Darcy, Mora, and Daidone (2016), a pioneering study in the field of SLA that explored the relationships among general cognition and language abilities, examined relationships between L2 speech perception and production and inhibitory control. In that study, Darcy et al. (2016) investigated to what extent variability in L2 pronunciation outcomes could be explained by individual differences in inhibitory control. Using a retrieval-induced inhibition task and measures of perception and production, Darcy et al. found important relationships between inhibition and perception, with little impact of inhibition on production, a finding that might be explained by the type of inhibition task used. Huensch thus addressed this limitation by adding a different measure of inhibition (Stroop task, Simon task) to the study design and keeping all other aspects of the study unmodified. To support replication, the initial study’s authors provided their materials and tasks on the Instrument Repository for Research in Second Language Studies (IRIS; see Marsden, Mackey, & Plonsky, 2016). To determine the replicability of an effect, means, and standard deviations from the descriptive statistics of the two studies were used to compute Hedge’s g effect sizes and corresponding CIs. Hedge’s g <. 40 and corresponding CIs passing through zero were taken to indicate negligible differences. Statistical tests were interpreted using the directionality (positive vs. negative) and magnitude of the effect size. Overall, the initial study’s findings were not replicated: “no strong, clear, or consistent relationship emerges between inhibitory control and L2 perception/production skills.” Thus, Huensch explores several factors that may explain these findings, including a lack of statistical power in the initial study.

Given substantial evidence to date about the benefits of providing aural support during reading (or reading-while-listening) for vocabulary learning (e.g., R. Brown, Waring, & Donkaewbua, 2008; Vu & Peters, 2022), very little is known about the ways in which L1 background may interact with the effectiveness of this instructional method, especially when the transparency of grapheme–phoneme correspondences varies between L1 and L2. To this end, the Twilex Group (2024) conducted a preregistered close replication study of Malone (2018), a methodologically rigorous, transparent, and important study in research examining the effectiveness of reading-while-listening, which systematically integrated L1 differences into the study design. In the initial study, the main research goal was to compare the effectiveness of reading-while-listening (RWL) and reading-only (RO) on incidental vocabulary learning, including to what extent does learning depend on the amount of exposure (two vs. four exposures) and working memory (WM). The replication’s major variable modification was the L1 background of the learners, achieved by recruiting L1 speakers of Chinese (Mandarin, Cantonese) and Germanic languages (German, Dutch), who were L2 learners of English. The initial study’s participants spoke a range of L1s. Any other modifications to the initial study’s design and methods were considered minor in scope (e.g., online instead of in-person data collection). The close replication study obtained all materials either through personal communication with the author of the initial study or from the IRIS database. Claims about the replicability of an effect were determined if the replication obtained the same direction and size of the correlations as those reported in the initial study. Overall, this close replication study obtained mixed results in comparison to the initial study. For example, the Twilex Group found few meaningful differences between the RWL and RO reading conditions, regardless of the amount of exposure, whereas the initial study found RWL to be more effective than RO after two exposures. Between-study similarities were found for working memory, though, showing that higher working memory capacities resulted in greater learning gains. In terms of the main objective of the replication, to assess what role L1 background played, the results indicated important interactions among L1 background, WM, and reading condition (RWL, RO). For example, the demand of WM was greater for L1 Chinese learners in RWL conditions. Taken together, this replication study sheds important light on the effectiveness of bimodal input on L2 learning outcomes and the potentially mediating role that L1 background can play.

Revisiting the role of transfer in third language (L3) acquisition, Parrish (2024) reports an approximate replication of Rothman (2011), a theoretically and empirically important study in the field examining how the perceived typological similarity of previously learned languages shaped L3 learning. Indeed, the Typological Primacy Model, an important theory in L3 research, is based on Rothman (2011). In addition, Parrish (2024) notes limitations with the initial study’s small sample size and the statistical tests used. In the approximate replication study, two major changes were implemented. First, new materials were created because the ones used in the initial study were lost and thus not available. These materials were recreated based on the descriptions and examples provided in the initial study. Second, new speaker groups were created. In the initial study, speakers were (a) L1 English–L2–Spanish–L3 Brazilian Portuguese and (b) L1 Italian–L2 English–L3 Spanish. In the replication study, mirror image groups of Spanish–English bilinguals were recruited: (a) L1 English–L2 Spanish–L3 Brazilian Portuguese and (b) L1 Spanish–L2 English–L3 Brazilian Portuguese. The sample size was also increased. Overall, Parrish’s (2024) analysis showed that some aspects of the initial study’s findings replicated while others did not. For example, whereas the initial study showed that both learner groups showed evidence of transfer by Spanish in their L3, these results were only partially replicated by Parrish (2024). The importance of adequate sample sizes is noted as a possible explanation for these findings, along with the use of different data collection materials. All in all, this replication calls for future research to consider the role of sample size in decisions about what to replicate going forward.

Revisiting a central question in bilingualism research about the ways in which L2 speakers access and activate words in their known languages, Pelzl and Van Hell (2024) conducted an approximate replication of Van Hell and Dijkstra (2002), an influential and frequently cited study in the bilingual word processing literature. In addition, Van Hell and Dijkstra (2002) focused on effects from L2 to L1, providing a particularly strong test of language nonselectivity, which serves as an empirical foundation for many theoretical models of bilingual processing in the field (e.g., Dijkstra, Wahl, Buytenhuijs, Van Halem, Al-Jibouri, De Korte, & Rekké, 2019; Shook & Marian, 2013). For these reasons, determining the replicability of Van Hell and Dijkstra’s (2002) findings is important for continued theorization and growth in the field. In this approximate replication study, Pelzl and Van Hell (2024) made two major modifications to the initial study. First, the target population was modified from Dutch–English–French trilinguals to Dutch–English bilinguals. This change was implemented to test whether the initial study’s findings held when there was no L3 present, thus strengthening and refining the claims made in the initial study. Second, the study was conducted online rather than in-person in a lab context to recruit a more diverse and larger group of participants, thus addressing small sample size concerns in the initial study. All other aspects of the initial study’s design and methods were not intentionally modified, with any other changes considered to be minor in scope. Also, since Van Hell was involved in both the initial study and this replication, additional information about the study and access to the materials was available (e.g., providing stimuli not provided in the initial study), thus providing a critical resource for the replication study. In terms of determining the replicability of the findings, Pelzl and Van Hell’s (2024) replication used two indices: the direction and statistical significance of effects. Overall, Pelzl and Van Hell (2024) found the same patterns of results as those reported in the initial study, confirming that L2 knowledge can influence L1 performance in important and detectable ways. Furthermore, by recruiting a large sample of participants, the replication study showed stronger effects than the initial study, some of which did not research statistical significance. For example, the initial study observed a 10-ms facilitation effect, a result that was not statistically significant, but a larger sample of participants resulted in a facilitation effect that was 19 ms, which did reach statistical significance. Taken together, this replication study not only confirmed the foundational claims made in the initial study but also strengthened them in many cases, thus providing a strong validation of the initial study claims.

In line with Pelzl and Van Hell (2024), the close replication study conducted by Ryan et al. (2024) also included a researcher from the initial study. The study selected for replication was Tavakoli and Foster (2008), a theoretically and empirically important study in the field of SLA that is also frequently cited in the literature. That study investigated the ways in which L2 oral performance was influenced by narrative task design, showing that narratives with both foreground and background information elicited significantly greater syntactic complexity than those with only foreground information. In this close replication study, Ryan et al. (2024) included one intentional modification only by adding L1 literacy. This was achieved by recruiting adult refugees to New Zealand with low L1 literacy. This variable modification is important for several reasons, not only because most SLA research is based on highly literate populations (see Andringa & Godfroid, 2020), but also because L1 literacy can strongly influence language processing and metalinguistic knowledge. Indeed, advanced levels of L1 literacy have been associated with more accurate L2 performance (Havron & Arnon, 2017). Thus, this close replication study set out to examine to what extent L2 oral performance can also be influenced by low levels of L1 literacy, thereby impacting many theories of L2 learning concerning explicit teaching, metalinguistics knowledge, the provision of corrective feedback, and key aspects of language instruction that depend on metalinguistic knowledge and its processing. All other aspects of the initial study’s design and methods were not intentionally modified, with all other differences considered minor in scope. Ryan et al. (2024) used indices of direction and statistical significance to determine the replicability of the initial study’s findings. Overall, the replication study did not confirm the initial study’s findings about interactions between narrative task design and L2 oral performance because background information in the narrative tasks had no impact on the syntactic complexity, lexical diversity, or fluency of learners’ performance. For example, participants in the replication recounted narrative tasks with and without background events in comparable ways, whereas the initial study found that background events led to performance with longer and more complex syntactic patterns. Thus, Ryan et al.’s (2024) findings indicated that learners let background events in the narrative pass unremarked, instead focusing on foreground events. Informed by these findings, Ryan et al. (2024) suggest that schooling experience and levels of visual literacy may explain these findings better than levels of low literacy. This is because, the authors argue, the replication did not require literacy-supported cognitive tools (e.g., awareness of word boundaries) to follow sequences of events as presented in picture-based narratives. Thus, as an active area for future SLA research, Ryan et al. (2024) encourage replication of key studies in task-based learning and teaching, which tend to presume experience and familiarity with visual prompts. Such study designs would appear not to work as well with all learner populations.

The last study in this special issue that included a researcher from the initial study is Wong and Prange (2024), a close replication of Wong, Zhao, and MacWhinney (2018), an important study in instructed SLA about the specific contribution of instruction couched in cognitive linguistics when compared with “traditional” instruction. In the initial study, Wong et al. (2018) developed an experimentalized computer-assisted language learning (eCALL) system called the English Preposition Tutor, to investigate the effectiveness of a cognitive linguistics approach to learning English prepositions, which indicated promising findings for this instructional approach. However, as noted by Wong and Prange (2024), the initial study included no delayed posttest, and the statistical analyses (analysis of variance, ANOVA) focused on analyzing separate effects only, leading to a limited understanding of the initial study’s effectiveness in supporting L2 development. Thus, in this close replication study, Wong and Prange (2024) integrated two intentional modifications to the initial design. First, rather than using ANOVAs to analyze separate effects, Bayesian mixed-effects models were used. Importantly, the initial study’s findings were used to inform the study’s priors (see Garcia, 2021; Norouzian, de Miranda, & Plonsky, 2018). Second, a three-week delayed posttest was added to examine the potential for extended effects of learning in the replication. In line with recommended practice, the sample sizes in the instructional groups were also increased. All other aspects were not intentionally modified. Overall, Wong and Prange’s results are broadly consistent with those reported in the initial study. For example, all groups improved significantly following the instruction, as also reported in the initial study. The incorporation of the delayed posttest was able to show that learning gains documented immediately after the instruction were broadly maintained after the instruction. Thus, in sum, Wong and Prange’s replication study was able to broadly confirm the findings reported in the initial study.

In an approximate replication study, Wang and Sun (2024) revisited the Second Language Motivational Self System (Dörnyei, 2005, 2009), an influential theoretical framework in programs of empirical research and theory-building about language learner’s motivation and its role in second language learning. However, recent commentary has expressed concerns with this framework, highlighting issues such as “the fantasy problem,” the “ought-to L2 self problem,” and the “context problem” (Al-Hoorie, Hiver, & In’nami, 2024). Papi et al. (2019) responded to these concerns about the Second Language Motivational Self System with the 2 × 2 model, designed to offer a more nuanced account of motivational dynamics. Indeed, subsequent research across educational and cultural contexts has indicated that the 2 × 2 model appears to more effectively predict language learning behaviors and outcomes (e.g., Zhou & Papi, 2023). Indeed, Papi and Khajavy (2021) showed how this model can account for language learning outcomes in an Iranian context, thus providing useful information about the applicability of the 2 × 2 model in a novel context within this line of research. In the current approximate replication, Wang and Sun (2024) sought to revisit and extend this line of research in a Chinese English as a foreign language (EFL) context given the relatively different educational and cultural contexts in Iran and China. For example, students receive English language instruction at a younger age in China, and China’s promotion of English aligns with its broader strategy for global competitiveness, factors that could impact motivational dynamics toward language learning outcomes. In their approximate replication study, Wang and Sun (2024) intentionally modified three variables: learner background, the language used in the questionnaire, and the method of measuring L2 achievement. The replication team used the same questionnaire as used in the initial study (obtained from IRIS) and increased the sample size from 324 in the initial study to 855. Using this design, Wang and Sun (2024) examined to what extent Papi and Khajavy’s (2021) findings hold in a Chinese EFL context. Overall, Wang and Sun (2024) findings broadly replicated those reported in the initial study, especially in terms of how regulatory focus plays a critical role in language learning. One notable difference, however, was that the replication reported stronger impacts of ought-to-selves, anxiety, and enjoyment on language learning outcomes. The authors discuss these different patterns of results in light of the educational and contextual differences between Iran and China.