Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-gxg78 Total loading time: 0 Render date: 2024-12-17T20:54:24.804Z Has data issue: false hasContentIssue false

24 - Random Factors and Research Generalization

from Part IV - Understanding What Your Data Are Telling You About Psychological Processes

Published online by Cambridge University Press:  12 December 2024

By
Charles M. Judd  and
David A. Kenny
Edited by
Harry T. Reis ,
Tessa West  and
Charles M. Judd
Harry T. Reis
Affiliation:
University of Rochester, New York
Tessa West
Affiliation:
New York University
Charles M. Judd
Affiliation:
University of Colorado Boulder
Get access

Summary

In most social psychological studies, researchers conduct analyses that treat participants as a random effect. This means that inferential statistics about the effects of manipulated variables address the question whether one can generalize effects from the sample of participants included in the research to other participants that might have been used. In many research domains, experiments actually involve multiple random variables (e.g., stimuli or items to which participants respond, experimental accomplices, interacting partners, groups). If analyses in these studies treat participants as the only random factor, then conclusions cannot be generalized to other stimuli, items, accomplices, partners, or groups. What are required are mixed models that allow multiple random factors. For studies with single experimental manipulations, we consider alternative designs with multiple random factors, analytic models, and power considerations. Additionally, we discuss how random factors that vary between studies, rather than within them, may induce effect size heterogeneity, with implications for power and the conduct of replication studies.

Keywords

random factorsmixed modelsresearch designexperimental designgeneralizationstatistical analysis
Type
Chapter
Information
Handbook of Research Methods in Social and Personality Psychology , pp. 602 - 621
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Ackerman, R. A., Kashy, D. A., and Corretti, C. A. (2015). A tutorial on analyzing data from speed-dating studies with heterosexual dyads. Personal Relationships, 22, 92–110.CrossRefGoogle Scholar
Barr, D. J., Levy, R., Scheepers, C., and Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255–278.CrossRefGoogle ScholarPubMed
Bates, D., Kliegl, R., Vasishth, S., and Baayen, H. (2015). Parsimonious mixed models. arXiv, 1506.04967v1 [stat.ME].Google Scholar
Beck, A. T., Steer, R. A., and Carbin, M. G. (1988). Psychometric properties of the Beck depression inventory: Twenty-five years of evaluation. Clinical Psychology Review, 8, 77–100.CrossRefGoogle Scholar
Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., and Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561–571.CrossRefGoogle ScholarPubMed
Belmi, P., and Schroeder, J. (2021). Human “resources”? Objectification at work. Journal of Personality and Social Psychology, 120, 384–417.CrossRefGoogle ScholarPubMed
Brauer, M., and Curtin, J. J. (2018). Linear mixed-effects models and the analysis of nonindependent data: A unified framework to analyze categorical and continuous independent variables that vary within-subjects and/or within-items. Psychological Methods, 23(3), 389–411.CrossRefGoogle ScholarPubMed
Brysbaert, M., and Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of Cognition, 1(1), 9, http://doi.org/10.5334/joc.10.CrossRefGoogle ScholarPubMed
Cronbach, L. J., Gleser, G. C., Nanda, H., and Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. John Wiley.Google Scholar
Eastwick, P. W., Finkel, E. J., Mochon, D., and Ariely, D. (2007). Selective versus unselective romantic D\desire: Not all reciprocity is created equal. Psychological Science, 18, 317–319.CrossRefGoogle Scholar
Fazio, R. H., Jackson, J. R., Dunton, B. C., and Williams, C. J. (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69(6), 1013–1027.CrossRefGoogle ScholarPubMed
Goenka, S., and Thomas, M. (2020). The malleable morality of conspicuous consumption. Journal of Personality and Social Psychology, 118, 562–583.CrossRefGoogle ScholarPubMed
Greenwald, A. G., McGhee, D. E., and Schwartz, J. L. (1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6), 1464–1480.CrossRefGoogle ScholarPubMed
Higgins, E. T., Rholes, W. S., and Jones, C. R. (1977). Category accessibility and impression formation. Journal of Experimental Social Psychology, 13, 141–154.CrossRefGoogle Scholar
Judd, C. M., Westfall, J., and Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103, 54–69.CrossRefGoogle ScholarPubMed
Judd, C. M., Westfall, J., and Kenny, D. A. (2017). Experiments with more than one random factor: Designs, analytic models, and statistical power. Annual Review of Psychology, 68, 601–625.CrossRefGoogle ScholarPubMed
Kenny, D. A., and Judd, C. M. (2019). The unappreciated heterogeneity of effect sizes: Implications for power, precision, planning of research, and replication. Psychological Methods, 24, 578–589.CrossRefGoogle ScholarPubMed
Klein, R. A., Vianello, M., Hasselman, F., et al. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1, 443–490.CrossRefGoogle Scholar
Kogan, N., and Wallach, M. A. (1967). Risky-shift phenomenon in small decision-making groups: A test of the information-exchange hypothesis. Journal of Experimental Social Psychology, 3, 75–84.CrossRefGoogle Scholar
Linden, A. H., and Hönekopp, J. (2021). Heterogeneity of research results: A new perspective from which to assess and promote progress in psychological science. Perspectives on Psychological Science, 16, 358–376.CrossRefGoogle ScholarPubMed
McArthur, L. A. (1972). The how and what of why: Some determinants and consequences of causal attribution. Journal of Personality and Social Psychology, 22, 171–193.CrossRefGoogle Scholar
McShane, B. B., and Böckenholt, U. (2014). You cannot step into the same river twice: When power analyses are optimistic. Perspectives on Psychological Science, 9, 612–625.CrossRefGoogle ScholarPubMed
Meagher, B. R. (2015). The effects of interpersonal differences within religious communities: A group actor–partner interdependence model of U.S. congregations. International Journal for the Psychology of Religion, 25, 74–90.CrossRefGoogle Scholar
Myers, D. G., and Lamm, H. (1976). The group polarization phenomenon. Psychological Bulletin, 83(4), 602–627.CrossRefGoogle Scholar
Simons, D. J., Holcombe, A. O., and Spellman, B. A. (2014). An introduction to registered replication reports at Perspectives on Psychological Science. Perspectives on Psychological Science, 9, 552–555.CrossRefGoogle ScholarPubMed
Stephenson, B., and Wicklund, R. A. (1984). The contagion of self-focus within a dyad. Journal of Personality and Social Psychology, 46, 163–168.CrossRefGoogle Scholar
Thorson, K. R., Mendes, W. B., and West, T. V. (2021). Controlling the uncontrolled: Are there incidental experimenter effects on physiologic responding? Psychophysiology, 57(3), e13500.CrossRefGoogle Scholar
Vesely, S., & Klöckner, C. A. (2020). Social desirability in environmental psychology research: Three meta-analyses. Frontiers in Psychology, 11, 1395, at https://doi.org/10.3389/fpsyg.2020.01395.CrossRefGoogle ScholarPubMed
Wallach, M. A., Kogan, N., and Bem, D. (1962). Group influence on individual risk taking. Journal of Abnormal and Social Psychology, 2, 72–86.Google Scholar
Westfall, J. (2017). A shiny app for power analysis with random targets and participants, https://jakewestfall.shinyapps.io/two_factor_power.Google Scholar
Westfall, J., Kenny, D. A., and Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143, 2020–2045.CrossRefGoogle ScholarPubMed
Wolsiefer, K., Westfall, J., and Judd, C. M. (2017). Modeling stimulus variation in three common implicit attitude tasks. Behavior Research Methods, 49, 1193–1209.CrossRefGoogle ScholarPubMed
Word, C. O., Zanna, M. P., and Cooper, J. (1974). The nonverbal mediation of self-fulfilling prophecies in interracial interaction. Journal of Experimental Social Psychology, 10(2), 109–120.CrossRefGoogle Scholar
Xie, S. Y., Flake, J. K., and Hehman, E. (2019). Perceiver and target characteristics contribute to impression formation differently across race and gender. Journal of Personality and Social Psychology, 117, 364–385.CrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×