Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-745bb68f8f-mzp66 Total loading time: 0 Render date: 2025-01-26T08:48:14.607Z Has data issue: false hasContentIssue false

1 - Student Samples in Research

from Part I - Quantitative Data Collection Sources

Published online by Cambridge University Press:  12 December 2024

By
Michael Basil
Edited by
John E. Edlund  and
Austin Lee Nichols
John E. Edlund
Affiliation:
Rochester Institute of Technology, New York
Austin Lee Nichols
Affiliation:
Central European University, Vienna
Get access

Summary

This chapter provides an overview on the use and validity of student samples in the behavioral and social sciences. In some instances, data collected from students can be of limited value or even inappropriate; however, in other cases, this approach provides useful data. I offer three general ways to evaluate the use of student samples. First, consider the research design. Descriptive studies that rely on students to draw inferences about the overall population are likely problematic. Second, statistical controls such as multivariate analyses that adjust for other factors may reduce some of the biases that may be introduced through sampling. Third, consider the theorized mechanism – a clear theoretical mechanism that does not vary based on the demographics of the sample allows us to put more faith in constrained samples. Despite these approaches, and regardless of our methods, statistics, and theoretical mechanism, we should be cautious with generalizability claims.

Type
Chapter
Information
The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
Volume 2: Performing Research
 , pp. 3 - 23
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Amaya, A., & Presser, S. (2016). Nonresponse bias for univariate and multivariate estimates of social activities and roles. Public Opinion Quarterly, 81(1), 1–36.Google Scholar
Aggarwal, R., & Ranganathan, P. (2019). Study designs: Part 2 – descriptive studies. Perspectives in Clinical Research, 10(1), 34–36.CrossRefGoogle ScholarPubMed
Ashraf, R., & Merunka, D. (2017). The use and misuse of student samples: An empirical investigation of European marketing research. Journal of Consumer Behaviour, 16(4), 295–308.CrossRefGoogle Scholar
Babbie, E. (1992). The Practice of Social Research, 6th ed. Wadsworth.Google Scholar
Baláž, V., Bačová, V., Drobná, E., Dudeková, K., & Adamík, K. (2013). Testing prospect theory parameters. Ekonomicky časopis, 61, 655–671.Google Scholar
Banyard, P., & Hunt, N. (2000). Reporting research: Something missing? The Psychologist: Bulletin of the British Psychological Society, 13(2), 68–71.Google Scholar
Barr, S. H., & Hitt, M. A. (1986). A comparison of selection decision models in manager versus student samples. Personnel Psychology, 39(3), 599–617.CrossRefGoogle Scholar
Basil, M. D. (1996). The use of student samples in communication research. Journal of Broadcasting and Electronic Media, 40, 431–440.CrossRefGoogle Scholar
Basil, M. D., Brown, W. J., & Bocarnea, M. C. (2002). Differences in univariate values versus multivariate relationships: Findings from a study of Diana, Princess of Wales. Human Communication Research, 28, 501–514.Google Scholar
Beebe, L. H. (2007). What can we learn from pilot studies? Perspectives in Psychiatric Care, 43(4), 213–218.CrossRefGoogle ScholarPubMed
Beets, M. W., Weaver, R. G., Ioannidis, J., Geraci, M., Brazendale, K., Decker, L., & Milat, A. J. (2020). Identification and evaluation of risk of generalizability biases in pilot versus efficacy/effectiveness trials: A systematic review and meta-analysis. International Journal of Behavioral Nutrition and Physical Activity, 17(1), 1–20.CrossRefGoogle ScholarPubMed
Bello, D., Leung, K., Radebaugh, L., Tung, R. L., & Van Witteloostuijn, A. (2009). From the editors: Student samples in international business research. Journal of International Business Studies, 40(3), 361–364.CrossRefGoogle Scholar
Berkowitz, L., & Donnerstein, E. (1982). External validity is more than skin deep: Some answers to criticisms of laboratory experiments. American Psychologist, 37(3), 245.CrossRefGoogle Scholar
Bowen, G. L. (1994). Estimating the reduction in nonresponse bias from using a mail survey as a backup for nonrespondents to a telephone interview survey. Research on Social Work Practice, 4, 115–128.CrossRefGoogle Scholar
Burnett, J. J., & Dune, P. M. (1986). An appraisal of the use of student subjects in marketing research. Journal of Business Research, 14(4), 329–343.CrossRefGoogle Scholar
Cappelen, A. W., Nygaard, K., Sørensen, E. Ø., & Tungodden, B. (2015). Social preferences in the lab: A comparison of students and a representative population. Scandinavian Journal of Economics, 117(4), 1306–1326.CrossRefGoogle Scholar
Casadevall, A., & Fang, F. C. (2008). Descriptive science. Infection and Immunity, 76(9), 3835–3836.CrossRefGoogle ScholarPubMed
Caspaldi, D., & Patterson, G. R. (1987). An approach to the problem of recruitment and retention rates for longitudinal research. Behavioral Assessment, 9, 169–177.Google Scholar
Clara, I. P., Cox, B. J., Enns, M. W., Murray, L. T., & Torgrudc, L. J. (2003). Confirmatory factor analysis of the multidimensional scale of perceived social support in clinically distressed and student samples. Journal of Personality Assessment, 81(3), 265–270.CrossRefGoogle ScholarPubMed
Coleman, S. (2007). Testing theories with qualitative and quantitative predictions. European Political Science, 6(2), 124–133.CrossRefGoogle Scholar
Compeau, D., Marcolin, B., Kelley, H., & Higgins, C. (2012). Research commentary – Generalizability of information systems research using student subjects – A reflection on our practices and recommendations for future research. Information Systems Research, 23(4), 1093–1109.CrossRefGoogle Scholar
Crabbe, B. D., & Pinkerton, K. A. (1992). Sources of bias in Health Commission and tobacco industry surveys in Australia. Australian Psychologist, 27, 103–108.CrossRefGoogle Scholar
Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93–99.CrossRefGoogle Scholar
Deffner, D., Rohrer, J. M., & McElreath, R. (2022). A causal framework for cross-cultural generalizability. Advances in Methods and Practices in Psychological Science, 5(3), 25152459221106366.CrossRefGoogle Scholar
Dura, J. R., & Kiecolt-Glaser, J. K. (1990). Sample bias in caregiving research. Journals of Gerontology, 45, P200–P204.CrossRefGoogle ScholarPubMed
Edgington, E. S. (1966). Statistical inference and nonrandom samples. Psychological Bulletin, 66(6), 485–487.CrossRefGoogle ScholarPubMed
Edlund, M. J., & Swann, A. C. (1989). Continuing in treatment as a form of selection bias. American Journal of Psychiatry, 146, 254–256.Google ScholarPubMed
Efron, B., & Tibshirani, R. (1991). Statistical data analysis in the computer age. Science, 253(5018), 390–395.CrossRefGoogle ScholarPubMed
Espinosa, J. A., & Ortinau, D. J. (2016). Debunking legendary beliefs about student samples in marketing research. Journal of Business Research, 69(8), 3149–3158.CrossRefGoogle Scholar
Falk, A., Meier, S., & Zehnder, C. (2013). Do lab experiments misrepresent social preferences? The case of self-selected student samples. Journal of the European Economic Association, 11(4), 839–852.CrossRefGoogle Scholar
Ferber, R. (1977). Research by convenience. Journal of Consumer Research, 4(1), 57–58.CrossRefGoogle Scholar
Fienberg, S. E., & Tanur, J. M. (1996). Reconsidering the fundamental contributions of Fisher and Neyman on experimentation and sampling. International Statistical Review/Revue Internationale de Statistique, 64(3), 237–253.Google Scholar
Frame, C. L., & Strauss, C. C. (1987). Parental informed consent and sample bias in grade-school children. Journal of Social and Clinical Psychology, 5, 227–236.CrossRefGoogle Scholar
Gold, R. L. (1997). The ethnographic method in sociology. Qualitative Inquiry, 3(4), 388–402.CrossRefGoogle Scholar
Goyal, R. (2015). Animal testing in the history of anesthesia: Now and then, some stories, some facts. Journal of Anaesthesiology, Clinical Pharmacology, 31(2), 149–151.CrossRefGoogle ScholarPubMed
Greenberg, J. (1987). The college sophomore as guinea pig: Setting the record straight. Academy of Management Review, 12(1), 157–159.CrossRefGoogle Scholar
Guttman, I. (1973). Care and handling of univariate or multivariate outliers in detecting spuriosity – a Bayesian approach. Technometrics, 15(4), 723–738.Google Scholar
Hallingberg, B., Turley, R., Segrott, J., et al. (2018). Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: A systematic review of guidance. Pilot and feasibility studies, 4(1), 1–12.CrossRefGoogle Scholar
Hanel, P. H., & Vione, K. C. (2016). Do student samples provide an accurate estimate of the general public? PLOS ONE, 11(12), e0168354.CrossRefGoogle ScholarPubMed
Harrison, D. A. (1995). Volunteer motivation and attendance decisions: Competitive theory testing in multiple samples from a homeless shelter. Journal of Applied Psychology, 80(3), 371–385.CrossRefGoogle Scholar
Heggestad, E. D., Rogelberg, S., Goh, A., & Oswald, F. L. (2015). Considering the effects of nonresponse on correlations between surveyed variables. Journal of Personnel Psychology, 14(2), 91–103.CrossRefGoogle Scholar
Henry, P. J. (2008). Student sampling as a theoretical problem. Psychological Inquiry, 19(2), 114–126CrossRefGoogle Scholar
Honigmann, J. J. (2003). Sampling in ethnographic fieldwork. In Burgess, R. G. (ed.), Field Research: A Sourcebook and Field Manual (pp. 134–152). Routledge.Google Scholar
Jackson, S. (1992). Message Effects Research: Principles of Design and Analysis. Guilford Press.Google Scholar
Jackson, S., & Jacobs, S. (1983). Generalizing about messages: Suggestions for design and analysis of experiments. Human Communication Research, 9(2), 169–191.CrossRefGoogle Scholar
James, W. L., & Sonner, B. S. (2001). Just say no to traditional student samples. Journal of Advertising Research, 41(5), 63–71.Google Scholar
Jennings, W., & Wlezien, C. (2018). Election polling errors across time and space. Nature Human Behaviour, 2(4), 276–283.CrossRefGoogle ScholarPubMed
Kahneman, D., Slovic, R., & Tversky, A. (1982). Judgment under Uncertainty: Heuristics and Biases. Cambridge University Press.CrossRefGoogle Scholar
Kardes, F. R. (1996). In defense of experimental consumer psychology. Journal of Consumer Psychology, 5, 279–296.CrossRefGoogle Scholar
Kim, H., Schimmack, U., Oishi, S., & Tsutsui, Y. (2018). Extraversion and life satisfaction: A cross‐cultural examination of student and nationally representative samples. Journal of Personality, 86(4), 604–618.CrossRefGoogle ScholarPubMed
Kish, L. (1957). Confidence intervals for clustered samples. American Sociological Review, 22(2), 154–165.CrossRefGoogle Scholar
Krackardt, D. (1987). QAP partialling as a test of spuriousness. Social Networks, 9(2), 171–186.CrossRefGoogle Scholar
Krupnikov, Y., Nam, H. H., Style, H., Druckman, J. N., & Green, D. P. (2021). Convenience samples in political science experiments. In Druckman, J. and Green, D. (eds.), Advances in Experimental Political Science (pp. 165–183). Cambridge University Press.CrossRefGoogle Scholar
Kruskal, W., & Mosteller, F. (1980). Representative sampling, IV: The history of the concept in statistics, 1895–1939. International Statistical Review/Revue Internationale de Statistique, 48(2), pp. 169–195.Google Scholar
Lamb, C. W. Jr., & Stem, D. E. Jr. (1980). An evaluation of students as surrogates in marketing studies. Advances in Consumer Research, 7(1), 796–799.Google Scholar
Lesser, V. M., & Kalsbeek, W. D. (1999). Nonsampling errors in environmental surveys. Journal of Agricultural, Biological, and Environmental Statistics, 4(4), 473–488.CrossRefGoogle Scholar
Lucas, J. W. (2003). Theory-testing, generalization, and the problem of external validity. Sociological Theory, 21, 236–253.CrossRefGoogle Scholar
Lynch, D. L., Stern, A. E., Oates, R. K., & O’Toole, B. I. (1993). Who participates in child sexual abuse research? Journal of Child Psychology and Psychiatry and Allied Disciplines, 34, 935–944.CrossRefGoogle ScholarPubMed
Lynch, J. G. (1982). The role of external validity in theoretical research. Journal of Consumer Research, 10, 109–111.CrossRefGoogle Scholar
McNemar, Q. (1946) Opinion attitude methodology. Psychological Bulletin, 43, 289–374.CrossRefGoogle ScholarPubMed
Meyer, J., Kohn, I., Stahl, K., Hakala, K., Seibert, J., & Cannon, A. J. (2019). Effects of univariate and multivariate bias correction on hydrological impact projections in alpine catchments. Hydrology and Earth System Sciences, 23, 1339–1354.CrossRefGoogle Scholar
Mishra, S. I., Dooley, D., Catalano, R., & Serxner, S. (1993). Telephone health surveys: Potential bias from noncompletion. American Journal of Public Health, 83, 94–99.CrossRefGoogle ScholarPubMed
Monin, B., & Oppenheimer, D. M. (2014). The limits of direct replications and the virtues of stimulus sampling. Social Psychology, 45(4), 299–300.CrossRefGoogle Scholar
Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: Investigating participant demand characteristics. Journal of General Psychology, 135(2), 151–166.CrossRefGoogle ScholarPubMed
Nielsen, M., Haun, D., Kärtner, J., & Legare, C. H. (2017). The persistent sampling bias in developmental psychology: A call to action. Journal of Experimental Child Psychology, 162, 31–38.CrossRefGoogle ScholarPubMed
Norden, K. A., Klein, D. N., Ferro, T., & Kasch, K. (1995). Who participates in a family study? Comprehensive Psychiatry, 36, 199–206.CrossRefGoogle Scholar
Payne, B. K., & Chappell, A. (2008). Using student samples in criminological research. Journal of Criminal Justice Education, 19(2), 175–192.CrossRefGoogle Scholar
Pernice, R. E., Ommundsen, R., Van Der Veer, K., & Larsen, K. (2008). On use of student samples for scale construction. Psychological Reports, 102(2), 459–464.CrossRefGoogle ScholarPubMed
Peterson, R. A. (2001). On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research, 28, 450–461.CrossRefGoogle Scholar
Peterson, R. A., & Merunka, D. R. (2014). Convenience samples of college students and research reproducibility. Journal of Business Research, 67(5), 1035–1041.CrossRefGoogle Scholar
Potter, W. J., Cooper, R., & Dupagne, M. (1993). The three paradigms of mass media research in mainstream communication journals. Communication Theory, 3, 317–355.CrossRefGoogle Scholar
Rindfuss, R. R., Choe, M. K., Tsuya, N. O., Bumpass, L. L., & Tamaki, E. (2015). Do low survey response rates bias results? Evidence from Japan. Demographic Research, 32, 797–828.CrossRefGoogle Scholar
Sears, D. O. (1986). College sophomore in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51, 515–530.CrossRefGoogle Scholar
Shapiro, M. A. (2002). Generalizability in communication research. Human Communication Research, 28(4), 491–500.CrossRefGoogle Scholar
Simonson, I., Carmon, Z., Dhar, R., Drolet, A., & Nowlis, S. M. (2001). Consumer research: In search of identity. In Fiske, S. T., Schacter, D. L., & Zahn-Waxler, C. (eds.), Annual Review of Psychology (vol. 52, pp. 249–275). Annual Reviews.Google Scholar
Smith, T. M. F. (1983). On the validity of inferences from non‐random samples. Journal of the Royal Statistical Society: Series A (General), 146(4), 394–403.CrossRefGoogle Scholar
Stephan, F. F. (1948). History of the uses of modern sampling procedures. Journal of the American Statistical Association, 43(241), 12–39.CrossRefGoogle Scholar
Tabachnick, B. G., & Fidell, L. S. (2018). Using Multivariate Statistics. Pearson.Google Scholar
Thomas, R. W. (2011). When student samples make sense in logistics research. Journal of Business Logistics, 32(3), 287–290.CrossRefGoogle Scholar
Traugott, M. (2011). The accuracy of opinion polling and its relation to its future. In Shapiro, R. I. & Jacobs, L. R. (eds.), The Oxford Handbook of American Public Opinion and the Media (pp. 316–331). Oxford University Press.CrossRefGoogle Scholar
Van Teijlingen, E., & Hundley, V. (2010). The importance of pilot studies. Social Research Update, 35(4), 49–59.Google Scholar
Walsch, J. P., Sproull, L. S., & Hesse, B. W. (1992). Self-selected and randomly selected respondents in a computer network. Public Opinion Quarterly, 56, 241–244.CrossRefGoogle Scholar
Wells, W. D. (1993). Discovery-oriented consumer research. Journal of Consumer Research, 19(4), 489–504.CrossRefGoogle Scholar
Wesiner, C., Schmidt, L., & Tam, T. (1995). Assessing bias in community-based prevalence estimates: Towards an unduplicated count of problem drinkers and drug users. Addiction, 90, 391–405.CrossRefGoogle Scholar
Wolfe, B. E. (2013). The value of pilot studies in clinical research: A clinical translation of the research article titled “In search of an adult attachment stress provocation to measure effect on the oxytocin system.” Journal of the American Psychiatric Nurses Association, 19(4), 192–194.CrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

  • Student Samples in Research
  • Edited by John E. Edlund, Rochester Institute of Technology, New York, Austin Lee Nichols, Central European University, Vienna
  • Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
  • Online publication: 12 December 2024
  • Chapter DOI: https://doi.org/10.1017/9781009000796.002
Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

  • Student Samples in Research
  • Edited by John E. Edlund, Rochester Institute of Technology, New York, Austin Lee Nichols, Central European University, Vienna
  • Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
  • Online publication: 12 December 2024
  • Chapter DOI: https://doi.org/10.1017/9781009000796.002
Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

  • Student Samples in Research
  • Edited by John E. Edlund, Rochester Institute of Technology, New York, Austin Lee Nichols, Central European University, Vienna
  • Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
  • Online publication: 12 December 2024
  • Chapter DOI: https://doi.org/10.1017/9781009000796.002
Available formats
×