Book contents
- Handbook of Research Methods in Social and Personality Psychology
- Cambridge Handbooks in Psychology
- Handbook of Research Methods in Social and Personality Psychology
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Introduction
- 1 The Romance of Research Methods
- Part I Before You Dive In
- Part II Basic Design Considerations to Know, No Matter What Your Research Is About
- 6 Research Design and Issues of Validity
- 7 Experimental Design
- 8 Quasi-Experimental Designs
- 9 Field Research Methods
- Part III Deep Dives on Methods and Tools for Testing Your Question of Interest
- Part IV Understanding What Your Data Are Telling You About Psychological Processes
- Index
- References
6 - Research Design and Issues of Validity
from Part II - Basic Design Considerations to Know, No Matter What Your Research Is About
Published online by Cambridge University Press: 12 December 2024
Book contents
- Handbook of Research Methods in Social and Personality Psychology
- Cambridge Handbooks in Psychology
- Handbook of Research Methods in Social and Personality Psychology
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Introduction
- 1 The Romance of Research Methods
- Part I Before You Dive In
- Part II Basic Design Considerations to Know, No Matter What Your Research Is About
- 6 Research Design and Issues of Validity
- 7 Experimental Design
- 8 Quasi-Experimental Designs
- 9 Field Research Methods
- Part III Deep Dives on Methods and Tools for Testing Your Question of Interest
- Part IV Understanding What Your Data Are Telling You About Psychological Processes
- Index
- References
Summary
The validity of conclusions drawn from specific research studies must be evaluated in light of the purposes for which the research was undertaken. We distinguish four general types of research: description and point estimation, correlation and prediction, causal inference, and explanation. For causal and explanatory research, internal validity is critical – the extent to which a causal relationship can be inferred from the results of variation in the independent and dependent variables of an experiment. Random assignment is discussed as the key to avoiding threats to internal validity. Internal validity is distinguished from construct validity (the relationship between a theoretical construct and the methods used to operationalize that concept) and external validity (the extent to which the results of a research study can be generalized to other contexts). Construct validity is discussed in terms of multiple operations and discriminant and convergent validity assessment. External validity is discussed in terms of replicability, robustness, and relevance of specific research findings.
Keywords
- Type
- Chapter
- Information
- Publisher: Cambridge University PressPrint publication year: 2024
References
Anderson, C. A., Allen, J. J., Plante, C., Quigley-McBride, A., Lovett, A., and Rokkum, J. N. (2019). The MTurkification of social and personality psychology. Personality and Social Psychology Bulletin, 45, 842–850.CrossRefGoogle ScholarPubMed
Aronson, E., and Mills, J. (1959). The effect of severity of initiation on liking for a group. Journal of Abnormal and Social Psychology, 59, 177–181.CrossRefGoogle Scholar
Aronson, E., Wilson, T., and Brewer, M. B. (1998). Experimentation in social psychology. In Gilbert, D., Fiske, S., and Lindzey, G. (eds) The Handbook of Social Psychology, 4th ed., vol. 1. Boston: McGraw-Hill.Google Scholar
Asch, S. E. (1956). Studies of independence and conformity: A minority of one against a unanimous majority. Psychological Monographs, 70(9), whole no. 416.CrossRefGoogle Scholar
Baron, R. M., and Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182.CrossRefGoogle ScholarPubMed
Berkowitz, L., and Donnerstein, E. (1982). External validity is more than skin deep: Some answers to criticisms of laboratory experiments. American Psychologist, 37, 245–257.CrossRefGoogle Scholar
Brewer, M. B. (1997). The social psychology of intergroup relations: Can research inform practice? Journal of Social Issues, 53(1), 197–211.CrossRefGoogle Scholar
Brunswik, E. (1956). Perception and the Representative Design of Psychological Experiments, 2nd ed. University of California Press.CrossRefGoogle Scholar
Bullock, J. G., and Green, D. P. (2021). The failings of conventional mediation analysis and a design-based alternative. Advances in Methods and Practices in Psychological Science, 4(4), 1–18.CrossRefGoogle Scholar
Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297–312.CrossRefGoogle Scholar
Campbell, D. T., and Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin, 56, 81–105.CrossRefGoogle ScholarPubMed
Campbell, D. T., and Stanley, J. C. (1963). Experimental and quasi-experimental designs for research on teaching. In Gage, N. (ed.) Handbook of Research on Teaching. Rand-McNally.Google Scholar
Campbell, D. T., and Stanley, J. C. (1966). Experimental and Quasi-experimental Designs for Research. Rand-McNally.Google Scholar
Chester, D. S., and Lasko, E. N. (2021). Construct validation of experimental manipulations in social psychology: Current practices and recommendations for the future. Perspectives on Psychological Science, 16, 377–395.CrossRefGoogle ScholarPubMed
Chmielewski, M., and Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11, 464–473.CrossRefGoogle Scholar
Cook, T. D., and Campbell, D. T. (1979). Quasi-experimentation: Design and Analysis Issues for Field Settings. Rand-McNally.Google Scholar
Cook, T. D., and Shadish, W. R. (1994). Social experiments: Some developments over the past fifteen years. Annual Review of Psychology, 45, 545–580.CrossRefGoogle Scholar
Crandall, C. S., and Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93–99.CrossRefGoogle Scholar
Cronbach, L. J. (1982). Designing Evaluations of Educational and Social Programs. Jossey-Bass.Google Scholar
Ejelov, E., and Luke, T. (2020). “Rarely safe to assume”: Evaluating the use and interpretation of manipulations checks in experimental social psychology. Journal of Experimental Social Psychology, 87, Article 103037.CrossRefGoogle Scholar
Fiedler, K., Schott, M., and Meiser, T. (2011). What mediation analysis can (not) do. Journal of Experimental Social Psychology, 47, 1231–1236.CrossRefGoogle Scholar
Flake, J. K., and Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3, 456–465.CrossRefGoogle Scholar
Flake, J. K., Pek, J., and Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8, 370–378.CrossRefGoogle Scholar
Gerard, H. B., and Mathewson, G. C. (1966). The effects of severity of initiation on liking for a group: A replication. Journal of Experimental Social Psychology, 2, 278–287.CrossRefGoogle Scholar
Greenwald, A. G. (1975). On the inconclusivenes of “crucial” cognitive tests of dissonance versus self-perception theories. Journal of Experimental Social Psychology, 11, 490–499.CrossRefGoogle Scholar
Greenwood, J. D. (1982). On the relation between laboratory experiments and social behaviour: Causal explanation and generalization. Journal for the Theory of Social Behaviour, 12, 225–250.CrossRefGoogle Scholar
Gruijters, S. L. K. (2022). Making inferential leaps: Manipulation checks and the roads towards strong inference. Journal of Experimental Social Psychology, 98, Article 104251.CrossRefGoogle Scholar
Henrich, J., Heine, S. J., and Norenzayan, A. (2010a). Beyond WEIRD: Towards a broad-based behavioral science. Behavioral and Brain Sciences, 33(2–3), 111–135.CrossRefGoogle Scholar
Henrich, J., Heine, S. J., and Norenzayan, A. (2010b). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.CrossRefGoogle ScholarPubMed
Imai, K., and Jiang, Z. (2020). Identification and sensitivity analysis of contagion effects in randomized placebo‐controlled trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 183(4), 1637–1657.CrossRefGoogle Scholar
Kenny, D. A. (1995). The multitrait–multimethod matrix: Design, analysis, and conceptual issues. In Shrout, P. E. and Fiske, S. T. (eds.) Personality Research, Methods, and Theory: A Festschrift Honoring Donald W. Fiske. Erlbaum.Google Scholar
Kihlstrom, J. F. (2021). Ecological validity and “ecological validity.” Perspectives on Psychological Science, 16, 466–471.CrossRefGoogle ScholarPubMed
Klein, R. A., et al. (2018). Many labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1, 443–490.CrossRefGoogle Scholar
Liu, X., and Wang, L. (2021). The impact of measurement error and omitting confounders on statistical inference of mediation effects and tools for sensitivity analysis. Psychological Methods, 26, 327–342.CrossRefGoogle ScholarPubMed
Markus, H. R., and Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98, 224–253.CrossRefGoogle Scholar
Marsh, H. W., Hau, K. T., Wen, Z., Nagengast, B., and Morin, A. J. S. (2011). Moderation. In Little, T. D. (ed.) Oxford Handbook of Quantitative Methods. Oxford University Press.Google Scholar
Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378.CrossRefGoogle ScholarPubMed
Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38, 379–387.CrossRefGoogle Scholar
Nadler, J., Baumgartner, S., and Washington, M. (2021). MTurk for working samples: Evaluation of data quality 2014–2020. North American Journal of Psychology, 23, 741–752.Google Scholar
Nosek, B. A., et al. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73, 719–748.CrossRefGoogle ScholarPubMed
Orne, M. (1962). On the social psychology of the psychological experiment. American Psychologist, 17, 776–783.CrossRefGoogle Scholar
Orne, M., and Holland, C. (1968). Some conditions of obedience and disobedience to authority. On the ecological validity of laboratory deceptions. International Journal of Psychiatry, 6, 282–293.Google ScholarPubMed
Petty, R. E., and Cacioppo, J. T. (1996). Addressing disturbing and disturbed consumer behavior: Is it necessary to change the way we conduct behavioral science? Journal of Marketing Research, 33, 1–8.CrossRefGoogle Scholar
Qin, X., and Yang, F. (2022). Simulation-based sensitivity analysis for causal mediation studies. PsychologicalMethods, https://doi.org/10.1037/met0000340.CrossRefGoogle Scholar
Rakover, S. S. (1981). Social psychology theory and falsification. Personality and Social Psychology Bulletin, 7, 123–130.CrossRefGoogle Scholar
Reichardt, C. S., and Coleman, S. C. (1995). The criteria for convergent and discriminant validity in a multitrait–multimethod matrix. Multivariate Behavioral Research, 30, 513–538.CrossRefGoogle Scholar
Rohrer, J. M. (2018). Thinking clearly about correlations and causation: Graphical causal models for observational data. Advances in Methods and Practices in Psychological Science, 1, 27–42.CrossRefGoogle Scholar
Rosenberg, M. J. (1969). The conditions and consequences of evaluation apprehension. In Rosenthal, R. and Rosnow, R. (eds) Artifact in Behavioral Research. Academic Press.Google Scholar
Rosenthal, R. (1966). Experimenter Effects in Behavioral Research. Appleton-Century-Crofts.Google Scholar
Sampson, E. E. (1977). Psychology and the American ideal. Journal of Personality and Social Psychology, 35, 767–782.CrossRefGoogle Scholar
Sears, D. O. (1986). College sophomores in the laboratory: Influence of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51, 515–530.CrossRefGoogle Scholar
Shadish, W. R., Cook, T. D., and Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.Google Scholar
Sherif, M. (1935). A study of some social factors in perception. Archives of Psychology, 27(187), 1–60.Google Scholar
Sivacek, J., and Crano, W. D. (1982). Vested interest as a moderator of attitude behavior consistency. Journal of Personality and Social Psychology, 43, 210–221.CrossRefGoogle Scholar
Toich, M. J., Schutt, E., and Fisher, D. M. (2022). Do you get what you pay for? Preventing insufficient effort responding in MTurk and student samples. Applied Psychology: An International Review, 71, 640–661.CrossRefGoogle Scholar
Vazire, S., Schiavone, S. R., and Bottesini, J. G. (2022). Credibility beyond replicability: Improving the four validities in psychological science. Current Directions in Psychological Science, 31, 162–168.CrossRefGoogle Scholar
Zanna, M., and Cooper, J. (1974). Dissonance and the pill: An attribution approach to studying the arousal properties of dissonance. Journal of Personality and Social Psychology, 29, 703–709.CrossRefGoogle ScholarPubMed
Zhou, H., and Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 111, 493–504.CrossRefGoogle ScholarPubMed
Zillman, D. (1983). Transfer of excitation in emotional behavior. In Cacioppo, J. T. and Petty, R. E. (eds) Social Psychophysiology: A Sourcebook. Guilford Press.Google Scholar