Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-6bf8c574d5-gr6zb Total loading time: 0 Render date: 2025-02-15T00:22:26.983Z Has data issue: false hasContentIssue false

11 - Statistical Power: How Not to Miss What’s Right in Front of You

from Part II - Important Methodological Considerations

Published online by Cambridge University Press:  12 December 2024

By
Erin M. Buchanan
Edited by
John E. Edlund  and
Austin Lee Nichols
John E. Edlund
Affiliation:
Rochester Institute of Technology, New York
Austin Lee Nichols
Affiliation:
Central European University, Vienna
Get access

Summary

In this chapter, we discuss the definitions of power and how to interpret power in Null Hypothesis Significance Testing. Next, the main determinants of power are outlined, including the sample size, effect size (and variability), α, and the type of statistical test. Each influence on power is demonstrated with example studies on statistics education and data literacy. Different types of power analyses, planning for sample sizes and sensitivity, are illustrated using power tables, popular programs, simulation, and accuracy in parameter estimation. Last, the limitations of power – especially what it does not tell you and what you should not do – are outlined to warn you about the potential misuses of power analyses. Suggestions on appropriate power planning are provided at the end of the chapter.

Type
Chapter
Information
The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
Volume 2: Performing Research
 , pp. 232 - 260
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aerts, M., Molenberghs, G., & Thas, O. (2021). Graduate education in statistics and data science: The why, when, where, who, and what. Annual Review of Statistics and Its Application, 8(1), 25–39. https://doi.org/10.1146/annurev-statistics-040620-032820CrossRefGoogle Scholar
Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017). Sample-size planning for more accurate statistical power: A method adjusting sample effect sizes for publication bias and uncertainty. Psychological Science, 28(11), 1547–1562. https://doi.org/10.1177/0956797617723724CrossRefGoogle Scholar
Anvari, F., & Lakens, D. (2021). Using anchor-based methods to determine the smallest effect size of interest. Journal of Experimental Social Psychology, 96, 104159. https://doi.org/10.1016/j.jesp.2021.104159CrossRefGoogle Scholar
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., et al. (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6–10. https://doi.org/10.1038/s41562-017-0189-zCrossRefGoogle ScholarPubMed
Buchanan, E. M., Valentine, K. D., & Maxwell, N. P. (2019). LAB: Linguistic annotated bibliography a searchable portal for normed database information. Behavior Research Methods, 51(4), 1878–1888. https://doi.org/10.3758/s13428-018-1130-8CrossRefGoogle ScholarPubMed
Champely, S., Ekstrom, C., Dalgaard, P., Gill, J., Weibelzahl, S., Anandkumar, A., et al. (2017). Pwr: Basic functions for power analysis [software]. https://cran.r-project.org/web/packages/pwrGoogle Scholar
Cohen, J. (2013). Statistical Power Analysis for the Behavioral Sciences. Routledge. https://doi.org/10.4324/9780203771587CrossRefGoogle Scholar
Coles, N. A., Hamlin, J. K., Sullivan, L. L., Parker, T. H., & Altschul, D. (2022). Build up big-team science. Nature, 601(7894), 505–507. https://doi.org/10.1038/d41586-022-00150-2CrossRefGoogle ScholarPubMed
Cuccolo, K., Irgens, M. S., Zlokovich, M. S., Grahe, J., & Edlund, J. E. (2021). What crowdsourcing can offer to cross-cultural psychological science. Cross-Cultural Research, 55(1), 3–28. https://doi.org/10.1177/1069397120950628CrossRefGoogle Scholar
DeBruine, L. (2021). Faux: Simulation for Factorial Designs [software]. Zenodo. https://doi.org/10.5281/ZENODO.2669586Google Scholar
Dienes, Z. (2008). Understanding Psychology as a Science. Palgrave Macmillan.Google Scholar
Dienes, Z. (2023). Testing theories with Bayes factors. In Nichols, A. L. & Edlund, J. E. (eds.), Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences (vol. 1, pp. 494–512). Cambridge University Press.CrossRefGoogle Scholar
Dziak, J. J., Dierker, L. C., & Abar, B. (2020). The interpretation of statistical power after the data have been gathered. Current Psychology, 39(3), 870–877. https://doi.org/10.1007/s12144-018-0018-1CrossRefGoogle ScholarPubMed
Erdfelder, E., Faul, F., & Buchner, A. (1996). GPOWER: A general power analysis program. Behavior Research Methods, Instruments, & Computers, 28(1), 1–11. https://doi.org/10.3758/BF03203630CrossRefGoogle Scholar
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146CrossRefGoogle Scholar
Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484CrossRefGoogle ScholarPubMed
Gignac, G. E., & Szodorai, E. T. (2016). Effect size guidelines for individual differences researchers. Personality and Individual Differences, 102, 74–78. https://doi.org/10.1016/j.paid.2016.06.069CrossRefGoogle Scholar
Goldfeld, K., & Wujciak-Jens, J. (2020). Simstudy: Illuminating research methods through data generation. Journal of Open Source Software, 5(54), 2763. https://doi.org/10.21105/joss.02763CrossRefGoogle Scholar
Gümüş, M. M., & Kukul, V. (2023). Developing a digital competence scale for teachers: Validity and reliability study. Education and Information Technologies, 28(3), 2747–2765. https://doi.org/10.1007/s10639-022-11213-2CrossRefGoogle ScholarPubMed
Heckman, M. G., Davis, J. M., & Crowson, C. S. (2022). Post hoc power calculations: An inappropriate method for interpreting the findings of a research study. Journal of Rheumatology, 49(8), 867–870. https://doi.org/10.3899/jrheum.211115CrossRefGoogle ScholarPubMed
JASP Team. (2022). JASP (version 0.16.3)[ software]. https://jasp-stats.orgGoogle Scholar
Kelley, K. (2007). Sample size planning for the coefficient of variation from the accuracy in parameter estimation approach. Behavior Research Methods, 39(4), 755–766. https://doi.org/10.3758/BF03192966CrossRefGoogle ScholarPubMed
Kelley, K. (2022). MBESS: The MBESS r package [software]. https://CRAN.R-project.org/package=MBESSGoogle Scholar
Kelley, K., Darku, F. B., & Chattopadhyay, B. (2018). Accuracy in parameter estimation for a general class of effect sizes: A sequential approach. Psychological Methods, 23(2), 226–243. https://doi.org/10.1037/met0000127CrossRefGoogle ScholarPubMed
Koch, C., & Jones, A. (2016). Big science, team science, and open science for neuroscience. Neuron, 92(3), 612–616. https://doi.org/10.1016/j.neuron.2016.10.019CrossRefGoogle ScholarPubMed
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4. https://doi.org/10.3389/fpsyg.2013.00863CrossRefGoogle Scholar
Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S. E., et al. (2018). Justify your alpha. Nature Human Behaviour, 2(3), 168–171. https://doi.org/10.1038/s41562-018-0311-xCrossRefGoogle Scholar
Legaki, N.-Z., Xi, N., Hamari, J., Karpouzis, K., & Assimakopoulos, V. (2020). The effect of challenge-based gamification on learning: An experiment in the context of statistics education. International Journal of Human-Computer Studies, 144, 102496. https://doi.org/10.1016/j.ijhcs.2020.102496CrossRefGoogle ScholarPubMed
Matli, W., & Ngoepe, M. (2020). Capitalizing on digital literacy skills for capacity development of people who are not in education, employment or training in South Africa. African Journal of Science, Technology, Innovation and Development, 12(2), 129–139. https://doi.org/10.1080/20421338.2019.1624008CrossRefGoogle Scholar
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537–563. https://doi.org/10.1146/annurev.psych.59.103006.093735CrossRefGoogle ScholarPubMed
McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and STAN, 2nd ed. Routledge.CrossRefGoogle Scholar
Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34(2), 103–115. https://doi.org/10.1086/288135CrossRefGoogle Scholar
Miller, J., & Ulrich, R. (2019). The quest for an optimal alpha. PLOS ONE, 14(1), e0208631. https://doi.org/10.1371/journal.pone.0208631CrossRefGoogle ScholarPubMed
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716Google Scholar
Pangrazio, L., Godhe, A.-L., & Ledesma, A. G. L. (2020). What is digital literacy? A comparative review of publications across three language contexts. E-Learning and Digital Media, 17(6), 442–459. https://doi.org/10.1177/2042753020946291CrossRefGoogle Scholar
Pek, J., Pitt, M. A., & Wegener, D. T. (in press). Uncertainty limits the use of power analysis. Journal of Experimental Psychology: General.Google Scholar
Porras-Segovia, A., Molina-Madueño, R. M., Berrouiguet, S., López-Castroman, J., Barrigón, M. L., Pérez-Rodríguez, M. S., et al. (2020). Smartphone-based ecological momentary assessment (EMA) in psychiatric patients and student controls: A real-world feasibility study. Journal of Affective Disorders, 274, 733–741. https://doi.org/10.1016/j.jad.2020.05.067CrossRefGoogle ScholarPubMed
R Core Team. (2022). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. www.gbif.org/tool/81287/r-a-language-and-environment-for-statistical-computingGoogle Scholar
Ralston, K. (2020). “Sociologists shouldn’t have to study statistics”: Epistemology and anxiety of statistics in sociology students. Sociological Research Online, 25(2), 219–235. https://doi.org/10.1177/1360780419888927CrossRefGoogle Scholar
Riesthuis, P., Mangiulli, I., Broers, N., & Otgaar, H. (2022). Expert opinions on the smallest effect size of interest in false memory research. Applied Cognitive Psychology, 36(1), 203–215. https://doi.org/10.1002/acp.3911CrossRefGoogle Scholar
Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International Journal of Human-Computer Studies, 74, 14–31. https://doi.org/10.1016/j.ijhcs.2014.09.006CrossRefGoogle Scholar
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4(1), 1–32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415CrossRefGoogle ScholarPubMed
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., et al. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337356.CrossRefGoogle Scholar
US Bureau of Labor Statistics. (n.d.). Occupational Outlook Handbook. www.bls.gov/ooh/math/data-scientists.htmGoogle Scholar
Vazire, S. (2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13(4), 411–417. https://doi.org/10.1177/1745691617751884CrossRefGoogle ScholarPubMed
Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428–443. https://doi.org/10.1037/1082-989X.10.4.428CrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×