Book contents
- Rethinking Clinical Research
- Rethinking Clinical Research
- Copyright page
- Contents
- Acknowledgments
- Introduction
- Part I Fundamentals
- Part II Rethinking Research
- 4 Asking the Right Questions
- 5 Bias in Randomized Controlled Trials
- 6 Is Bigger Always Better?
- 7 Achieving Transparency
- 8 Understanding and Restraining Study Costs
- 9 Implementation
- 10 Health Outcomes
- 11 Confronting the Crises in Peer Review and Academic Publishing
- 12 Ontologies
- Part III Case Studies
- Index
- References
6 - Is Bigger Always Better?
On Sample Size, Statistical Significance, and Big Data
from Part II - Rethinking Research
Published online by Cambridge University Press: 13 March 2025
Book contents
- Rethinking Clinical Research
- Rethinking Clinical Research
- Copyright page
- Contents
- Acknowledgments
- Introduction
- Part I Fundamentals
- Part II Rethinking Research
- 4 Asking the Right Questions
- 5 Bias in Randomized Controlled Trials
- 6 Is Bigger Always Better?
- 7 Achieving Transparency
- 8 Understanding and Restraining Study Costs
- 9 Implementation
- 10 Health Outcomes
- 11 Confronting the Crises in Peer Review and Academic Publishing
- 12 Ontologies
- Part III Case Studies
- Index
- References
Summary
The criteria for evaluating research studies often include large sample size. It is assumed that studies with large sample sizes are more meaningful than those that include a fewer number of participants. This chapter explores biases associated with the traditional application of null hypothesis testing. Statisticians now challenge the idea that retention of the null hypothesis signifies that a treatment is not effective. A finding associated with an exact probability value of p = 0.049 is not meaningfully different from one in which p = 0.051. Yet the interpretation of these two studies can be dramatically different, including the likelihood of publication. Large studies are not necessarily more accurate or less biased. In fact, biases in sampling strategy are amplified in studies with large sample sizes. These problems are of increasing concern in the era of big data and the analysis of electronic health records. Studies that are overpowered (because of very large sample sizes) are capable of identifying statistically significant differences that are of no clinical importance.
Keywords
- Type
- Chapter
- Information
- Rethinking Clinical ResearchMethodology and Ethics, pp. 117 - 136Publisher: Cambridge University PressPrint publication year: 2025
References
Lauer, MS. From hot hands to declining effects: The risks of small numbers. J Am Coll Cardiol. 2012; 60(1):72–74.CrossRefGoogle Scholar
Schneeweiss, S, Avorn, J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005; 58(4):323–337. doi:10.1016/j.jclinepi.2004.10.012.CrossRefGoogle ScholarPubMed
Budoff, MJ. Current Utility of the Coronary Calcium Score for the Initial Evaluation of Suspected Coronary Artery Disease. BMJ Publishing Group Ltd and British Cardiovascular Society; 2023:659–660.Google ScholarPubMed
Budoff, MJ, Kinninger, A, Gransar, H, et al. When does a calcium score equates to secondary prevention?: Insights from the multinational CONFIRM registry. JACC: Cardiovasc Imaging. 2023; 16(9):1181–1189.Google ScholarPubMed
Eghtedari, B, Kinninger, A, Roy, SK, Budoff, MJ. Coronary artery calcium progression and all-cause mortality. Coron Artery Dis. 2023; 34(4):244–249.CrossRefGoogle ScholarPubMed
Abuzaid, A, Saad, M, Addoumieh, A, et al. Coronary artery calcium score and risk of cardiovascular events without established coronary artery disease: A systemic review and meta-analysis. Coron Artery Dis. 2021; 32(4):317–328.CrossRefGoogle ScholarPubMed
Tripepi, G, Chesnaye, NC, Dekker, FW, Zoccali, C, Jager, KJ. Intention to treat and per protocol analysis in clinical trials. Nephrology. 2020; 25(7):513–517.CrossRefGoogle ScholarPubMed
Lindholt, JS, Søgaard, R, Rasmussen, LM, et al. Five-year outcomes of the Danish Cardiovascular Screening (DANCAVAS) trial. N Engl J Med. 2022; 387(15):1385–1394.CrossRefGoogle ScholarPubMed
Golub, IS, Termeie, OG, Kristo, S, et al. Major global coronary artery calcium guidelines. Cardiovas Imaging. 2023; 16(1):98–117.Google ScholarPubMed
DeYoreo, M, Lansdorp-Vogelaar, I, Knudsen, AB, Kuntz, KM, Zauber, AG, Rutter, CM. Validation of colorectal cancer models on long-term outcomes from a randomized controlled trial. Med Decis Making. 2020; 40(8):1034–1040.CrossRefGoogle ScholarPubMed
Buskermolen, M, Cenin, DR, Helsingen, LM, et al. Colorectal cancer screening with faecal immunochemical testing, sigmoidoscopy or colonoscopy: A microsimulation modelling study. BMJ. 2019; 36(7):l5383.Google Scholar
Zheng, S, Schrijvers, JJ, Greuter, MJ, Kats-Ugurlu, G, Lu, W, de Bock, GH. Effectiveness of Colorectal Cancer (CRC) screening on all-cause and CRC-specific mortality reduction: A systematic review and meta-analysis. Cancers. 2023; 15(7):1948.CrossRefGoogle ScholarPubMed
Amrhein, V, Greenland, S, McShane, B. Scientists rise up against statistical significance. Nature. 2019; 567(7748):305–307.CrossRefGoogle ScholarPubMed
Greenland, S. Valid p-values behave exactly as they should: Some misleading criticisms of p-values and their resolution with s-values. Am Stat. 2019; 73(sup1):106–114.CrossRefGoogle Scholar
Rao, CR. RA Fisher: The founder of modern statistics. Stat Sci. 1992; 7(1):34–48.CrossRefGoogle Scholar
Glassman, JR, Jauregui, A, Milstein, A, Kaplan, RM. Caring for people with depression: Costs among 43 million commercially insured patients with or without comorbid illnesses. Ann Behav Med. 2023; 57(5):380–385.CrossRefGoogle ScholarPubMed
Greenwald, AG, Gonzalez, R, Harris, RJ, Guthrie, D. Effect sizes and p values: What should be reported and what should be replicated? Psychophysiology. 1996; 33(2):175–183.CrossRefGoogle ScholarPubMed
Stampfer, MJ, Colditz, GA, Willett, WC, et al. Postmenopausal estrogen therapy and cardiovascular disease. Ten-year follow-up from the nurses’ health study. N Engl J Med. 1991; 325(11):756–762. doi:10.1056/NEJM199109123251102.CrossRefGoogle ScholarPubMed
Prentice, RL, Langer, R, Stefanick, ML, et al. Combined postmenopausal hormone therapy and cardiovascular disease: Toward resolving the discrepancy between observational studies and the Women’s Health Initiative clinical trial. Research Support, N.I.H., Extramural Research Support, U.S. Gov’t, P.H.S. Am J Epidemiol. 2005; 162(5):404–414. doi:10.1093/aje/kwi223.CrossRefGoogle Scholar
Grodstein, F, Manson, JE, Stampfer, MJ. Postmenopausal hormone use and secondary prevention of coronary events in the nurses’ health study. a prospective, observational study. Ann Intern Med. 2001; 135(1):1–8.CrossRefGoogle ScholarPubMed
Kyzas, PA, Denaxa-Kyza, D, Ioannidis, JP. Almost all articles on cancer prognostic markers report statistically significant results. Eur J Cancer. 2007; 43(17):2559–2579. doi:10.1016/j.ejca.2007.08.030.CrossRefGoogle ScholarPubMed
Vul, E, Pashler, H. Voodoo and circularity errors. NeuroImage. 2012; 62(2):945–948. doi:10.1016/j.neuroimage.2012.01.027.CrossRefGoogle ScholarPubMed
Kriegeskorte, N, Simmons, WK, Bellgowan, PS, Baker, CI. Circular analysis in systems neuroscience: The dangers of double dipping. Nat Neurosci. 2009; 12(5):535–540. doi:10.1038/nn.2303.CrossRefGoogle ScholarPubMed
Ioannidis, JP, Ntzani, EE, Trikalinos, TA, Contopoulos-Ioannidis, DG. Replication validity of genetic association studies. Meta-Analysis Research Support, Non-U.S. Gov’t. Nat Genet. 2001; 29(3):306–309. doi:10.1038/ng749.CrossRefGoogle Scholar
Begley, CG, Ellis, LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012; 483(7391):531–533. doi:10.1038/483531a.CrossRefGoogle ScholarPubMed
Kho, AN, Pacheco, JA, Peissig, PL, et al. Electronic medical records for genetic research: Results of the eMERGE consortium. Sci Transl Med. 2011; 3(79):79re1. doi:10.1126/scitranslmed.3001807.CrossRefGoogle ScholarPubMed
Lagu, T, Krumholz, HM, Dharmarajan, K, et al. Spending more, doing more, or both? An alternative method for quantifying utilization during hospitalizations. J Hosp Med. 2013; 8(7):373–379. doi:10.1002/jhm.2046.CrossRefGoogle ScholarPubMed
Hersh, WR, Weiner, MG, Embi, PJ, et al. Caveats for the use of operational electronic health record data in comparative effectiveness research. Med Care. 2013; 51(8 Suppl 3):S30–S37. doi:10.1097/MLR.0b013e31829b1dbd.CrossRefGoogle ScholarPubMed
Overhage, JM, Overhage, LM. Sensible use of observational clinical data. Stat Methods Med Res. 2013; 22(1):7–13.doi:10.1177/0962280211403598.CrossRefGoogle ScholarPubMed
Schroeder, SA. Shattuck Lecture. We can do better – improving the health of the American people. Research Support, Non-U.S. Gov’t. New Engl J Med. 2007; 357(12):1221–1228. doi:10.1056/NEJMsa073350.CrossRefGoogle Scholar
Council NR. Capturing Social and Behavioral Domains in Electronic Health Records: Phase 1. vol 1. The National Academies Press; 2014.Google Scholar
Estabrooks, PA, Boyle, M, Emmons, KM, et al. Harmonized patient-reported data elements in the electronic health record: Supporting meaningful use by primary care action on health behaviors and key psychosocial factors. J Am Med Inform Assoc. 2012; 19(4):575–582. doi:10.1136/amiajnl-2011-000576.CrossRefGoogle ScholarPubMed
Kilbourne, AM, Neumann, MS, Pincus, HA, Bauer, MS, Stall, R. Implementing evidence-based interventions in health care: Application of the replicating effective programs framework. Implement Sci. 2007; 2:42. doi:10.1186/1748-5908-2-42.CrossRefGoogle ScholarPubMed
Kaplan, RM, Chambers, DA, Glasgow, RE. Big data and large sample size: A cautionary note on the potential for bias. Clin Transl Sci. 2014; 7(4):342–346.CrossRefGoogle ScholarPubMed