Hostname: page-component-77c89778f8-5wvtr Total loading time: 0 Render date: 2024-07-21T05:26:35.393Z Has data issue: false hasContentIssue false
  • English
  • Français

Planned missingness: An underused but practical approach to reducing survey and test length

Published online by Cambridge University Press:  09 March 2023

Charlene Zhang Open the ORCID record for Charlene Zhang [Opens in a new window]  and
Paul R. Sackett Open the ORCID record for Paul R. Sackett [Opens in a new window]
Charlene Zhang*
Affiliation:
University of Minnesota, 75 E River Rd #N496, Minneapolis, MN 55455, USA, now at Amazon
Paul R. Sackett
Affiliation:
University of Minnesota, 75 E River Rd #N475, Minneapolis, MN 55455, USA
*
*Corresponding author. Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

I-O psychologists often face the need to reduce the length of a data collection effort due to logistical constraints or data quality concerns. Standard practice in the field has been either to drop some measures from the planned data collection or to use short forms of instruments rather than full measures. Dropping measures is unappealing given the loss of potential information, and short forms often do not exist and have to be developed, which can be a time-consuming and expensive process. We advocate for an alternative approach to reduce the length of a survey or a test, namely to implement a planned missingness (PM) design in which each participant completes a random subset of items. We begin with a short introduction of PM designs, then summarize recent empirical findings that directly compare PM and short form approaches and suggest that they perform equivalently across a large number of conditions. We surveyed a sample of researchers and practitioners to investigate why PM has not been commonly used in I-O work and found that the underusage stems primarily from a lack of knowledge and understanding. Therefore, we provide a simple walkthrough of the implementation of PM designs and analysis of data with PM, as well as point to various resources and statistical software that are equipped for its use. Last, we prescribe a set of four conditions that would characterize a good opportunity to implement a PM design.

Keywords

planned missingnesssurvey designmissing datashort formsurvey length
Type
Practice Forum
Information
Industrial and Organizational Psychology , Volume 16 , Issue 1 , March 2023 , pp. 129 - 142
Copyright
© The Author(s), 2023. Published by Cambridge University Press on behalf of the Society for Industrial and Organizational Psychology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Arbuckle, J. L., & Marcoulides, G. A. (1996). Full information estimation in the presence of incomplete data. In Marcoulides, G. A. & Schumacker, R. E. (Eds.), Advanced structural equation modeling (pp. 243–277). Psychology Press. https://www.jstor.org/stable/2289545 Google Scholar
Barbot, B. (2019). Measuring creativity change and development. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 203. https://doi.org/10.1037/aca0000232 CrossRefGoogle Scholar
Berry, K., Rana, R., Lockwood, A., Fletcher, L., & Pratt, D. (2019). Factors associated with inattentive responding in online survey research. Personality and Individual Differences, 149, 157–159. https://doi.org/10.1016/j.paid.2019.05.043 CrossRefGoogle Scholar
Bowling, N. A., Gibson, A. M., & DeSimone, J. (2021). Questionnaire length and scale validity. Paper presented at the 36th Annual Conference of Society for Industrial and Organizational Psychology, New Orleans, LA (virtual).Google Scholar
Cederman-Haysom, T. (2021). New advanced logic feature: Random assignment. SurveyMonkey. https://www.surveymonkey.com/curiosity/random-assignment/ Google Scholar
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330–351. https://doi.org/10.1037//1082-989X.6.4.330 CrossRefGoogle ScholarPubMed
Conrad-Hiebner, A., Schoemann, A. M., Counts, J. M., & Chang, K. (2015). The development and validation of the Spanish adaptation of the Protective Factors Survey. Children and Youth Services Review, 52, 45–53. https://doi.org/10.1016/j.childyouth.2015.03.006 CrossRefGoogle Scholar
Cortina, J. M., Sheng, Z., Keener, S. K., Keeler, K. R., Grubb, L. K., Schmitt, N., Tonidandel, S., Summerville, K. M., Heggestad, E. D., & Banks, G. C. (2020). From alpha to omega and beyond! A look at the past, present, and (possible) future of psychometric soundness in the Journal of Applied Psychology . Journal of Applied Psychology, 105(12), 1351–1381. https://doi.org/10.1037/apl0000815 Google Scholar
Deutskens, E., de Ruyter, K., Wetzels, M., & Oosterveld, P. (2004). Response rate and response quality of internet-based surveys: An experimental study. Marketing Letters, 15(1), 21–36. https://doi.org/10.1023/B:MARK.0000021968.86465.00 CrossRefGoogle Scholar
Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006). The mini-IPIP scales: Tiny-yet-effective measures of the Big Five factors of personality. Psychological Assessment, 18(2), 192–203. https://doi.org/10.1037/1040-3590.18.2.192 CrossRefGoogle ScholarPubMed
Enders, C. K. (2010). Applied missing data analysis. Guilford Press.Google Scholar
Enders, C. K., & Bandalos, D. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8(3), 430–457.CrossRefGoogle Scholar
Fielding, S., Fayers, P. M., McDonald, A., McPherson, G., Campbell, M. K., & the RECORD Study Group. (2008). Simple imputation methods were inadequate for missing not at random (MNAR) quality of life data. Health and Quality of Life Outcomes, 6(1), 57. https://doi.org/10.1186/1477-7525-6-57 CrossRefGoogle Scholar
Foorman, B. R., Herrera, S., Petscher, Y., Mitchell, A., & Truckenmiller, A. (2015). The structure of oral language and reading and their relation to comprehension in kindergarten through Grade 2. Reading and Writing, 28(5), 655–681. https://doi.org/10.1007/s11145-015-9544-5 CrossRefGoogle ScholarPubMed
Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349–360. https://doi.org/10.1093/poq/nfp031 CrossRefGoogle Scholar
Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychology in Europe, 7(1), 7–28.Google Scholar
Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8(3), 206–213. https://doi.org/10.1007/s11121-007-0070-9 Google ScholarPubMed
Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006). Planned missing data designs in psychological research. Psychological Methods, 11(4), 323. https://doi.org/10.1037/1082-989X.11.4.323 CrossRefGoogle ScholarPubMed
IBM Support. (2018). Difference between FIML (Full information maximum likelihood) and EM (expectation maximization) method in the Missing Values [CT741]. https://www.ibm.com/support/pages/difference-between-fiml-full-information-maximum-likelihood-and-em-expectation-maximization-method-missing-values Google Scholar
Lee, T., & Shi, D. (2021). A comparison of full information maximum likelihood and multiple imputation in structural equation modeling with missing data. Psychological Methods, Advance online publication. https://doi.org/10.1037/met0000381 CrossRefGoogle ScholarPubMed
Little, R. J., & Rubin, D. B. (2019). Statistical analysis with missing data (Vol. 793). Wiley & Sons.Google Scholar
Little, T. D., & Rhemtulla, M. (2013). Planned missing data designs for developmental researchers. Child Development Perspectives, 7(4), 199–204. https://doi.org/10.1111/cdep.12043 CrossRefGoogle Scholar
Little, T. D., Gorrall, B. K., Panko, P., & Curtis, J. D. (2017). Modern practices to improve human development research. Research in Human Development, 14(4), 338–349. https://doi.org/10.1080/15427609.2017.1370967 CrossRefGoogle Scholar
Liu, M., & Wronski, L. (2018). Examining completion rates in web surveys via over 25,000 real-world surveys. Social Science Computer Review, 36(1), 116–124. https://doi.org/10.1177/0894439317695581 CrossRefGoogle Scholar
Lüdtke, O., Robitzsch, A., & Grund, S. (2016). Multiple imputation of missing data in multilevel designs: A comparison of different strategies. Psychological Methods, 22(1), 141–165. https://doi.org/10.1037/met0000096 CrossRefGoogle ScholarPubMed
Marcus-Blank, B., Kuncel, N. R., & Sackett, P. R. (2015). Does rationality predict performance in major life domains? Paper presented at the 30th Annual Conference of Society for Industrial and Organizational Psychology, Philadelphia, PA.Google Scholar
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455.CrossRefGoogle ScholarPubMed
Medeiros, R. (2016). Handling missing data in Stata: Imputation and likelihood-based approaches. 2016 Swiss Stata Users Group Meeting. https://www.stata.com/meeting/switzerland16/slides/medeiros-switzerland16.pdf Google Scholar
Mistler, S. A., & Enders, C. K. (2012). Planned missing data designs for developmental research. In B. Laursen, T. D. Little, & N. A. Card (Eds.), Handbook of Developmental Research Methods, 742–754.Google Scholar
Newman, D. A. (2003). Longitudinal modeling with randomly and systematically missing data: A simulation of ad hoc, maximum likelihood, and multiple imputation techniques. Organizational Research Methods, 6(3), 328–362. https://doi.org/10.1177/1094428103254673 CrossRefGoogle Scholar
Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17(4), 372–411. https://doi.org/10.1177/1094428114548590 Google Scholar
Qualtrics. (2021). Qualtrics Support-Randomizer. https://www.qualtrics.com/support/survey-platform/survey-module/survey-flow/standard-elements/randomizer/ Google Scholar
Revelle, W. (2021). psych: Procedures for psychological, psychometric, and personality research (2.1.3) [R]. Comprehensive R Archive Network (CRAN). https://CRAN.R-project.org/package=psych Google Scholar
Revelle, W., Dworak, E. M., & Condon, D. M. (2020). Exploring the persome: The power of the item in understanding personality structure. Personality and Individual Differences, 109905. https://doi.org/10.1016/j.paid.2020.109905 Google Scholar
Revilla, M., & Ochoa, C. (2017). Ideal and maximum length for a web survey. International Journal of Market Research, 59(5), 557–565. https://doi.org/10.2501/IJMR-2017-039 Google Scholar
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. https://doi.org/10.1093/biomet/63.3.581 CrossRefGoogle Scholar
Sackett, P. R., Berry, C. M., Wiemann, S. A., & Laczo, R. M. (2006). Citizenship and counterproductive behavior: Clarifying relations between the two domains. Human Performance, 19(4), 441–464. https://doi.org/10.1207/s15327043hup1904_7 CrossRefGoogle Scholar
SAS Help Center. (2019). 30.15 The Full Information Maximum Likelihood Method. SAS/STAT User’s Guide. https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_calis_examples36.htm Google Scholar
Schafer, J. L., & Graham, J. W. (2002). Missing data: our view of the state of the art. Psychological Methods, 7(2), 147. https://doi.org/10.1037/1082-989X.7.2.147 CrossRefGoogle ScholarPubMed
Smits, N., & Vorst, H. C. M. (2007). Reducing the length of questionnaires through structurally incomplete designs: An illustration. Learning and Individual Differences, 17(1), 25–34. https://doi.org/10.1016/j.lindif.2006.12.005 Google Scholar
Speidel, M., Drechsler, J., & Jolani, S. (2020). hmi: Hierarchical Multiple Imputation (1.0.0) [R]. https://CRAN.R-project.org/package=hmi Google Scholar
SPSS Statistics. (2021). Impute missing data values (multiple imputation). https://prod.ibmdocs-production-dal-6099123ce774e592a519d7c33db8265e-0000.us-south.containers.appdomain.cloud/docs/en/spss-statistics/23.0.0?topic=imputation-impute-missing-data-values-multiple Google Scholar
The Psychology Series. (2019). Multiple imputation with SPSS syntax (quick and easy). https://www.youtube.com/watch?v=NKQC9YPSnU4 Google Scholar
Typeform. (2021). Build your typeform. https://community.typeform.com/build-your-typeform-7/is-there-a-way-to-randomize-the-order-of-questions-within-a-section-243 Google Scholar
van Buuren, S. (2018). Flexible imputation of missing data (2nd ed.). Chapman & Hall/CRC. https://stefvanbuuren.name/fimd/ Google Scholar
van Buuren, S., & Groothuis-Oudshoorn, K. (2010). MICE: Multivariate imputation by chained equations in R [Article]. Journal of Statistical Software. http://dspace.library.uu.nl/handle/1874/44635 Google Scholar
Verbeke, G., Molenberghs, G., Thijs, H., Lesaffre, E., & Kenward, M. G. (2001). Sensitivity analysis for nonrandom dropout: a local influence approach. Biometrics, 57(1), 7–14. https://doi.org/10.1111/j.0006-341X.2001.00007.x Google ScholarPubMed
Vink, G., & van Buuren, S. (2011). Ad hoc methods and mice. https://www.gerkovink.com/miceVignettes/Ad_hoc_and_mice/Ad_hoc_methods.html Google Scholar
Vink, G., & van Buuren, S. (2014). Pooling multiple imputations when the sample happens to be the population. ArXiv:1409.8542 [Math, Stat]. http://arxiv.org/abs/1409.8542 Google Scholar
Wood, J., Matthews, G. J., Pellowski, J., & Harel, O. (2019). Comparing different planned missingness designs in longitudinal studies. Sankhya B, 81(2), 226–250. https://doi.org/10.1007/s13571-018-0170-5 CrossRefGoogle Scholar
Wu, W., Jia, F., Rhemtulla, M., & Little, T. D. (2016). Search for efficient complete and planned missing data designs for analysis of change. Behavior Research Methods, 48(3), 1047–1061. https://doi.org/10.3758/s13428-015-0629-5 Google ScholarPubMed
Yamada, T. (2020). Organizational and work correlates of sleep. Dissertation, University of Minnesota. http://conservancy.umn.edu/handle/11299/216378 Google Scholar
Yoon, H. R., & Sackett, P. R. (2016). Addressing time constraints in surveys: Planned missingness vs. Short forms. Paper presented at the 31st Annual Conferences of the Society for Industrial and Organizational Psychology, Anaheim, CA.Google Scholar
Yuan, Y. C. (2000). Multiple imputation for missing data: Concepts and new development. Proceedings of the Twenty-Fifth Annual SAS Users Group International Conference, 267.Google Scholar
Zhang, C. (2021). Planned missingness: A sheep in wolf’s clothing. Dissertation, University of Minnesota.Google Scholar
Zhang, C., & Sackett, P. R. (2021). Short form versus planned missingness for reducing survey length: A simulation study. Poster presented at the 36th Annual Conference of the Society for Industrial and Organizational Psychology, New Orleans, LA (virtual).Google Scholar
Zhang, C., & Yu, M. C. (2021). Planned missingness: How to and how much? Organizational Research Methods. https://doi.org/10.1177/10944281211016534 Google Scholar