Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-lnqnp Total loading time: 0 Render date: 2024-12-25T04:53:45.095Z Has data issue: false hasContentIssue false

Chapter 16 - Measuring Student Character: Modernizing Predictors of Academic Success

from Part III - Assessments Used in Higher Education Admissions

Published online by Cambridge University Press:  09 January 2020

By
Nathan Kuncel ,
Khue Tran  and
Shu Han (Charlene) Zhang
Edited by
María Elena Oliveri  and
Cathy Wendler
María Elena Oliveri
Affiliation:
Educational Testing Service, Princeton, New Jersey
Cathy Wendler
Affiliation:
Educational Testing Service, Princeton, New Jersey
Get access

Summary

This chapter describes noncognitive assessments that measure skills and traits such as leadership, moral character, empathy, social consciousness, and civic responsibility that may be used in admissions decisions. The traditional measures of noncognitive factors, such as letters of recommendation, as well as more modern tools such as situational judgment tests and biodata, are described, and existing research, possible sources of bias, and concerns related to their use reviewed. Recommendations for using practice for the various assessments are discussed.

Keywords

noncognitive assessmentspredictive validityfairnesstraditional measures of noncognitive skillsmodern measures of noncognitive skills
Type
Chapter
Information
Higher Education Admissions Practices
An International Perspective
 , pp. 276 - 302
Publisher: Cambridge University Press
Print publication year: 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aamodt, M. G., Bryan, D. A., & Whitcomb, A. J. (1993). Predicting performance with letters of recommendation. Public Personnel Management, 22, 81–90. https://doi.org/10.1177/009102609302200106.Google Scholar
Al Ansari, A., Al Khalifa, K., Al Azzawi, M., Al Amer, R., Al Sharqi, D., Al-Mansoor, A., & Munshi, F. M. (2015). Cross-cultural challenges for assessing medical professionalism among clerkship physicians in a Middle Eastern country (Bahrain): Feasibility and psychometric properties of multisource feedback. Advances in Medical Education and Practice, 6, 509. https://doi.org/10.2147/AMEP.S86068.Google Scholar
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.Google Scholar
Beach, L. R. (2010). The psychology of narrative thought. Bloomington, IN: Xlibris.Google Scholar
Biernat, M., & Eidelman, S. (2007). Translating subjective language in letters of recommendation: The case of the sexist professor. European Journal of Social Psychology, 37, 1149–1175. https://doi.org/10.1002/ejsp.432.CrossRefGoogle Scholar
Bouton, L. F. (1995). A cross-cultural analysis of the structure and content of letters of reference. Studies in Second Language Acquisition, 17, 211–244. https://doi.org/10.1017/S0272263100014169.Google Scholar
Brown, A., & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18, 36–52. https://doi.org/10.1037/a0030641.Google Scholar
Campion, M. A., Palmer, D. K., & Campion, J. E. (1997). A review of structure in the selection interview. Personnel Psychology, 50, 655–702. https://doi.org/10.1111/j.1744-6570.1997.tb00709.x.Google Scholar
Cirillo, M., Smith, T., & Kiran, J. (2008). The development of biographical inventory for use in the prediction of first-year college success. TCNJ Journal of Student Scholarship, 10, 1–11.Google Scholar
Connelly, B. S., & Ones, D. S. (2010). Another perspective on personality: Meta-analytic integration of observers’ accuracy and predictive validity. Psychological Bulletin, 136, 1092. https://doi.org/10.1037/a0021212.Google Scholar
Costa, P. T. Jr., Terracciano, A., & McCrae, R. R. (2001). Gender differences in personality traits across cultures: Robust and surprising findings. Journal of Personality and Social Psychology, 81, 322–331. https://doi.org/10.1037/0022-3514.81.2.322.Google Scholar
Credé, M., & Kuncel, N. R. (2008). Study habits, study skills, and study attitudes: A meta-analysis of their relationship to academic performance among college students. Perspectives on Psychological Science, 3, 425–453. https://doi.org/10.1111/j.1745-6924.2008.00089.x.Google Scholar
Credé, M., Tynan, M. C., & Harms, P. D. (2017). Much ado about grit: A meta-analytic synthesis of the grit literature. Journal of Personality and Social Psychology, 113, 492–511. https://doi.org/10.1037/pspp0000102.Google Scholar
Educational Testing Service. (1974). SIGI: A computer-based system of interactive guidance and information. Princeton, NJ: Educational Testing Service.Google Scholar
Fajardo, D. M. (1985). Author race, essay quality, and reverse discrimination. Journal of Applied Social Psychology, 15, 255–268. https://doi.org/10.1111/j.1559-1816.1985.tb00900.x.Google Scholar
Finegold, A. (1994). Gender differences in personality: A meta-analysis. Psychological Bulletin, 116, 429–456. https://doi.org/10.1037/0033-2909.116.3.429.CrossRefGoogle Scholar
Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51, 327–358. http://dx.doi.org/10.1037/h0061470.Google Scholar
Foldes, H. J., Duehr, E. E., & Ones, D. S. (2008). Group differences in personality: Meta-analyses comparing five U.S. racial groups. Personnel Psychology, 61, 579–616. https://doi.org/10.1111/j.1744-6570.2008.00123.x.Google Scholar
Friedman, R., Fang, C. H., Hasbun, J., Han, H., Mady, L. J., Eloy, J. A., & Kalyoussef, E. (2017). Use of standardized letters of recommendation for otolaryngology head and neck surgery residency and the impact of gender. Laryngoscope, 127(12), 2738–2745. https://doi.org/10.1002/lary.26619.Google Scholar
Gale, J., Ooms, A., Grant, R., Paget, K., & Marks-Maran, D. (2016). Student nurse selection and predictability of academic success: The multiple mini interview project. Nurse Education Today, 40, 123–127. https://doi.org/10.1016/j.nedt.2016.01.031.CrossRefGoogle ScholarPubMed
Girzadas, D. V. Jr., Harwood, R. C., Dearie, J., & Garrett, S. (1998). A comparison of standardized and narrative letters of recommendation. Academic Emergency Medicine, 5(11), 1101–1104. https://doi.org/10.1111/j.1553-2712.1998.tb02670.x.CrossRefGoogle ScholarPubMed
Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical-statistical controversy. Psychology, Public Policy, and Law, 2, 293–323. https://doi.org/10.1037/1076-8971.2.2.293.Google Scholar
Hawkins, D. A., & Lautz, J. (2005). State of college admission. Alexandria, VA: National Association for College Admission Counseling.Google Scholar
Henderson, M. C., Kelly, C. J., Griffin, E., Hall, T. R., Jerant, A., Peterson, E. M., Rainwater, J. A., Sousa, F. J., Wofsy, D. & Franks, P. (2018). Medical school applicant characteristics associated with performance in multiple mini-interviews: A multi-institutional study. Academic Medicine: Journal of the Association of American Medical Colleges , 93(7), 1029–1034.Google Scholar
Holland, J. L., & Nichols, R. C. (1964). Prediction of academic and extra-curricular achievement in college. Journal of Educational Psychology, 55, 55. https://doi.org/10.1037/h0047977.CrossRefGoogle Scholar
Houser, C., & Lemmons, K. (2017). Implicit bias in letters of recommendation for an undergraduate research internship. Journal of Further and Higher Education, 38, 1–11. https://doi.org/10.1080/0309877X.2017.1301410.Google Scholar
Huber, C. (2017). Faking and the validity of personality tests: Using new faking-resistant measures to study some old questions (Doctoral dissertation). University of Minnesota.Google Scholar
Huffcutt, A. I., & Arthur, W. (1994). Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. Journal of Applied Psychology, 79(2), 184–190. http://dx.doi.org/10.1037/0021-9010.79.2.184.Google Scholar
Huffcutt, A. I., Conway, J. M., Roth, P. L., & Stone, N. J. (2001). Identification and meta-analytic assessment of psychological constructs measured in employment interviews. Journal of Applied Psychology, 86, 897–913. https://doi.org/10.1037/0021-9010.86.5.897.Google Scholar
Huffcutt, A. I., & Roth, P. L. (1998). Racial group differences in employment interview evaluations. Journal of Applied Psychology, 83, 179–189. https://doi.org/10.1037/0021-9010.83.2.179.Google Scholar
Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: The big five revisited. Journal of Applied Psychology, 85, 869–879. https://doi.org/10.1037/0021-9010.85.6.869.Google Scholar
Keiser, H. N., Sackett, P. R., Kuncel, N. R., & Brothen, T. (2016). Why women perform better in college than admission scores would predict: Exploring the roles of conscientiousness and course-taking patterns. Journal of Applied Psychology, 101, 569–581. https://doi.org/10.1037/apl0000069.Google Scholar
Kryger, B. R., & Shikiar, R. (1978). Sexual discrimination in the use of letters of recommendation: A case of reverse discrimination. Journal of Applied Psychology, 63(3), 309–314. http://dx.doi.org/10.1037/0021-9010.63.3.309.CrossRefGoogle Scholar
Kuncel, N. R., Borneman, M., & Kiger, T. (2011). Innovative item response process and Bayesian faking detection methods: More questions than answers. In Ziegler, M., Maccann, C., & Roberts, R. D. (Eds.). New perspectives on faking in personality assessment. Oxford: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195387476.003.0036.Google Scholar
Kuncel, N. R., Brenneman, M., Petway, K., & Liu, J. (2018). Validation of the character skills snapshot (CSS). Skillman, NJ: Enrollment Management Association.Google Scholar
Kuncel, N. R., & Hezlett, S. A. (2010). Fact and fiction in standardized admissions testing. Current Directions in Psychological Science, 19, 339–345. https://doi.org/10.1177/0963721410389459.Google Scholar
Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology, 98, 1060–1072. https://doi.org/10.1037/a0034156.CrossRefGoogle ScholarPubMed
Kuncel, N. R., Kochevar, R. J., & Ones, D. S. (2014). A meta-analysis of letters of recommendation in college and graduate admissions. Reasons for hope. International Journal of Selection and Assessment, 22, 101–107. https://doi.org/10.1111/ijsa.12060.Google Scholar
Kuncel, N. R., & Tellegen, A. (2009). The measurement of the social desirability of items: A conceptual and empirical reexamination. Personnel Psychology, 62, 201–228. https://doi.org/10.1111/j.1744-6570.2009.01136.x.Google Scholar
Kuncel, N. R., & Sackett, P. R. (2014). Resolving the assessment center construct validity problem (as we know it). Journal of Applied Psychology, 99(1), 38. http://dx.doi.org/10.1037/a0034147.Google Scholar
Kuncel, N. R., & Sackett, P. R. (2018, March 10). The gatekeeper tests. The Wall Street Journal, pp. C1–C2.Google Scholar
Lerner, J. S., & Tetlock, P. E. (2003). Bridging individual, interpersonal, and institutional approaches to judgment and decision making: The impact of accountability on cognitive bias. In Schneider, S. L. & Shanteau, J. (Eds.). Emerging Perspectives on Judgment and Decision Research (pp. 431–457). https://doi.org/10.1017/CBO9780511609978.015.Google Scholar
Lievens, F., Buyse, T., & Sackett, P. R. (2005). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90, 442–452. http://dx.doi.org/10.1037/0021-9010.90.3.442.CrossRefGoogle ScholarPubMed
Lievens, F., Buyse, T., Sackett, P. R., & Connelly, B. S. (2012). The effects of coaching on situational judgment tests in high-stakes selection. International Journal of Selection and Assessment, 20, 272–282. https://doi.org/10.1111/j.1468-2389.2012.00599.x.Google Scholar
Lievens, F., & Sackett, P. R. (2006). Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology, 91(5), 1181. http://dx.doi.org/10.1037/0021-9010.91.5.1181.Google Scholar
Lievens, F., Sackett, P. R., & Buyse, T. (2009). The effects of response instructions on situational judgment test performance and validity in a high-stakes context. Journal of Applied Psychology, 94, 1095–1101. http://dx.doi.org/10.1037/a0014628.CrossRefGoogle Scholar
Lumb, A. B., Homer, M., & Miller, A. (2010). Equity in interviews: Do personal characteristics impact on admission interview scores? Medical Education, 44, 1077–1083. https://doi.org/10.1111/j.1365-2923.2010.03771.x.Google Scholar
McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Predicting job performance using situational judgement tests: A clarification of the literature. Journal of Applied Psychology, 86, 730–740.Google Scholar
Mackenzie, R. S. (1967). Predictive validity of a biographical inventory in higher education. Journal of Applied Psychology, 51, 544–546. https://doi.org/10.1037/h0025108.Google Scholar
Max, B. A., Gelfand, B., Brooks, M. R., Beckerly, R., & Segal, S. (2010). Have personal statements become impersonal? An evaluation of personal statements in anesthesiology residency applications. Journal of Clinical Anesthesia, 22, 346–351. https://doi.org/10.1016/j.jclinane.2009.10.007.Google Scholar
Morgan, W. B., Elder, K. B., & King, E. B. (2013). The emergence and reduction of bias in letters of recommendation. Journal of Applied Social Psychology, 43(11), 2297–2306. https://doi.org/10.1111/jasp.12179.Google Scholar
Murphy, S. R., Klieger, D. M., Borneman, M., & Kuncel, N. R. (2009). The predictive power of personal statements in admissions: A meta-analysis and cautionary tale. College and University, 84(4), 83–88.Google Scholar
Nicklin, J. M., & Roch, S. G. (2008). Biases influencing recommendation letter contents: physical attractiveness and gender. Journal of Applied Social Psychology, 38, 3053–3074. http://dx.doi.org/10.1111/j.1559-1816.2008.00425.x.Google Scholar
Nye, C. D., Su, R., Rounds, J., & Drasgow, F. (2012). Vocational interests and performance: A quantitative summary of over 60 years of research. Perspectives on Psychological Science, 7, 384–403. https://doi.org/10.1177/1745691612449021.Google Scholar
Oliveri, M. E., McCaffrey, D., Ezzo, C., & Holtzman, S. (2017). A multilevel factor analysis of third-party evaluations of noncognitive constructs used in admissions decision making. Applied Measurement in Education, 30, 297–313. https://doi.org/10.1080/08957347.2017.1353989.Google Scholar
Osman, N. Y., Schonhardt‐Bailey, C., Walling, J. L., Katz, J. T., & Alexander, E. K. (2015). Textual analysis of internal medicine residency personal statements: Themes and gender differences. Medical Education, 49, 93–102. https://doi.org/10.1111/medu.12487.CrossRefGoogle ScholarPubMed
Oswald, F. L., Schmitt, N., Kim, B. H., Ramsey, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187–207. http://dx.doi.org/10.1037/0021-9010.89.2.187.Google Scholar
The papers of Benjamin Franklin. n.d. Retrieved from http://franklinpapers.org/.Google Scholar
Pau, A., Jeevaratnam, K., Chen, Y. S., Fall, A. A., Khoo, C., & Nadarajah, V. D. (2013). The multiple mini-interview (MMI) for student selection in health professions training: A systematic review. Medical Teacher, 35, 1027–1041. https://doi.org/10.3109/0142159X.2013.829912.Google Scholar
Pavlov, G., Maydeu-Olivares, A., & Fairchild, A. J. (2018). Effects of applicant faking on forced-choice and Likert scores. Organizational Research Methods, 1–30. https://doi.org/10.1177/1094428117753683.Google Scholar
Peeters, H., & Lievens, P. (2005). Situational judgment tests and their predictiveness of college students’ success: The influence of faking. Educational and Psychological Measurement, 65, 70–89. https://doi.org/10.1177/0013164404268672.Google Scholar
Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135, 322–338. https://doi.org/10.1037/a0014996.Google Scholar
Precht, K. (1998). A cross-cultural comparison of letters of recommendation. English for Specific Purposes, 17, 241–265. https://doi.org/10.1016/S0889–4906(97)00012-4.Google Scholar
Roberts, B. W., Kuncel, N. R., Shiner, R., Caspi, A., & Goldberg, L. R. (2007). The comparative predictive validity of personality traits, SES, and cognitive ability. Perspectives on Psychological Science, 2, 331–345. https://doi.org/10.1111/j.1745-6916.2007.00047.x.Google Scholar
Salgado, J. F. (1997). The five factor model of personality and job performance in the European community. Journal of Applied Psychology, 82, 30–43. https://doi.org/10.1037/0021-9010.82.1.30.Google Scholar
Shahani, C., & Dipboye, R. L. (1993). Attractiveness bias in the interview: Exploring the boundaries of an effect. Basic and Applied Social Psychology, 14, 317–328. https://doi.org/10.1207/s15324834basp1403_5.Google Scholar
Sisk, V. F., Burgoyne, A. P., Sun, J., Butler, J. L., & Macnamara, B. N. (2018). To what extent and under what circumstance are growth mind-sets important to academic achievement? Two meta-analyses. Psychological Science, 29, 549–571. https://doi.org/10.1177/0956797617739704.CrossRefGoogle ScholarPubMed
Stark, S., Chernyshenko, O. S., & Drasgow, F. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions. Applied Psychological Measurement, 29, 184–203. https://doi.org/10.1177/0146621604273988.Google Scholar
Terregino, C. A., McConnell, M., & Reiter, H. I. (2015). The effect of differential weighting of academics, experiences, and competencies measured by multiple mini interview (MMI) on race and ethnicity of cohorts accepted to one medical school. Academic Medicine: Journal of the Association of American Medical Colleges, 90, 1651–1657. https://doi.org/10.1097/ACM.0000000000000960.Google Scholar
Watson, C. (1987). Sex-linked differences in letters of recommendation. Women and Language, 10(2), 26. Retrieved from http://erepo.usiu.ac.ke/handle/11732/1283?show=full.Google Scholar
Whetzel, D. L., McDaniel, M. A., & Nguyen, N. T. (2008). Subgroup differences in situational judgment test performance: A meta-analysis. Human Performance, 21(3), 291-309.Google Scholar
Wright, S. R., & Bradley, P. M. (2010). Has the UK clinical aptitude test improved medical student selection? Medical Education, 44, 1069–1076. https://doi.org/10.1111/j.1365-2923.2010.03792.x.Google Scholar
Zhang, C. S. H., & Kuncel, N. R. (2018). Moving beyond the brag sheet: Predicting student outcomes with biodata measures. Unpublished manuscript. University of Minnesota.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×