Hostname: page-component-cd9895bd7-mkpzs Total loading time: 0 Render date: 2024-12-24T16:35:17.608Z Has data issue: false hasContentIssue false
  • English
  • Français

Why Assessment Centers Do Not Work the Way They Are Supposed To

Published online by Cambridge University Press:  07 January 2015

Charles E. Lance
Charles E. Lance*
Affiliation:
The University of Georgia
*
E-mail: [email protected], Address: Department of Psychology, The University of Georgia, Athens, Georgia 30602-3013
Get access Rights & Permissions [Opens in a new window]

Abstract

Assessment centers (ACs) are often designed with the intent of measuring a number of dimensions as they are assessed in various exercises, but after 25 years of research, it is now clear that AC ratings that are completed at the end of each exercise (commonly known as postexercise dimension ratings) substantially reflect the effects of the exercises in which they were completed and not the dimensions they were designed to reflect. This is the crux of the long-standing “construct validity problem” for AC ratings. I review the existing research on AC construct validity and conclude that (a) contrary to previous notions, AC candidate behavior is inherently cross-situationally (i.e., cross-exercise) specific, not cross-situationally consistent as was once thought, (b) assessors rather accurately assess candidate behavior, and (c) these facts should be recognized in the redesign of ACs toward task- or role-based ACs and away from traditional dimension-based ACs.

Type
Focal Article
Information
Industrial and Organizational Psychology , Volume 1 , Issue 1 , March 2008 , pp. 84 - 97
Copyright
Copyright © Society for Industrial and Organizational Psychology 2008 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

Department of Psychology, The University of Georgia

This work was supported in part by National Institute on Aging Grant 5R01AG15321-10 and by Gail M. Williamson, principal investigator. Thanks go to Filip Lievens, Duncan Jackson, Brian Hoffman, and Lillian Eby for comments on an earlier version of this paper

References

Arthur, W. Jr., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125–154.Google Scholar
Arthur, W. Jr., Woehr, D. J., & Maldegan, R. (2000). Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical re-examination of the assessment center construct-related validity paradox. Journal of Management, 26, 813–835.Google Scholar
Beaubein, J. M., Baker, D. P., & Salvaggio, A. N. (2004). Improving the construct validity of line operational simulation ratings: Lessons learned from the assessment center. International Journal of Aviation Psychology, 14, 1–17.Google Scholar
Bowler, M. C., & Woehr, D. J. (2006). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology, 91, 1114–1124.Google Scholar
Bray, D. W., Campbell, R. J., & Grant, D. L. (1974). Formative years in business: A long-term AT&T study of managerial lives. New York: Wiley.Google Scholar
Bray, D. W., & Grant, D. L. (1966). The assessment center in the measurement of potential for business management. Psychological Monographs, 80 (17, Whole No. 625), 1–27.Google Scholar
Buffett, J. (1999). I don’t know and I don’t care. On beach house on the moon [CD]. Key West, FL: Margaritaville/Island.Google Scholar
Byham, W. C. (1977). Application of the assessment center method. In Moses, J. L. & Byham, W. C. (Eds.), Applying the assessment center method (pp. 31–43). New York: Pergamon.Google Scholar
Campbell, C. H., Ford, P., Rumsey, M. G., Pulakos, E. D., Borman, W. C., Felker, D. B., et al. (1990). Development of multiple job performance measures in a representative sample of jobs. Personnel Psychology, 43, 277–300.10.1111/j.1744-6570.1990.tb01559.xGoogle Scholar
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.Google Scholar
Donahue, L. M., Truxillo, D. M., Cornwell, J. M., & Gerrity, M. J. (1997). Assessment center construct validity and behavioral checklists: Some additional findings. Journal of Social Behavior and Personality, 12(5), 85–108.Google Scholar
Dunnette, M. D. (1971). The assessment of managerial talent. In McReynolds, P. (Ed.), Advances in psychological assessment (Vol. 2, pp. 79–108). Palo Alto, CA: Science and Behavior Books.Google Scholar
Epstein, S., & O’Brien, E. J. (1985). The person-situation debate in historical and current perspective. Psychological Bulletin, 98, 513–537.Google Scholar
Funder, D. C. (1987). Errors and mistakes: Evaluating the accuracy of social judgment. Psychological Bulletin, 101, 75–90.Google Scholar
Gaugler, B. B., Rosenthal, D. B., Thornton, G. C. III, & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72, 493–511.Google Scholar
Gaugler, B. B., & Thornton, G. C. III. (1989). Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology, 74, 611–618.Google Scholar
Haaland, S., & Christiansen, N. D. (2002). Implications of trait-activation theory for evaluating the construct validity of assessment center ratings. Personnel Psychology, 55, 137–163.Google Scholar
Hardison, C. M., & Sackett, P. R. (2004, April). Assessment center criterion-related validity: A meta-analytic update. Paper presented at the 19th Annual Conference of the Society for Industrial and Organizational Psychology, Chicago, IL.Google Scholar
Harris, M. M., Becker, A. S., & Smith, D. E. (1993). Does the assessment center scoring method affect the cross-situational consistency of ratings? Journal of Applied Psychology, 78, 675–678.Google Scholar
Hennessy, J., Mabey, B., & Warr, P. (1998). Assessment center observation procedures: An experimental comparison of traditional, checklist and coding methods. International Journal of Selection and Assessment, 6, 222–231.Google Scholar
Howard, A. (1997). A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality, 12(5), 13–52.Google Scholar
International Task Force on Assessment Center Guidelines. (2000). Guidelines and ethical considerations for assessment center operations. Public Personnel Management, 28, 315–331.Google Scholar
Jackson, D. J. R., Barney, A. R., Stillman, J. A. & Kirkley, B. (2007). When traits are behaviors: The relationship between behavioral responses and trait-based overall assessment center ratings. Human Performance, 20, 415–432.Google Scholar
Jackson, D. J. R., Stillman, J. A., & Atkins, S. G. (2005). Rating tasks versus dimensions in assessment centers: A psychometric comparison. Human Performance, 18, 213–241.Google Scholar
Joyce, L. W., Thayer, P. W., & Pond, S. B. III. (1994). Managerial functions: An alternative to traditional assessment center dimensions? Personnel Psychology, 47, 109–121.Google Scholar
Kleinmann, M. (1993). Are rating dimensions in assessment centers transparent for participants? Consequences for criterion and construct validity. Journal of Applied Psychology, 78, 988–993.Google Scholar
Kleinmann, M., & Köller, O. (1997). Construct validity of assessment centers: Appropriate use of confirmatory factor analysis and suitable construction principles. Journal of Social Behavior and Personality, 12(5), 65–84.Google Scholar
Knapp, D. (2006). The U.S. joint-service job performance measurement project. In Bennett, W. Jr., Lance, C. E., & Woehr, D. J. (Eds.), Performance measurement: Current perspectives and future challenges (pp. 113–140). Mahwah, NJ: Erlbaum.Google Scholar
Kolk, N. J., Born, M. P., & Van Der Flier, H. (2002). Impact of common rater variance on construct validity of assessment center dimension judgments. Human Performance, 15, 325–337.Google Scholar
Kolk, N. J., Born, M. P., & Van Der Flier, H. (2003). The transparent assessment center: The effects of revealing dimensions to candidates. Applied Psychology: An International Review, 52, 648–668.Google Scholar
Lance, C. E., Foster, M. R., Gentry, W. A., & Thoresen, J. D. (2004). Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology, 89, 22–35.Google Scholar
Lance, C. E., Foster, M. R., Nemeth, Y. M., Gentry, W. A., & Drollinger, S. (2007). Extending the nomological network of assessment center construct validity: Prediction of cross-situationally consistent and specific aspects of assessment center performance. Human Performance, 20, 345–362.Google Scholar
Lance, C. E., Lambert, T. A., Gewin, A. G., Lievens, F., & Conway, J. M. (2004). Revised estimates of dimension and exercise variance components in assessment center post-exercise dimension ratings. Journal of Applied Psychology, 89, 377–385.Google Scholar
Lance, C. E., Newbolt, W. H., Gatewood, R. D., Foster, M. R., French, N., & Smith, D. E. (2000). Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13, 323–353.Google Scholar
Lance, C. E., Noble, C. L., & Scullen, S. E. (2002). A critique of the correlated trait–correlated method (CTCM) and correlated uniqueness (CU) models for multitrait-multimethod (MTMM) data. Psychological Methods, 7, 228–244.Google Scholar
Lievens, F. (1998). Factors which improve the construct validity of assessment centers: A review. International Journal of Selection and Assessment, 6, 141–152.Google Scholar
Lievens, F. (2001a). Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86, 255–264.Google Scholar
Lievens, F. (2001b). Assessors and use of assessment centre dimensions: A fresh look at a troubling issue. Journal of Organizational Behavior, 22, 203–221.Google Scholar
Lievens, F. (2002). Trying to understand the different pieces of the construct validity puzzle of assessment centers: An examination of assessor and assessee effects. Journal of Applied Psychology, 87, 675–686.Google Scholar
Lievens, F., Chasteen, C. S., Day, E. A., & Christiansen, N. D. (2006). Large-scale investigation of the role of trait activation theory for understanding assessment center convergent and discriminant validity. Journal of Applied Psychology, 91, 247–258.Google Scholar
Lievens, F., & Conway, J. M. (2001). Dimensions and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 1202–1222.Google Scholar
Lievens, F., & Klimoski, R. J. (2001). Understanding the assessment centre process: Where are we now? International Review of Industrial and Organizational Psychology, 16, 246–286.Google Scholar
Lovler, B., Rose, M., & Wesley, S. (2002, April). Finding assessment center construct validity: Try behaviors instead of dimensions. Paper presented at the 17th Annual Conference of the Society for Industrial and Organizational Psychology, Toronto, Ontario, Canada.Google Scholar
Lowry, P. E. (1995). The assessment center process: Assessing leadership in the public sector. Public Personnel Management, 24, 443–450.Google Scholar
Lowry, P. E. (1997). The assessment center process: New directions. Journal of Social Behavior and Personality, 12(5), 53–62.Google Scholar
MacKinnon, D. W. (1977). From selecting spies to selecting managers—The OSS assessment program. In Moses, J. L. & Byham, W. C. (Eds.), Applying the assessment center method (pp. 13–30). New York: Pergamon.Google Scholar
McArthur, L. Z., & Baron, R. M. (1983). Toward an ecological theory of social perception. Psychological Bulletin, 90, 215–238.Google Scholar
Moses, J. L. (1977). The assessment center method. In Moses, J. L. & Byham, W. C. (Eds.), Applying the assessment center method (pp. 3–11). New York: Pergamon.Google Scholar
Moses, J. L. & Byham, W. C. (Eds.). (1977). Applying the assessment center method. New York: Pergamon.Google Scholar
Murray, H. A. (1938). Explorations in personality. New York: Oxford University Press.Google Scholar
Neidig, R. D., & Neidig, P. J. (1984). Multiple assessment center exercises and job relatedness. Journal of Applied Psychology, 69, 182–186.Google Scholar
Pervin, L. A. (1989). Persons, situations, and interactions: The history of a controversy and a discussion of theoretical models. Academy of Management Review, 14, 350–360.Google Scholar
Reilly, R. R., Henry, S., & Smither, J. W. (1990). An examination of the effects of using behavior checklists on the construct validity of assessment center dimensions. Personnel Psychology, 43, 71–84.Google Scholar
Robie, C., Osburn, H. G., Morris, M. A., Etchegaray, J. M., & Adams, K. A. (2000). Effects of the rating process on the construct validity of assessment center dimension evaluations. Human Performance, 13, 355–370.Google Scholar
Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity: Updating and integrating some classic literature. Personnel Psychology, 58, 1009–1037.Google Scholar
Russell, C. J., & Domm, D. R. (1995). Two field tests of an explanation of assessment centre validity. Journal of Occupational and Organizational Psychology, 68, 25–47.Google Scholar
Sackett, P. R., & Dreher, G. F. (1982). Constructs and assessment center dimensions: Some troubling findings. Journal of Applied Psychology, 67, 401–410.Google Scholar
Sackett, P. R., & Dreher, G. F. (1984). Situation specificity of behavior and assessment center validation strategies: A rejoinder to Neidig and Neidig. Journal of Applied Psychology, 69, 187–190.Google Scholar
Sackett, P. R., & Harris, M. J. (1988). A further examination of the constructs underlying assessment center ratings. Journal of Business Psychology, 3, 214–229.Google Scholar
Sackett, P. R., & Tuzinski, K. (2001). The role of dimensions in assessment center judgment. In London, M. (Ed.), How people evaluate others in organizations (pp. 111–129). Mahwah, NJ: Erlbaum.Google Scholar
Sagie, A., & Magnezy, R. (1997). Assessor type, number of distinguishable categories, and assessment centre construct validity. Journal of Occupational and Organizational Psychology, 67, 401–410.Google Scholar
Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735–746.Google Scholar
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274.Google Scholar
Schneider, J. R., & Schmitt, N. (1992). An exercise design approach to understanding assessment center dimension and exercise constructs. Journal of Applied Psychology, 77, 32–41.Google Scholar
Silverman, W. H., Dalessio, A., Woods, S. B., & Johnson, R. L. Jr. (1986). Influence of assessment center methods on assessors’ ratings. Personnel Psychology, 39, 565–578.Google Scholar
Smith, F. D. (1991). Work samples as measures of performance. In Wigdor, A. K. & Green, B. F. (Eds.), Performance assessment for the workplace (Vol. 2, pp. 27–52). Washington, DC: National Academy Press.Google Scholar
Swann, W. B. (1984). Quest for accuracy in person perception: A matter of pragmatics. Psychological Review, 91, 457–477.Google Scholar
Teachout, M. S., & Pellum, M. W. (1991). Air force research to link standards for enlistment to on-the-job performance (AFHRL-TR-90–90). Brooks AFB, TX: Air Force Human Resources Laboratory, Training Systems Division.Google Scholar
Tett, R. P., & Burnett, D. D. (2003). A personality trait-based interactionist model of job performance. Journal of Applied Psychology, 88, 500–517.Google Scholar
Thoresen, J. D. (2002, October). Do we need dimensions? Dimensions limited or unlimited. Paper presented at the meeting of the International Congress of Assessment Center Methods, Pittsburg, PA.Google Scholar
Thornton, G. C. III, & Byham, W. C. (1982). Assessment centers and managerial performance. New York: Academic.Google Scholar
Thornton, G. C. III, & Mueller-Hanson, R. A. (2004). Developing organizational simulations: A guide for practitioners and students. Mahwah, NJ: Erlbaum.Google Scholar
Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1–26.Google Scholar
Woehr, D. J., & Arthur, W. Jr. (2003). The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Journal of Management, 29, 231–258.Google Scholar