Identifying the Factors Affecting Construct Validity of the Assessment Centers by Using Meta-Synthesis Approach

Document Type : Research Paper

Authors

1 Corresponding Author, Ph.D. Candidate, Department of Human Resource Management, Faculty of Management, University of Tehran, Tehran, Iran

2 Full Professor, Department of Public Administration, Faculty of Management, University of Tehran, Tehran, Iran

3 Assistant Professor, Department of Public Administration, Faculty of Management, University of Tehran, Tehran, Iran

Abstract

Due to the changing the type of competencies required by the organizations and on the other hand, the necessity of controlling human resource costs, proper employment of people and then their development and maintenance have become one of the most important concerns of the global organizations. Process of assessment center as a means with multiple approaches, has been assisting organizations in this regards. In this study , the factors affecting the construct validity of the assessment centers have been studied . Despite of many researches, there are still controversial issues in this field, for instance, one of them is the topic of construct-related validity. This study has been done with the aim of developing knowledge in the field of construct validity of assessment centers. In spite of the fact that predictive validity of assessment centers was reported in many studies at the appropriate level, construct validity was weak. Therefore this study is focused on factors which can influence the construct validity. By utilizing meta-synthesis approach and thematic analysis method.78 related papers have been studied and 5 factors including dimensions, assessors, methods of observing and evaluating, methods of integrating information and situational exercises have been identified as effective factors on the construct validity of assessment centers.

Keywords


ابویی‌اردکان، محمد؛ لبافی، سمیه؛ آذرپور، سمانه؛ جاللپور، مهدیه (1393)، شناسایی عوامل حیاتی موفقیت تفکر راهبردی در دیدگاه مدیران سازمان‌های فرهنگی شهر اصفهان، پژوهش‌نامه مدیریت اجرایی، 6-11.
آذرفر، امیرعلی؛ شیری، محمدمهدی؛ محمدی، محمدحسین (1397)، اثر همانندی موقعیت بر امتیازات داوطلبان در کانون‌های ارزیابی، پژوهش های مدیریت منابع انسانی، 10 (4): 24-1.
اکبری،  مهناز؛  فیاضی، مرجان؛  جمشیدی، حمزه (1395)، شناسایی  و  اولویت بندی عارضه‌های کانون‌های ارزیابی  و توسعه در ایران، پژوهش های مدیریت منابع انسانی، 8 (1): 103-130.
خشوعی، مهدیه‌سادات؛ عریضی، حمیدرضا؛ جهان‌بازی، افشین (1397)، اعتباریابی کانون ارزیابی مدیران در شرکت گاز اصفهان، دانش و پژوهش در روان‌شناسی کاربردی، 19 (3): 49-34.
خشوعی، مهدیه‌سادات؛ عریضی، حمیدرضا؛ نوری، ابوالقاسم (1393)، طراحی و اجرای کانون ارزیابی ارتقایی مدیران، پژوهش‌های مدیریت منابع انسانی، 6 (3): 47-27.
خواستار، حمزه (1387). بررسی عوامل مؤثر بر خودکارآمدی ملی در زمینه علم و فناوری برای تحقق اهداف چشمانداز، پایان‌نامه کارشناسی‌ارشد (چاپ‌نشده)، به راهنمایی دکتر آرین قلی‌پور، دانشکده مدیریت دانشگاه تهران.
Arthur W, Woehr DJ, Maldegen R. Convergent and Discriminant Validity of Assessment Center Dimensions: A Conceptual and Empirical Reexamination of the Assessment Center Construct-Related Validity Paradox. Journal of Management. 2000;26(4):813-835.
     Ballantyne Iain and Nigel Povah. (2016) .Assessment & development centers, second edition, Published by Routledg.
    Bowler,Mark C.  Woehr,b David J. (2009) Assessment center construct-related validity: Stepping beyond the MTMM matrix, Journal of Vocational Behavior.Volume 75, Issue 2, October 2009, Pages 173-182
   Bagozzi,R.P.,Yi,Y and Philips,L. (1991).Assessing construct validity in organizational research.Administratitive Science, Quarterly.36,421-436
   Baker,T.A. (1986) Multi trait-Multi method analysis of performance ratings using behaviorally anchored and behavioral checklist formats, Unpublished Master’s thesis,Old Dominion University,Norfolk
   Bernardin, H.J. and Buckley, M.R. (1984) Strategies in rater training.Academy of management review.6.205-212
Bining,J.F. Adorno, A.J. and Kroeck, K.G. (1997) Validity of behavior checklist and assessor judgmental ratings in an operational assessment center. Paper presented at the conference of society for industrial and organizational psychology.St. Louis, MO
Bowler,Mark C.  Woehr,b David J. (2009) Assessment center construct-related validity: Stepping beyond the MTMM matrix, Journal of Vocational Behavior.Volume 75, Issue 2, October 2009, Pages 173-182
Brannick, M. T. (2008). Back to basics of test construction and scoring. Industrial and Organizational Psychology, 1(1),131-133.
Buckett, A., Becker, J. R., Melchers, K. G., & Roodt, G. (2020). How different indicator-dimension ratios in assessment center ratings affect evidence for dimension factors.Frontiers in Psychology, 11, 459.
Cahoon, M. V., Bowler, M. C., & Bowler, J.L. (2012). A reevaluation of assessment center construct-Related validity. International Journal of Business and Management, 7 (9), 3- 19.
Campbell, W.J. (1986) Construct validation of roleplaying exercises in an assessment center using BARS and behavioral checklist formats. Unpublished Master's thesis, Old Dominion University, Norfolk.
Connelly, B. S., Ones, D. S., Ramesh, A., & Goff, M. (2008). A pragmatic view of assessment center exercises and dimensions. Industrial and Organizational Psychology, 1(1),121-124
Donahue, L.M., Truxillo, D.M., Cornwell, J.M. and Gerrity, M.J. (1997) Assessment center construct validity and behavioral checklists: some additional findings. Journal of Social Behavior and Personality, 12, 85-108.
Dugan, B. (1988) Effects of assessor training on information use. Journal of Applied Psychology, 73,
743-748.
Gaugler, B.B. and Thornton, G.C. (1989) Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology, 74,611-618.
Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., III, & Bentson, C. (1987). Meta‑analysis of assessment center validity. Journal of Applied Psychology, 72, 493‑511.
Haaland,S., and Christiansen, N. D., (2002) Implications of trait activation theory for evaluating of construct validity of assessment center ratings. Personnel psychology, 55,137-163
Harris, M.M., Becker, A.S. and Smith, D.E. (1993) Does the assessment center scoring method affect the cross-situational consistency of ratings? Journal of Applied Psychology, 78, 675-678.
Hennessy, J., Mabey, B., & Warr, P. (1998). Assessment centre observation procedures: An experimental comparison of traditional, checklist and coding methods. International Journal of Selection and Assessment, 6(4), 222-231.
Highhouse, S., & Harris, M. M. (1993). The measurement of assessmentcenter situations: Bem’stemplate matching technique for examining exercise similarity. Journal of Applied Social Psychology, 23, 140–155
Höfling, V., Schermelleh-Engel, K., & Moosbrugger, H. (2009). Analyzing multitrait-multimethod data: A comparison of three approaches. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(3), 99–111
Howard, A. (1997). A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality, 12(5), 13–52.
Howard, A. (2008). Making assessment centers work the way they are supposed to. Industrial and Organizational Psychology, 1(1), 98-104.
Jaeger, K. S. (2011). The Role of Assessment Center Work Simulation Exercises in Determining or Influencing Assessors’ Judgments of Leadership Competencies. Unpublished doctoral dissertation, University of Minnesota
Joyse, Leslie W., Thayer, Paul W., Pond III, Samuel B., (1994), Managerial Functions: An Alternative to Traditional Assessment Center Dimensions ?, Personnel Psychology , 47 (1):109 - 121
Kalleberg, Arne L,. Kluegel, James R., (1975) , Analysis of the Multitrait-Multimethod Matrix: Some Limitations and An Alternative , Journal of Applied Psychology ,Vol. 60, No. 1, 1-9
Kauffman, J. R., Jex, S. M., Love, K. G., & Libkuman, T. M. (1993). The Construct Validity of Assessment Centre Performance Dimensions. International Journal of Selection and Assessment, 1(4), 213-223.
Kleinmann,M.,& Ingold, P.V. (2019) Toward a better understanding of assessment centers:A conceptual review.Annual review of organizational psychology and organizational behavior, 6, 349-372
Kleinmann, M. (1993) Are rating dimensions in assessment centers transparent for participants? Consequences for criterion and construct validity. Journal of Applied Psychology, 78, 988-993.
Kleinmann, M., Kuptsch, C. and Koller, O. (1996) Transparency: A necessary requirement for the construct validity of assessment centres. Applied Psychology: An international Review, 45, 67-84.
Kolk, N. J., Born, M. P., & Der Flier, H. V. (2003). The transparent assessment centre: The effects of revealingmdimensions to candidates. Applied Psychology, 52(4), 648-668.
Krause,D,E., & Thornton, G.C (2009) Across cultural look at assessment center practices: Survey result from Western Europe and North America.Applied psychology: An international review, 58(4), 557-585
Lance, C. E., Lambert, T. A., Gewin, A. G., Lievens, F., & Conway, J. M. (2004). Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. Journal of Applied Psychology, 89 , 377–385
Lance, C,E. (2008) Why assessment centers do not work the way they are supposed to.Industrial organizational psychology:perspectives on science and practice,1, 84-97
Lievens, Filip and Klimoski, Richard J. Understanding the assessment center process: Where are we now. (2001). International Review of Industrial and Organizational Psychology. 16, 245-286. Research Collection Lee Kong Chian School Of Business
Lievens, F., & Sackett, P. R. (2007). Situational judgment tests in high-stakes settings: Issues and strategies with generating alternate forms. Journal of Applied Psychology, 92(4), 1043.
Lievens, F. (1998). Factors which Improve the Construct Validity of Assessment Centers: A Review. International Journal of Selection and Assessment, 6(3), 141–152.
Lorenzo, R.V. (1984) Effects of assessorship on managers' proficiency in acquiring, evaluating,and communicating information about people.Personnel Psychology, 37, 617-634. 
Lorenzo, R.V. (1984) Effects of assessorship on managers' proficiency in acquiring, evaluating,and communicating information about people.Personnel Psychology, 37, 617-634. 
Louiselle, K.G. (1986, March) Confirmatory factor analysis of two assessment center rating procedures, Paper presented at the IO/OB Graduate Student Conference, Minneapolis, MS.
Marsh, H.W. (1989) Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361.
Petrides, K. V. , Weinstein, Yana , Chou, Jenny , Furnham, Adrian and Swami, Viren(2010) 'An investigation into assessment centre validity, fairness, and selection drivers', Australian Journal of Psychology, 62: 4, 227- 235
Mattimore, K. Nalodka, T. and McCormik, S. (1995) Direct, Indirect and Controlled observation and ratinf accuracy.Journal of applied psychology, 80, 664-670
Moses, J. (2008). Assessment centers work, but for different reasons. Industrial and Organizational Psychology, 1(1),134-136.
Reilly, R.R., Henry, S. and Smither, J.W. (1990) An examination of the effects of using behavior checklists on the construct validity of assessment center dimensions. Personnel Psychology, 43, 71-84.
Riggio.R,E. Myes, B. T, and Riggio R.E., (2003) Using assessment center methods for measuring undergraduate business student outcomes,Journal of management inquiry, 12, 68-78
Russell,C.J. (1985) Individual decision processes in an assessment center.Journal of applied psychology, 70, 737-746
Russell,C.J. and Domm, D.R.(1995) Two field tests of an explanation of assessment center validity.Journal of occupational and organizational psychology, 68, 25-47
Ryan, A.M., Daum, D., Bauman, T., Grisez, M., Mattimore, K., Nalodka, T. and McCormick, S. (1995) Direct, indirect, and controlled observation and rating accuracy. Journal of Applied Psychology, 80, 664-670.
Sackett, P. R., & Dreher, G. F. (1982). Constructs and assessment center dimensions: Some troubling findings. Journal of Applied Psychology, 67, 401–410.
Sackett, P. R., & Tuzinski, K. A. (2001). The role of dimensions and exercises in assessment center judgments. In M. London (Ed.), How people evaluate others in organizations (pp. 111–129).
Sackett, P. R., & Dreher, G. F. (1984). Situation specificity of behavior and assessment center validation strategies: A rejoinder to Neidig and Neidig. Journal of Applied Psychology, 69, 187–190.
Sackett, P. R., & Lievens, F. (2008). Personnel selection. Annual review of psychology, 59, 419–450.
Sackett, P. R., Shewach, O. R., & Keiser, H. N. (2017).Assessment centers versus cognitive ability tests:Challenging the conventional wisdom on criterion-related validity. Journal of Applied Psychology, 102, 1435-1447.
Sagie, A. and Magnezy, R. (1997) Assessor type, number of distinguishable categories, and assessment centre construct validity. Journal of Occupational and Organizational Psychology, 70,103-108.
Salehi,Mohammad and Tayebi Alireza.(2012) Differential Item Functioning: Implications for Test Validation,Journal of Language Teaching and Research, Vol. 3, No. 1, pp. 84-92,
Salimi, Majid and Shokrkon, Hossein.(2020) A critical approach to the construct-related validity of assessment centers: A study of assessment center of international Irainian Oil Company.International journal of psychology, Vol.14, No 2, PP:218-247
Sandelowski, M., & Barroso, M. (2007). Handbook for synthesizing qualitative research. New York: Springer.
Silverman, W.H., Dalessio, A., Woods, S.B. and Johnson, R.L. (1986) Influence of assessment center methods on assessors' ratings. Personnel Psychology, 39, 565-578.
Schleicher, Deidra J., Day, David V., Mayes, Bronston T., Riggio, Ronald E.(2002), ‘A New Frame for Frame-of-Reference Training: Enhancing the Construct Validity of Assessment Centers’, Journal of Applied Psychology, 87, (4)
Schneider, J. R., & Schmitt, N. (1992). An exercise design approach to understanding assessment center dimension and exercise constructs. Journal of Applied Psychology, 77, 32–41
Slamoulis, D.T. and Hauenstein, N.M.A. (1993) Rater training and rating accuracy:Training for dimensional accuracy versus training for rate differentiation.Journal of applied psychology,78,994-1003
Spector, Paul E., Schneider, Jeffrey R., Vance, Carol A., Hezlett Sarah A., (2006), The Relation of Cognitive Ability and Personality Traits to Assessment Center Performance, Jounal of Applied Social Psychology volume 30, issue 7,1474-1491
Spychalski,A.C. Quinones, M.A. Gaugler, B.B. and Pholey, K.(1997) A survey of assessment center practices in organizations in the United States, Personnel psychology, 50, 71-90
Tan,M.(1996) The effect of role-player standardization on the construct validity of dimensions in assessment exercises.Unpublishes doctoral dissertation,University of Amesterdam.
Task Force on Assessment Center Guidelines. Guidelines and ethical considerations for Assessment Center operations.(2015) Journal of management
Thornton, G. C., III, Rupp, D. E., & Hoffman, B. J. (2015). Assessment center perspectives for talent management strategies (2nd ed.). New York: Routledge
Thornton, G. C., Ill, & Rupp, D. E. (2006). Assessment centers in human resource  managemewnt: strategies for prediction, diagnosis and development. Lawrence Erlbaum Associates
Thornton,C.George and Gibbson, Alissa,M.(2009) Validity of assessment centers for personnel selection.Human resource management review 19,169-187
Thornton, G. C., Ill., & Mueller-Hanson, R. A. (2004). Developing organizational simulations: A guide for practitioners and students. Lawrence Erlbaum Associates.
Tseng, T. (2011). Development of an Assessment Center as a Selection Method for I/O Graduate Applicants. Unpublished Master’s thesis, Minnesota State University, Mankato.
Wiese, C.W. (2015). Assessment Center Structure and Construct Validity: A New Hope.Electronic Theses and Dissertations, 2004-2019. 733
Wirz, A., Melchers, K. G., Kleinmann, M., Lievens, F., Annen,H., Blum, U., & Ingold, P. V. (2020). Do overall dimension ratings from assessment centers show external constructrelated validity? European Journal of Work and Organizational Psychology, 29(3), 405-420.
Woehr, D,J. & Arthur, W. (2003) The construct related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors, Journal of management, 29. 231-258
Woehr, D,J. (1994) Understanding frame-of-refrence training:The impact of training on the recall of performance information. Journal of applied psychology, 79. 525-534
Woehr, D,J. (1992) Performance dimension accessibility:implications for rating accuracy. Journal of organizational behavior,13,357-367
Zimmer L,.(2006) Qualitative meta-synthesis: a question of dialouging with texts. Methodological issues in nursing research.53(3):311-8