Examining personality assessment in asynchronous video interviews (AVI): convergence between human personality judgements and AI/ML scoring

Cronje, Jacobus Fouche

Examining personality assessment in asynchronous video interviews (AVI): convergence between human personality judgements and AI/ML scoring

Thesis / Dissertation

2025

Publisher

University of Cape Town

Department

Organisational Psychology

Faculty

Faculty of Commerce

Abstract

The assessment of personality is an essential component of personnel selection due to its validity in predicting job performance. To assess personality, asynchronous video interviews (AVIs) scored using artificial intelligence (AI) algorithms are increasingly used, allowing candidates to record responses to interview prompts that are subsequently evaluated automatically by AI algorithms and/or human raters. As questions remain about the validity of AI-based AVI scoring approaches, this study examines the convergence between human-and AI-scored personality assessments. To measure personality, the study focuses on the HEXACO model, which measures Honesty-Humility, Emotionality, Extraversion, Agreeableness, Conscientiousness, and Openness to Experience. Verbal responses were transcribed from videotaped AVIs of 161 mock interview candidates who answered five AVI questions. Responses were scored by 15 trained human raters and a closed-dictionary text-analysis keyword-counting AI algorithm developed for this study, respectively. The correlation between trait-level scores produced by human judges and AI scoring was tested both across traits and within traits (trait-level) to assess scoring convergence. Moreover, in addition to comparing score levels produced by the two scoring methods (AI vs. human raters), score spread (i.e., variability), rank-order stability, and rating reliability were evaluated. The findings revealed a moderately positive and significant overall convergence (r = .29, p < .001) across traits between human and AI evaluations, which suggests that AI scoring may potentially be useful as a replacement of human evaluations when general screening is desired. Trait-level convergence varied between scoring methods, with the scoring consensus between human raters and AI being higher for some traits than for others, suggesting that these methods rely on different information and/or may interpret interview responses differently. The research highlights the potential of AI to complement human- based scoring of AVIs used in recruitment, selection, and assessment while also identifying the limitations of algorithm-based scoring in capturing complex human behaviour in interviews. The findings may further contribute to understanding the role of AI in personality assessment and implications for organisational practices.