Alan Howard, Max Choi & Johnny Mitchell

COMBINING behaviour-based and questionnaire-based assessments of personality into one single test is exciting, but not new. Indeed, it may be traced back to Professor Raymond Cattell’s original plan back in the 1950s. Cattell experimented with Q-data (questionnaires), T-data (task or behaviour-based approaches) e.g. Cattell and Warburton (1967) and L-data (life data from observations e.g. 360 degree ratings, assessment centres etc). Q-data was relatively the easiest to progress, so through his work he achieved the famous 16PF, whereas T-data was hampered by a lack of computing power, so he could not easily progress that.

What exactly is a behaviour-based approach to the assessment of personality – or what is T-data looking like today?

Already several gamification providers have developed online games to assess personality. Our Mosaic behaviour-based approach focuses on tasks rather than games, as playing games may not be as acceptable for much of the working population. The assessment runs like any other online psychometric assessment. However, it is not at all obvious to respondents what aspects of personality are being assessed by each game or task, thus preventing the assessment from becoming transparent and therefore open to manipulation. The tasks capture many behavioural measures e.g. use of time; error or omission rates; the order in which elements of activities are completed; measures of memory for one type of material over others; etc.

Does a behaviour-based approach to assessing personality work?

 We can check this by conducting validation studies and checking that the results are replicable and robust. Eisenberg et al. (2019), in a study using 22 self-report measures and 37 behaviour-based measures of impulsivity found significant but modest correlations of <0.2 between the two. However, the authors state that the study was not as good as it could have been because the validity behind many of the already published behaviour-based measures of impulsivity used in the study had not been rigorously checked beforehand, merely their creators expected that their behavioural measures would pick up impulsivity. Whereas, in our Mosaic tasks research, we cross-validated to select and then repeatedly check that the right behaviour-based measures were being used. We found an average correlation of 0.32 against 20 NEO personality questionnaire facets e.g. Mitchell et al. (2019), Howard et al. (2021). As we are not really comparing ‘apples with apples’ this 0.32 correlation is deemed to be strong evidence of the behaviour-based approach’s capacity to measure personality.

Looking next at the self-report questionnaire approach to assessing personality, it potentially introduces two types of error:

1. Massaged or fake responses from candidates i.e. social desirability responses where individuals present themselves in a better light. Many questionnaires have a Social Desirability scale included, which flags up this issue. But it does not inform us of how a candidate’s socially desirable responses have exactly impacted on the rest of their profile. So we are looking for an approach that will enlighten us on this.

2. Error due to blind spots. We rely on a person having sufficient self-awareness. But no one has perfect self-insight and awareness, so everyone has blind spots to some extent. And this will inevitably influence the profile obtained.

As the behaviour-based approach overcomes these two issues, one could argue that it is time to ditch the personality questionnaire approach for good. What is the value of keeping it? Well, research data clearly suggest that questionnaires overall still do a decent job i.e. sometimes the two sources of error may mean there is an issue, but typically it works. This leaves practitioners with a problem at the micro level (can we trust this candidate’s results?), even if researchers find that things are working not too badly at the macro level. Personality questionnaires are well-established and supported by robust research. Also, many studies have demonstrated the relationships between various personality traits and overall job performance measures e.g. Barrick & Mount (1991); Hough (1992); Salgado (1997, 1998), Bartram et al. (2003). In other words, personality questionnaires are valid and ‘work’.

Therefore, by combining a questionnaire with another personality assessment approach, this will highlight any inaccuracies with the questionnaire results obtained for an individual, and we would get a better overall personality assessment. 

So, what else could be used to supplement questionnaires or verify their accuracy?

We can use ‘Peer’ ratings of personality. However, these suffer from the same social desirability issues as self-report questionnaires. For example, 360-degree ratings tend to be either positive or very positive.

Other techniques available rely on obtaining personality data via less transparent means i.e. that can’t easily be manipulated or consciously controlled. Fitting this would be:

■ Projective techniques such as the Rorschach inkblot test (Rorschach, 1927) – but they have reliability problems and are not acceptable in a business context anyhow.

■ Techniques based on recent social media behaviour – but are unacceptable in a business context, this time for privacy e.g. GDPR reasons.

■ Personality assessment based on Neuropsychology, whilst showing some promise (e.g. Cai et al., 2020), is again not a practical or cost-effective option.

Therefore, behaviour-based, objective task measures look to be our best way of practically verifying personality questionnaire data. So by combining both it offers a broader and more robust assessment of personality than can be achieved by one method alone.

Our rationale is quite simple: Combining the two provides additional powerful insight to truly understanding personality. Many personality questionnaire completions will generate valid, useful information. Sometimes the Social Desirability score might be high, but we still do not really know whether the profile is useful. By combining the two approaches we can conduct a validity check. Does the questionnaire data match the behaviour-based data?

Further, by looking at the differences between the two sources of data, we can give participants very useful insight into their strengths and potential blind spots. That is, for any personality trait e.g. Conscientiousness, does the person’s own view via their questionnaire match, exceed, or undervalue their actual objective behaviour-based assessment of Conscientiousness?  If the two sources match, we have a particularly robust assessment of Conscientiousness.  If the questionnaire is significantly higher or lower, have we uncovered a blind spot? This is very helpful from a developmental perspective e.g. when coaching clients or working with leaders. It is also particularly useful in high-stakes selection contexts.

We were also able to develop a Self-Perception scale (so a good improvement on what the Social Desirability scale was attempting to achieve) by researching the scales that are prone to Social Desirability and comparing them with our Mosaic measures. In a selection setting, those scoring around stens 3–8 on our Self-Perception scale returned useable questionnaire data. Those scoring 9 and 10 have exaggerated their strengths i.e. is not supported by the Mosaic behavioural tasks data. Those scoring stens 1 and 2 have underplayed their strengths and may have more to offer than they actually report in their questionnaire! Practitioners can make use of this information when interpreting reports i.e. relying more on the Mosaic tasks results as opposed to questionnaire results when more extreme scores are obtained on the self-perception scale.

Therefore, the objective is to use the two personality approaches in complementary ways rather than to argue that one is clearly better than the other. With our research we have been able to produce a behavioural tasks and questionnaire-based assessment of the full range of adult personality scales, which we believe may be the only tool available worldwide to do this in a fully integrated way i.e. one assessment, one personality model.

So perhaps we are just completing something that Raymond Cattell had intended to do about 70 years ago – well, better late than never!

The Authors of : The best of both worlds? Combining behaviour-based and questionnaire-based assessments of personality.

Alan Howard, Max Choi & Johnny Mitchell are all Chartered Occupational Psychologists and Directors of Mosaic Assessments Limited. Correspondence: alan@mosaictasks.com for further information.

References

Barrick, M.R. & Mount, M.K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44(1), 1–26.

Bartram, D., Kurtz, R. & Baron, H. (2003). The Great Eight Competencies: Meta-analysis using a criterion-centric approach to validation. Paper presented at SIOP, Orlando, April 2003.

Cai, H., Zhu, J & Yu, Y. (2020). Robust prediction of individual personality from brain function connectome. Social Cognitive and Affective Neuroscience. April 2020.

Cattell, R.B. & Warburton, F.W. (1967). Objective personality and motivation tests. Urbana: University of Illinois Press.

Eisenberg, I.W., Bissett, P.G., Zeynep, A. et al. (2019). Uncovering the structure of self regulation through data-driven ontology discovery. Nature Communications. doi.org/10.1038/s41467-019-10301-1

Hough, L.M. (1992). The Big Five personality variables – construct confusion: description versus prediction. Human Performance 5, 139–155.

Howard, A.J. Choi, M & Mitchell, J.D. (2021). An Alternative to Personality Questionnaires: Introducing Personality Tasks. Paper presented as part of a Symposium ‘Opportunities and Challenges for Personality Assessment’ International Test Commission Colloquium 2021 University of Luxembourg.

Mitchell, J.D., Choi, M. & Howard, A. (2019). ‘An Alternative to Personality Questionnaires’. Paper presented at the BPS Division of Occupational Psychology Conference, January 2019.

Rorschach, H. (1927). Rorschach Test – Psychodiagnostics Plates. Cambridge, MA: Hogrefe Publishing Corp.

Salgado, J.F. (1997). The five factor model of personality and job performance in the European Community. Journal of Applied Psychology, 8 2, 30–43.

Salgado, J.F. (1998). Big Five personality dimensions and job performance in army and civil occupations: A European perspective. Human Performance, 11, 271–288.