How Well Do Assessment Centres Work?

This is the first question to appreciate. If you cannot run your assessment centres (ACs), you want to try to understand what you are actually losing. The reality is that many assessment centres are not that effective!  Organisations that run them clearly think they are, but hardly anyone bothers to conduct any proper evaluation to prove how effectively they actually work – so many organisations do not know for sure how effective their ACs really are!

Below we will take you through how effective ACs are working today.

We are the experts in this area and we know that literally every organisation’s assessment centre can be significantly improved as there are so many things that can contribute to error and poor effectiveness. Therefore, converting them to virtual assessment centres will typically make them even more less effective! Unless you clearly know what you are doing, this is going to be the outcome. Not convinced? Here is a simple real example of the issues.

Example: Graduate Assessment Centres

Graduate Assessment Centres love their Group exercises. But most of these Group Exercises never provide consistent and accurate assessments. The reality is when you put candidates who do not know each other in a group then what happens can be quite varied and the quality of the assessments is rather variable.

For example, you might have a situation where 2 individuals have a bit of an argument and certain observations and assessments are derived from that e.g.

• Candidate A “stood his ground and defended his case”;
• Candidate B “acted aggressively and attacked the other candidate”.

However, the problem is that we did not give all the candidates an opportunity to be assessed on these qualities of “standing their ground and defending their case” and we have not assessed whether others might “lose their temper when provoked”. It really falls down on a fundamental aspect of fair and effective assessment – we have NOT given all candidates a level playing field to demonstrate their capabilities in exactly the same areas.

Are Virtual Exercises the same?

It is not a good starting point when the exercise is pretty bad already.  Then we are going to turn it into a virtual Group Exercise using video camera technology. Do we really believe that the Group Exercise is identical when this happens?

It clearly is not.

For example, in a room you can clearly make eye contact and direct your facial expressions at a specific individual. You might state: “Great idea!”  You say that cynically meaning the opposite and you will be staring at Chris with your facial expression. Chris clearly has got the message!  But when you do that same behaviour using video, then both your communication and what others pick up from it is not likely to be identical! Therefore, the two environments are not identical or equivalent. 

Therefore, it is not simply just loading your traditional Assessment Centre Exercises onto the virtual platform and assuming everything will work exactly the same. So quite challenging – but not impossible to achieve effective virtual Assessment Centres. Let us save that for a future article.

As it is not at all easy to get a virtual Assessment Centre right, plus the fact that it is going to be very resource hungry and expensive – we want to give you another alternative. What is the next best thing? Possibly it is the BETTER THING – in terms of validity, efficiencies, and costs.  Do you want to understand why I say this?

Assessment Centre Validation Research

Let’s see what Assessment Centre validity research tells us over the decades. A meta-analysis (which is a large research study looking at many assessment centres) in 1987 produced an average correlation of 0.36 between Overall Assessment Rating (OAR) and job performance ratings.

A more recent meta-analysis in 2007 produced an average correlation of 0.28 between Overall Assessment Rating (OAR) and supervisory job performance ratings.

One needs to be careful as we are not necessarily perfectly comparing apples with apples but it is a relatively equivalent comparison, so it does seem to appear that assessment centres may now be less effective. Perhaps more organisations are applying assessment centres and some may not be running them that well. Therefore, the reality is that many organisations are putting in a lot of resources and money to run assessment centres – but they may not be working that effectively.

So lets explore an alternative.

These are our Situational Judgement tests (SJTs). First, it might be useful to understand how we came up with developing the first SJTs back in the late 1990’s. In terms of testing at management level, recruiters were relying heavily on verbal and numerical critical reasoning tests. But it was obvious to me that having a good score on these tests does not prove that they are going to be effective managers. A good score on the numerical test just tells us they have useful business numeracy skills and on the verbal test that they can assimilate and understand business written information.

For Management Scenarios we obtained a correlation of 0.43 between the Overall Test Performance and the Manager’s Job Level. Also, a range of other validation studies provide correlations all above 0.28. This is at or better than what we get for assessment centres.  Again, we need to be very careful as we are clearly not comparing apples with apples here.  But we are likely to be able to get a better result by using the Scenarios SJT approach rather than running a full-blown assessment centre.  Moreover, when we supplement with a verbal and numerical test and a personality instrument, the validity goes up further.

Here is a simple example:

Example Scenario

You have just been promoted to run a department which was reputedly well managed and highly effective under your predecessor. She has subsequently been promoted to a new post within the organisation in another part of the country.

During your initial week you would:

1. Personally conduct an immediate and critical review of all existing departmental policies and practices.
2. Inform your staff that as the new manager you will be bringing in your preferred processes and methods immediately.
3. Ask your staff for ideas on how to make further improvements to the department.

We conducted detailed research to develop the scenarios and the questions, and also used subject matter experts (SMEs) to identify the correct answers. The test comprises 16 Scenarios.

In terms of the managerial judgement model we would allow the statistical evidence to guide us.  That is to say, we wanted our model to be evidence-based and not built on unproven theories or models of management or leadership. The statistical analysis identified three main areas of managerial judgement:

• Managing Objectives
• People Management
• Reputation Management

Each question loads onto one of these three sub-scales. The three sub-scales also combine to give a total score on Managerial Judgement. Here is a sample Profile based on the Management Scenarios (but similar sections for the other two Scenarios tests, Executive and Graduate Scenarios):

MANAGING OBJECTIVES (30 items)

Making the best possible use of one’s energies at work:

Focus energies on appropriate issues. Have strong critical awareness, and quickly see the important underlying issues. Know what to tackle personally or to delegate. View objectives as part of the organisational whole and realise the importance of aligning efforts with colleagues. Likely to be a strong networker who seeks to get involved in wider initiatives and projects.

PEOPLE MANAGEMENT (28 items)

Effectively managing a team of direct reports.

Have strong insight into managing people and an excellent understanding of how best to build and maintain strong relationships with staff members over the long-term. Managing actions will be informed by a strong insight into how best to motivate and utilise people positively. Will consider the impact of actions on on-going relationships with staff. Take a proactive but non-judgemental approach to dealing with staff issues, preferring to deal positively and constructively with problems.

REPUTATION MANAGEMENT (26 items)

Awareness of how to manage one’s reputation in an organisation.

Balance the long-term interests of the organisation with the pursuit of their own agenda. Tackle organisational issues and build a good team reputation. Ensure contributions are recognised by the important people, and have a sound understanding of how to access key decision makers and make use of informal networks. Give appropriate consideration to how actions/decisions might impact on the different reporting lines e.g. bosses, peers and direct reports. Focus on the needs of the organisation and have the company’s interests at heart. Deal tactfully with clashes of interest.

From the research findings we could also identify some behavioural measures and this would be very useful to support feedback to individuals. We have actually developed a test that provides both a performance/ ability measure as well as a behavioural measure.  Our professional body, the British Psychological Society could not cope with that and classified it simply as an ability test!

The behavioural measures are called Elements. We worked up two Elements for each of the three sub-scales (descriptions describe the right-hand side of the scale):

MANAGING OBJECTIVES

Big Picture: Advocates wide consultation with other teams on objectives and decisions. Lines up own efforts behind wider organisational objectives.

Delegative: Avoids detail, tends to involve others and to delegate. Distills out and tackles important tasks and sets self priorities.

PEOPLE MANAGEMENT

One-to-One: Tackles an individual who has motivational and performance problems. Offers support and coaching whilst emphasising the need to meet objectives.

Team: Consults on, clarifies and tackles team motivation and performance problems, promotes and maintains team motivation. Communicates and consults on change.

REPUTATION MANAGEMENT

Personal Recognition: Takes a balanced approach in attempting to manage their own image within the organisation. May overestimate the importance of letting results speak for themselves rather than occasionally highlighting own achievements.

Company Protocol: More likely to focus on reaching decisions and achieving objectives through established protocol. More willing to accommodate the needs of other managers and to tackle the organisation’s collective problems.

The Management Scenarios Test comprises of 16 Scenarios. It is not strictly timed as we need a response to all the items. Typically, it takes 40 – 45 minutes to complete.  All the above scales are measured and compared against the most appropriate norm group.

There are three Scenarios tests assessing managerial judgement at different levels:

1. Graduate Scenarios: For graduates, undergraduates. As these individuals are likely to be our future managers then it is useful to assess their potential for managerial judgement.
2. Management Scenarios: The test described above. For managers, team leaders, group heads, senior professionals.
3. Executive Scenarios: For senior managers, directors, executives.

How the Tests Could be Used Instead of Running an AC?

To Support Recruitment

The purpose of running an AC is to assess the candidates’ suitability for the role. Therefore, our view (and proven by research) is that you can actually do that better by:

Using the appropriate Scenarios test – as this assesses the correct level of managerial judgement required (see above).

Using appropriate aptitude tests i.e. verbal critical reasoning and numerical critical reasoning tests. These assess their ability to assimilate business information. Note that these assess useful but narrow areas of skill e.g. the numerical test provides information on their business numeracy and data handling – but that is all it tells us.

Using a personality Interview Report to support a Competence-Based Interview. This is important as intellect and good judgement alone does not lead to effective performance. One needs to understand an individual’s personal style of behaving and working. Can they work effectively with others? Can they organise their own and others’ work? Are they driven and motivated to make things happen? How do they handle challenging situations? Our Interview Report applies eight generic competencies that works very well for all senior level roles. The competencies covered are:

• Influencing
• Leading Others
• Team Working
• Decision Making
• Change
• Organisation
• Dealing with Pressure
• Motivation

By using all the above information we will get a very good assessment of the candidates’ strengths and weaknesses or development areas for the role. Therefore, this psychometric approach actually provides a much more robust and effective solution than the traditional AC approach. It is highly efficient. What’s more, it is objective and replicable.

The only potential downside is that much of it lacks 2-way interaction i.e. only during the Competence-based interview will there be 2-way interaction. Therefore, organisations need to consider how they can provide candidates with a good candidate experience. But candidates will appreciate the need to be assessed fairly, objectively, and effectively, so they are likely to see the positive side of these assessments being part of their selection procedure.  As the psychometrics are done online it is also extremely efficient and requires no or little resourcing by the organisation.

To Support Development

The Scenarios Tests and the above psychometrics can also be used to support staff development activities. It is obvious they will not replace a Development Centre or Development or Talent Programme as these psychometrics are not interactive or provide the engagement. However, they will provide the useful information to support fruitful discussion and help support with identifying strengths and development areas, and the reports will guide with useful development tips.

So a bit controversial – pack up your high fidelity ACs and replace with low fidelity psychometrics! But it works! Even when Covid 19 is well under control, we will not be returning to the old ‘normal’. Remote working will be the new norm for many people.  Managers will have the challenge of managing their staff remotely from their home. But we are not quite there with running highly effective remote assessment centres for remote workers and remote managers. Let’s save that for a future article when organisations have been doing that for a while and they are getting to understand the challenges better. When they are ready  we are there to  help them.