Research Fellow Roger Kline reviews a paper on the impact of joint-evaluation methods in NHS recruitment
In a discussion with a group of NHS staff recently I was asked what their Trust might do to speed up progress on race equality in recruitment and career progression. I shared some of the evidence on removing bias from processes and inserting accountability, and in passing suggested that if the Trust wanted to be adventurous it could remove the future line manager from the final appointment decision in order to reduce affinity bias as there was some evidence this could be effective.
They laughed and said. “Ah, that is just what happened by accident here when the line manager was off sick. The interviews went ahead and we got a much more diverse set of appointments”.
A fascinating short paper by Sheila Cunliffe and Catherine Wilkins suggesting this approach might be more widely effective has just been published. I recommend you read the original paper but with the authors’ permission I summarise it here and suggest why they may have found what they did.
The case study
A high profile NHS Trust identified a number of issues with their existing process for recruiting Band 5 nurses and midwives. 63% of applicants were invited to interview and 12% of those received an unconditional offer. The Trust found:
- Some managers were taking too long to shortlist leading to high non-attendance at interview and too long to return appointment paperwork after interview
- Managers wanted the ‘perfect’ candidate and were unwilling to take someone who needed development.
- Candidates applied for multiple roles and, if invited to interview for several of these, only attended interviews for one.
- There was substantial attrition of candidates after unconditional offer which meant the number of applicants finally onboarded was below 10% of applicants.
The Trust wanted to know why only 12.3% of candidates reached Unconditional Offer stage. A deep dive into 32 past recruitment campaigns was undertaken by the Nursing Workforce team doing an exercise in which candidates who had applied for these roles were shortlisted for both –
- Suitability for the particular role advertised, and
- Suitability for a Registered Nurse (RN) role somewhere in the Trust.
The data produced from this exercise was then compared with the data from the original recruitment exercise. The outcome was quite extraordinary. 89.1% of applicants were assessed as suitable for a RN role somewhere in the Trust, but in the original recruitment many were being rejected at shortlisting stage as they were not considered suitable for the specific role applied for. In addition, others were not considered to be in the top 4-6 candidates for that role and rejected in order to have a ‘manageable’ shortlist. This was then followed by candidates who were unsuccessful for the role on offer being rejected after interview with no attempt to offer them a role elsewhere in the Trust. The cumulative effect was that only 12.3% of candidates reached Unconditional Offer stage.
This exercise led to substantial changes in recruitment processes in Adult Nursing including:
- Reviewing the ‘Killer Questions’ criteria in the recruitment system to ensure only candidates who were professionally qualified and registered (or about to be) could apply
- Stopping shortlisting. Instead all applicants were automatically invited to book into a generic Assessment Centre at a date suitable for them (the centres were run on a 3-weekly basis)
- The Assessment Centres were established on the principle of ‘wrapping our arms around the candidate’, ie with a focus on a more positive candidate experience and giving information about the Trust as a whole, including discussion of potential future career progression
- Applicants attending the Assessment Centre were given a generic interview and Situational Judgement Test. The interviews were conducted by panels who were assessing suitability for a RN role in the Trust as a whole and not for a specific post. All candidates were assessed against a common standard. This future line manager was not always on the panel and even if they were, they were assessing for generic roles rather than simply for ones they would be managing in future
- A Values and Behaviours 10-minute online test developed with a Psychometric company which reported against the Trust Values and Behaviours framework was also trialled, but this was stopped as a result of insufficient HR resource being available for analysis and evaluation
- Individual clinical teams had ‘stalls’ at the Assessment day, enabling candidates to speak with specialists and find out more about their work and then have the opportunity to state their preference for specific areas they wished to work in and/or discuss the areas they were interested in from a career progression perspective
- Successful candidates were then placed according to the Assessment Centre results and their preferences where possible
- A more flexible approach was taken with candidates – eg if a candidate wished to work in a particular specialism but wasn’t considered ready, they could be offered a role which would give them that additional experience in a 6-to-12-month period
- Many candidates received an offer on the day of the Assessment Centre.
The Trust gained an immediate benefit which addressed the original rationale for the exercise. The volumes of applicants recruited improved substantially with 30.3% of all applicants now getting through to unconditional offer compared with 12.3% before these changes were made.
However, when the team designing these changes then reviewed the success rate of applicants of different ethnicities for Band 5 and 6 posts (as part of a separate investigation into bias in recruitment) they discovered something quite astonishing.
They found that whilst the differences by ethnicity in the likelihood of applicants who applied being shortlisted were significant but small, at interview stage the outcomes were very different.
Fig 1 shows the difference between the interview success rates overall, and by ethnicity, before and after the process changes were introduced.
Fig 1. Success rates of different recruitment processes by ethnicity
| Ethnicity | % Success Rate from Application when interviewed for specific Trust job (12 months) | % Success Rate from Application when interviewed for an RN role somewhere in Trust (6 months) | Ratio of White Offers to BAME Offers |
|---|---|---|---|
| Asian | 16.1% | 43.68% | 0.76 |
| Black | 12.9% | 35.15% | 0.94 |
| Mixed | 16.7% | 25.58% | 1.29 |
| Not stated | 18.7% | 73.53% | 0.45 |
| Other | 14.5% | 33.33% | 0.99 |
| White | 24.5% | 33.01% | 1.00 |
The changes recorded in interview outcomes are striking, resulting in the proportions of ethnic minority applicants getting unconditional offers being much closer to the proportion of white candidates at each stage.
Explanations?
Why might the outcomes change with a change of process? There are a number of possible reasons.
One explanation is Bohnet’s insight that joint-evaluation of candidates succeeds in helping employers choose, irrespective of an employee’s gender and the implicit stereotypes the employer may hold. Bohnet found employers tasked to choose an employee for future performance were influenced by the candidate’s gender in separate evaluation. Bohnet’s findings have implications for organizations that want to decrease the likelihood that hiring, promotion, and job assignment decisions will be based on irrelevant criteria triggered by stereotypes. In contrast, in joint-evaluation, gender was found to be irrelevant – employers were significantly more likely to choose the higher rather than the lower performing employee.
They concluded that research in behavioural decision-making suggests that employers may decide differently in joint than in separate evaluation because they switch from a more intuitive evaluation mode based on heuristics in separate evaluation to a more reasoned mode when comparing alternatives in joint-evaluation. In addition, joint-evaluation might also affect choices by providing additional data that employers can use to update their stereotypical beliefs about a group to which an employee belongs. By definition, an employer has more data points available in joint than in separate evaluation. Bohnet found that only about 8 percent of the employers engaging in joint-evaluation, as compared to about 51 percent of the employers engaging in separate evaluation, chose the underperforming employee. It seems quite possible that the same principle might apply to the influence of ethnicity on decision making.
This is one possible explanation for some of the difference the change of process made, though without additional information it is not possible to say how significant this change might have been.
A second possible explanation is suggested by research on whether having more than one candidate who is female or is from an ethnic minority affects their likelihood of being appointed from interview. Johnson and colleagues suggested it makes a very considerable difference. Pooling results from three studies they found that when there was only one woman or minority candidate in a pool of four finalists, their odds of being hired were statistically zero.
However when they created a new status quo among the finalist candidates adding just one more woman or minority candidate, the decision makers did consider hiring a woman or minority candidate. The difference that increasing the number of female or minority candidates made was remarkably large. Why does being the only woman (or ethnic minority person) in a pool of finalists matter? The researchers suggest this is because it highlights how different they are from the norm “and deviating from the norm can be risky for decision makers, as people tend to ostracize people who are different from the group. For women and minorities, having your differences made salient can also lead to inferences of incompetence.”
Clearly in batch recruitment of the sort in the Trust studied, half[SC1] of those shortlisted prior to the change of process were White applicants, so it is possible this was a factor, although it is unlikely to be the main one since substantial numbers of individual shortlists would have had two or more BME candidates (Harvard Business Review).
The third possible explanation, and probably the most significant one, is that the removal of the line manager from the decision making process for specific roles they would manage in future made a difference. Using an assessment centre in which the future line manager was not always present, and even if a manager with a vacancy was present they were they were not necessarily interviewing ‘their candidates’, is likely to have substantially reduced affinity bias. In No more tick boxes I summarised some of the powerful evidence that affinity bias is an important factor in creating biased recruitment and career progression decision. There is evidence that the absence of the future line manager can help to prevent some of that bias. Google follow that approach even though “Managers hate the idea that they can’t hire their own people. Interviewers can’t stand being told that they have to follow a certain format for the interview or for their feedback.” (Wired)
Conclusion
It is likely that a combination of a very structured process, without the future line manager’s decision being key, possibly assisted by either or both the impact of joint evaluation and more diverse shortlists, helped make the remarkable difference, Cunliffe and Wilkins found.
Either way, there appear to be important lessons for other NHS employers where batch recruitment is possible since, when implemented properly, it may make a very significant contribution to diversity as well as the overall effectiveness of recruitment.
The authors would be really interested in other examples of batch recruitment and the use of assessment centres for such recruitment at enquiries@citou.com, and so would I.
Roger Kline is Research Fellow, Middlesex University Business School
If you found this interesting, you might want to read Roger’s blog about No more tick boxes, his extended review of “what works” and what doesn’t in creating fair recruitment and career progression