Psychometrically brilliant, psychologically flawed

March 6, 2014

stethoscopeDoctor, doctor

First, imagine the scenario. You’re lying in a hospital bed. The doctor comes to see you. You are given a physical examination and asked a series of probing questions. A stethoscope is applied to your chest and to your back; and your temperature and blood pressure charts are scrutinized. Some blood tests are ordered and you’re sent for a scan. Later that day the doctor returns and provides a diagnosis.

The doctor has used the information in a particular way. A number of hypotheses about what may be wrong with you have been considered. Along the way, individual pieces of information may or may not have been useful; indeed may not have been relevant at all, or completely misleading. However the final diagnosis is based on a skilled integration of all the available information. It has been filtered through the doctor’s general understanding of how the body works, plus expert knowledge of a range of conditions and illnesses.

Now, a second scenario: You are an HR manager wanting to recruit a senior manager. You have an up-to-date job description and have compiled a thorough specification of the person you think is required to do the job well. You decide to use a variety of methods to gather information about the candidates. These include asking about their track record, and any particular knowledge and skills that are relevant to the job. You also decide to use psychometric tests to measure specific abilities, and a personality questionnaire to explore how candidates are likely to behave; plus an interview to gather evidence on key competencies, and on things like commitment and motivation.

The right blend

The blending of assessment information requires the same skills as those used by the doctor. The information from an interview depends on the candidate telling the truth, as does anything that is said about previous work history – although to a certain extent this can be verified by previous employers. The results of psychometric tests are of most utility if they are used to measure abilities that are directly relevant to the job – albeit that having a handle on a candidate’s general mental ability has been shown to be one of the best predictors of work performance.

Personality questionnaires fall into a different category, as despite sophisticated designs they are open to a candidate putting a positive spin on their answers. However, that being said, trait-based questionnaires can accurately predict behaviour, and as such they are genuinely useful in providing prompts for interview questions.

Of course no process is one hundred percent predictive, and there will be occasions when the wrong selection decision is made. This is inevitable, but a well designed process, using a balanced range of exercises, will significantly increase the odds of picking a winner. Indeed the odds will be increased to maximum if the person dealing with the information is appropriately trained and experienced: like a business psychologist. This is where the true value is added, especially when it comes to deftly combining information that ranges from the robust and objective, to other sources which may be completely ‘self-report’.

It doesn’t work (or does it!)

Criticisms when things go wrong are often of a binary nature, especially in the Press. For example, if tests or questionnaires have been used (journalists unfailingly lump them together), and the ‘wrong’ candidate is perceived to have been recruited, then it is all too easy to blame the tools. The argument is often that the person sailed through the psychometrics but turned out to be a psychological liability. Thus the answer to the – do ‘tests’ work? question – is obviously a resounding ‘no’. The problem with this logic is that it ignores everything else that has happened, including the due diligence that should have occurred before candidates completed any tests or questionnaires.  But let’s face it, poking fun at psychometrics makes a better story, and provides a golden opportunity to use a picture of an inkblot.

In reality psychometrics are the most predictive tools that are available, certainly many times better than merely relying on an interview. So it’s something of a paradox that the more senior the position the less likely the candidates are to be assessed fully. It’s a bit like going to hospital, announcing that you are the CEO of a bank, and the doctor sitting on the end of the bed, asking a couple of questions, and making an instant diagnosis. No need for all those irritating tests, I can tell what’s wrong with you by enquiring about your golf handicap.

To be serious again, perhaps this is where the attention should be focused, on getting the recruitment process for the ‘top jobs’ properly sorted out: and the case is actually made for more assessment, not less.

The research: Do tests work?

Stop being such a cow!

October 14, 2011

Picture the scene: you’re walking down the street trying to decide where to eat. But it’s early evening and all the restaurants are empty. So you pick what you consider to be the most appealing – you think it has the tastiest menu – and go in. After a while some more people come along and see you sitting at a table. They assume that if you’re in there it must be OK, and enter and sit at another table. Very soon there are lots of people in the restaurant and all the other possible eateries are lagging well behind. This process is called the information cascade.

Information cascades are important in economic psychology as they are used to explain the behaviour of financial markets. That’s because they feed the process of speculation, including frantic buying or selling: bubbles v crashes. Now of course all this starts at a vaguely rational level (remember the restaurant example) but rapidly moves into irrational herd-like behaviour. Thus maybe it’s no accident that we talk about confident or ‘bull’ markets! Although perhaps in reality it’s more cow-like: put your head down, eat the grass and stick with the herd.

The bigger point is that information cascades affect all manner of decision-making. And the movement of the ‘herd’ (group, team, board) can be very difficult to resist, even if you, the individual, recognise it as being irrational. People en masse are easily convinced they are moving in the right direction, and frequently do not stop to ask the obvious question: why are we doing this?

At work this is made worse by the fact that many managers do not know how to probe the thinking of their colleagues. They simply base their understanding on what they assume is going on, and assumptions, as we all know, are often vague and unpredictable beasts. The real trick is to return to the beginning and pose clearly framed questions about the logic of projects and plans, and also to ask questions about the organisation. Not just, ‘why are we doing this?’ but ‘why are we doing it this way?’ So maybe the competency lists that are so beloved of HR professionals should always include the ‘ability to ask good ‘why’ questions’ behaviour…


What are candidate assessment measures measuring?

September 8, 2011

Do candidates who are better at predicting what employers are looking for – who have a well-developed ‘ability to identify criteria’ – actually do better at assessment events? Yes… in fact research seems to show that this ability correlates more strongly with job performance than assessment centre scores themselves!

Read this BPS Research Digest article for the full story: What exactly are candidate selection measures measuring?

How do you weigh an elephant without using a weighing machine?

May 26, 2011

Isn’t it refreshing to know that people are still being asked ridiculous interview questions. Follow this link to find some real crackers:

And a couple of my personal favourites: ‘When was the last time you cried?’ and ‘Do you prefer cats or dogs?’ In the first one you are damned if you do and damned if you don’t. Admit to blubbing on a regular basis and you’ll be labelled as wet and over-sensitive, and of course if you don’t cry at all you’re just  a hard-nosed and uncaring sort of person. Cats or dogs? Well, it depends on which the interviewer prefers. Try and spot the pet hairs on his/her clothing and see if that provides a clue. Joking apart, I thought we had got over all this stuff – interview questions should relate directly, unambiguously and fairly to the job in question.

Picture credit: Tom Curtis/

Do situational judgement tests work?

May 25, 2011

Situational judgement tests (SJTs), or tests that assess a candidates ‘preferred’ responses to a range of work-based scenarios, are growing in popularity and are now commonplace in many graduate and management recruitment processes. But do they work? Well, it seems there’s pretty good evidence that SJTs do predict job related criteria such as sales performance or ratings by managers. The first really thorough analysis, conducted by McDaniel et al (2001) across 95 different studies, concluding that the correlation between SJTs and job performance is in the region of 0.34. Incidentally McDaniel also found that when SJTs were closely matched to the job in question – via a properly conducted job analysis – the figure rose to 0.38. 

The same figure was reported earlier this year by SHL Group, with a composite of 0.38 being achieved for a ‘relating & networking’ criterion and one of their SJTs which is being used by a global retailing organisation.

In addition various studies have looked at whether SJTs significantly add to the prediction of job performance over and above that which is achieved by using measures of cognitive ability (psychometric reasoning tests), job experience and personality. Again McDaniel et al (2007) have found that SJTs provide incremental validity over cognitive ability of between 3 and 5 per cent, i.e. they add something extra to an understanding of ‘thinking’ competencies; and of 6 to 7 per cent compared to personality questionnaires, i.e. they add even more to an understanding of how someone deploys their personality at work.

P.S. In the great scheme of things 0.3, which is a ‘moderate’ correlation, is the point at which things are starting to get particularly useful, especially if the assessment method in question is being used for volume recruitment.

Want to know more?

McDaniel, M.A. and Nguyen, N.T. (2001). Situational Judgment Tests: A Review of Practice and Constructs Assessed. International Journal of Selection and Assessment, 9(1-2), 103-113.

McDaniel, M.A., Hartman, N.S., Whetzel, D.L. and Grubb, W.L. (2007). Situational Judgment Tests, Response Instructions and Validity: A Meta-analysis. Personnel Psychology, 60(1), 63-91.

Lievens, F., Peeters, H. and Schollaert, E. (2008). Situational Judgement Tests: A Review of Recent Research. Personnel Review, 37 (4), 426-441.

Psychometrics and strength-based testing

September 12, 2010

If you’re on top of your positive psychology you will know that there’s a movement to look at what people enjoy doing, rather than what they’re good at. Not a bad idea because what you like (to do) and what you do are not necessarily the same thing. And we’ve all met people who are perfectly competent but not performing at their best. Thus it’s all quite reasonable that if someone uses their real strengths at work, those that engage enjoyment and competence, they will actually be at their best, work harder, work smarter etc. If you’re interested you’ll find more information on Martin Seligman’s website:

Talent spotting for beginners

September 10, 2010

RFIf you have ever watched an Olympic swimming race you may have noticed that the best swimmer always seems to be in Lane 4. You won’t find Michael Phelps, for example, starting off in Lane 1 or Lane 8, or indeed any of the other remaining lanes. This is because lanes are assigned by seeding time, with the fastest, by tradition, being placed in Lane 4. And so that’s where we look when the race is underway. Our eyes are drawn to the likely winner.

The same sort of thing happens when we are selecting talented staff. We look for a winner. This means we want everything to be in place, the ‘full package’ to be on display, with all the boxes ticked. Frankly what we are after is someone who is above average; preferably well above average, right across the board.

Mind the spikes

But is this sensible? On the surface it appears to be the most logical thing to do, however really successful people often have ‘spiky’ profiles. It doesn’t matter what you look at, and how many competencies you measure, the real hotshots are exceptional at some things and pretty average at others. Pick any successful entrepreneur or business leader and what you notice is their key strength. Chances are this is what has got them to the top. So it’s not that they don’t have any weaknesses, in this respect they are the same as everyone else, but what they do possess is an area of massive competence.

Back with our swimming analogy, sure the best thing to do is to look in Lane 4 if you’re after a champion swimmer; but if you’re looking for something else, make sure you cover the entire pool. Otherwise you might just miss what you really want. So it is that Richard Branson famously has trouble distinguishing between gross and net profit, and yet runs a multi-billion pound empire; and other business guru’s frequently lack the personal touch, but still manage to inspire us.

This brings us onto another important point. Well two actually. Firstly it’s wise to remember that a spiky profile can be the sign of great potential – but that in a well balanced organisation any weaknesses that come with it can be compensated for by people with complementary strengths. Secondly, that you may ultimately be wasting a great deal of money on training and development! Why? Because if all of this is true you want people to play to their strengths, to reinforce their strengths, not to try and drag an area of comparative weakness up a few notches – but probably never to a level that will make any difference.

In search of excellence

The upshot is that you should go out of your way to attract as broad a range of people to your organisation as you can. But you obviously want those with high levels of self-awareness as they will have the insight to be able to maximise their strengths. This is something on which you cannot compromise, and one way of detecting this ability is to explore a person’s emotional intelligence. Also to be tuned to those factors that might ‘derail’ success. What is it in a person’s character that in uncontrolled excess could lead to disaster?

However the main point is to give everyone the opportunity to display excellence. Spot their distinctive talent and you will discover the key to their potential success. But to do this may take more than just an interview and a few psychometric tests. You’re now in the domain of serious business psychology.


Get every new post delivered to your Inbox.

Join 89 other followers