Psychometrically brilliant, psychologically flawed

March 6, 2014

stethoscopeDoctor, doctor

First, imagine the scenario. You’re lying in a hospital bed. The doctor comes to see you. You are given a physical examination and asked a series of probing questions. A stethoscope is applied to your chest and to your back; and your temperature and blood pressure charts are scrutinized. Some blood tests are ordered and you’re sent for a scan. Later that day the doctor returns and provides a diagnosis.

The doctor has used the information in a particular way. A number of hypotheses about what may be wrong with you have been considered. Along the way, individual pieces of information may or may not have been useful; indeed may not have been relevant at all, or completely misleading. However the final diagnosis is based on a skilled integration of all the available information. It has been filtered through the doctor’s general understanding of how the body works, plus expert knowledge of a range of conditions and illnesses.

Now, a second scenario: You are an HR manager wanting to recruit a senior manager. You have an up-to-date job description and have compiled a thorough specification of the person you think is required to do the job well. You decide to use a variety of methods to gather information about the candidates. These include asking about their track record, and any particular knowledge and skills that are relevant to the job. You also decide to use psychometric tests to measure specific abilities, and a personality questionnaire to explore how candidates are likely to behave; plus an interview to gather evidence on key competencies, and on things like commitment and motivation.

The right blend

The blending of assessment information requires the same skills as those used by the doctor. The information from an interview depends on the candidate telling the truth, as does anything that is said about previous work history – although to a certain extent this can be verified by previous employers. The results of psychometric tests are of most utility if they are used to measure abilities that are directly relevant to the job – albeit that having a handle on a candidate’s general mental ability has been shown to be one of the best predictors of work performance.

Personality questionnaires fall into a different category, as despite sophisticated designs they are open to a candidate putting a positive spin on their answers. However, that being said, trait-based questionnaires can accurately predict behaviour, and as such they are genuinely useful in providing prompts for interview questions.

Of course no process is one hundred percent predictive, and there will be occasions when the wrong selection decision is made. This is inevitable, but a well designed process, using a balanced range of exercises, will significantly increase the odds of picking a winner. Indeed the odds will be increased to maximum if the person dealing with the information is appropriately trained and experienced: like a business psychologist. This is where the true value is added, especially when it comes to deftly combining information that ranges from the robust and objective, to other sources which may be completely ‘self-report’.

It doesn’t work (or does it!)

Criticisms when things go wrong are often of a binary nature, especially in the Press. For example, if tests or questionnaires have been used (journalists unfailingly lump them together), and the ‘wrong’ candidate is perceived to have been recruited, then it is all too easy to blame the tools. The argument is often that the person sailed through the psychometrics but turned out to be a psychological liability. Thus the answer to the – do ‘tests’ work? question – is obviously a resounding ‘no’. The problem with this logic is that it ignores everything else that has happened, including the due diligence that should have occurred before candidates completed any tests or questionnaires.  But let’s face it, poking fun at psychometrics makes a better story, and provides a golden opportunity to use a picture of an inkblot.

In reality psychometrics are the most predictive tools that are available, certainly many times better than merely relying on an interview. So it’s something of a paradox that the more senior the position the less likely the candidates are to be assessed fully. It’s a bit like going to hospital, announcing that you are the CEO of a bank, and the doctor sitting on the end of the bed, asking a couple of questions, and making an instant diagnosis. No need for all those irritating tests, I can tell what’s wrong with you by enquiring about your golf handicap.

To be serious again, perhaps this is where the attention should be focused, on getting the recruitment process for the ‘top jobs’ properly sorted out: and the case is actually made for more assessment, not less.

The research: Do tests work?

A dating agency for job hunters!

February 20, 2012

Looking for a job on the web is set to get more like using a dating site as reliable psychometrics are used to sift candidates… Keep an eye on this stuff as it’s a growing trend! And my thanks to Angus McDonald for drawing my intention to this article:

What are candidate assessment measures measuring?

September 8, 2011

Do candidates who are better at predicting what employers are looking for – who have a well-developed ‘ability to identify criteria’ – actually do better at assessment events? Yes… in fact research seems to show that this ability correlates more strongly with job performance than assessment centre scores themselves!

Read this BPS Research Digest article for the full story: What exactly are candidate selection measures measuring?

How do you weigh an elephant without using a weighing machine?

May 26, 2011

Isn’t it refreshing to know that people are still being asked ridiculous interview questions. Follow this link to find some real crackers:

And a couple of my personal favourites: ‘When was the last time you cried?’ and ‘Do you prefer cats or dogs?’ In the first one you are damned if you do and damned if you don’t. Admit to blubbing on a regular basis and you’ll be labelled as wet and over-sensitive, and of course if you don’t cry at all you’re just  a hard-nosed and uncaring sort of person. Cats or dogs? Well, it depends on which the interviewer prefers. Try and spot the pet hairs on his/her clothing and see if that provides a clue. Joking apart, I thought we had got over all this stuff – interview questions should relate directly, unambiguously and fairly to the job in question.

Picture credit: Tom Curtis/

The Apprentice selection process

May 20, 2011

The new season of The Apprentice is now firmly established on BBC1. However wouldn’t it be interesting to put the ‘top entrepreneurs’ through a rigorous assessment process before the series, to see who has what it takes, and then compare the eventual winner with their assessment profile? In the meantime here’s how the candidates are selected :)

(c) BBC 2011

Cats with thumbs

May 6, 2011

When I first went to University I studied Biology and I well remember being asked in some interview or other how you could breed a dog that could climb trees. Seriously. And in response I think I waffled on about the need for it to develop opposable thumbs or to grow claws like grappling hooks. I’m sure you get the idea. Anyhow this is a long winded way of introducing a great and funny advert about… cats with thumbs. Nothing to do with business, I just like it!

Google job mania

February 4, 2011

You think you’ve got problems! Google recently received 75,000 job applications in a single week. OK, they reckon to have 6000 vacancies coming up, but that’s still heck of a number. Why the recruitment drive? They’ve decided to take on both Facebook and Apple. Not sure who my money would be on, but I’m surprised they didn’t apply their considerable brain power to designing a better recruitment process. This one was surely crying out for rigorous ‘self-selection’, right at the start.

Check out this Bloomberg article.

Do psychometric tests work?

February 3, 2011

Good question. What people usually mean when they ask if they work is: do tests predict anything useful about future work performance? The short answer is a resounding ‘yes’. As long as a test is used to measure an ability that is actually required of a particular job, then predictive validities are often in the 0.5-0.6 range. What this means is that at the top end of the scale, a test (the predictor) explains 36% ((0.6 x 0.6) x100) of the variance in the criterion – the criterion being something like a measure of productivity. By way of contrast other assessment methods such as the interview are often far less effective. A semi-structured interview would weigh in at 0.38 (14%) or thereabouts. And to get the whole thing in perspective, just in case you’re not impressed, in other fields such as the drug industry, predictive-type validities are often lower. For example, the association between Ibuprofen (the well-known anti-inflammatory) and pain reduction is in the region of 0.14 (2%) – see Robert Hogan’s article, details below.

Want to know more, here are some key references:

  • Bertua, C., Anderson, N., and Salgado, J.F. (2005). The Predictive Validity of Cognitive Ability Tests: A UK Meta-Analysis. Journal of Occupational and Organizational Psychology, 78(3), 387-410.
  • Hogan, R. (2005). In Defense of Personality Measurement: New Wine in Old Whiners. Human Performance, 18, 331-341.
  • Hunter, J.E, & Hunter, R.F. (1984). Validity and Utility of Alternative Predictors of Job Performance. Psychological Bulletin, 96, 72-98.
  • Schmidt, F.L, & Hunter, J.E. (1998). The Validity and Utility of Selection Methods in Personnel Psychology: Practical Implications of 85 Years of Research Findings. Psychological Bulletin, 124, 262-274.

No work experience, no graduate job!

January 19, 2011

High Fliers Research conduct an annual survey of UK graduate job prospects. This year, whilst vacancies are set to rise, recruiters report that one-third of this year’s positions will be filled by graduates who they already know – either through work placements, internships or sponsorships. In addition two-thirds of employers say that graduates with no work experience at all are very unlikely to get through the selection process, and thus have little or no chance of receiving a job offer.

This is sobering stuff and highlights the need for all undergraduates to seek out placements or acquire meaningful work experience. It also means that many will need to work on their interview technique, and to practice psychometric and situational judgement tests (see my SJT post). All of these assessment methods are becoming increasingly popular for selecting placement students and interns.

Brains and a ‘winning’ personality? Now that would be dangerous!

November 23, 2010

If you were only allowed to look for one thing in a prospective manager, what would it be? Business psychology tells us that it should be ‘brains’, or rather general intelligence, or if you want to be precise, the ‘fluid’ bit of general intelligence. That’s the sort of intelligence that helps you to solve problems you haven’t come across before. Why? Of all the things we could assess, general intelligence – or having sufficient brain power – is the most predictive of work performance. Now if you could pick a second thing (this is getting a bit like the three wishes granted by a genie, I admit), what would that be? Again if we go with what’s most predictive we would have select ‘conscientiousness’ – the personality attribute that’s associated with self-organisation, discipline, thoroughness and a need to achieve. It also happens to be the best predictor across all types of work. And if a third choice was available? Then it would have to be emotional stability: being positive, calm and relaxed and able to take what comes your way.

Do I need to pick anything else? Obviously knowledge and previous experience come into the frame, and it might also be useful to have a sociable (extrovert) manager, and maybe one who was open to new ideas, who was concerned for others, and honest, with a touch of insight… Stop. Actually we’ve already got the top three and we’ve known what they are for at least the last 20 years.

However it’s not quite that simple. Here’s the thing: whilst general intelligence and conscientiousness are both predictive of success at work, they do not correlate with each other; indeed some people have found a negative relationship between the two. What’s going on?  As you can probably imagine this has been the subject of much debate. One of the ideas is that a negative relationship is due to fluid intelligence affecting the development of conscientiousness. This has the snappy title of ‘intelligence compensation theory’ and it goes like this: fluid intelligence, being innate, is the most likely to influence a growing personality; and to cut to the chase, what then happens is that those with less intelligence compensate by developing higher levels of conscientiousness – and vice versa for those with higher intelligence. Well, it ties in with the statistics, but as you can imagine it’s rather controversial. Mind you it does help to explain the bright individual who flies by the seat of their pants (low conscientiousness) and who nevertheless tends to get away with it.

But getting back to our prospective manager, perhaps there’s a less esoteric explanation. Conscientiousness is a mix of different attributes, which usually include dutifulness and deliberation on the one hand, and achievement orientation and competence on the other. Thus it’s likely that the dutiful plodder aspects of this personality factor are negatively associated with intelligence; and the achieving, competent, striving bits are positively associated. So could it just be that we’ve been looking at personality at the wrong level?

So here’s the punch line -  looking for general intelligence, consciousness and emotional stability is still good advice, but don’t be surprised if those with brains can look like riskier bets because they sometimes get lower overall conscientiousness scores. You’re going to have to dig deeper to find out who you’re really dealing with!

Note: If you’re worried about the other bit of general intelligence, the learnt or ‘crystallised’ aspect, there’s an ongoing argument about whether that is or isn’t related to conscientiousness, and in what combination (or not) with fluid intelligence. Let alone those that think intelligence is part of personality. I expect you get the idea.

Barrick, M.R. & Mount, M.K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1-26.

Moutafi, J., Furnham, A. & Paltiel, L. (2004). Why is conscientiousness negatively correlated with intelligence? Personality and Individual Differences, 37(5), 1013-1022.

Schmidt, F.L. & Hunter, J.E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.  Psychological Bulletin, 124, 262-274.

Photo credit: Francesco Marino/


Get every new post delivered to your Inbox.

Join 85 other followers