Method · Assessment design

Four ways to assess an executive.

Most firms run one process on every search, because one process is what they have. The design is a decision, it is made before sourcing begins, and it is driven by the role rather than by the firm’s habits. Here are the four we use, what the published evidence says about each, and the conditions that decide which one you get.

The short versionThe evidenceThe four designsChoosing oneCombining the evidenceBy sector

Reference material. The short version is immediately below; everything after it is the reasoning behind it, and you do not need it in order to hire us.

The short version

If you read nothing else on this page.

Every search starts with a scorecard: the outcomes the role has to produce in its first eighteen months, agreed in writing before anyone is approached.

After that there are four ways to gather evidence against it. Most searches use one as the spine and a second in support.

Which one a role calls for is the first real decision in any search, and we make it with you at kickoff — not after sourcing has already started.

What actually predicts performance

Start with the uncomfortable numbers.

In 2022 a team of selection researchers re-ran the meta-analyses this field has quoted for twenty-five years and found that a standard correction had been applied too aggressively for decades. Nearly every estimate came down, some of them by half. The table below is the corrected picture, with the figures the industry still cites beside it.

We publish this because it undercuts a great deal of what gets said in this business, including things we would find it convenient to say.

MethodCorrected (2022)Previously citedWhat it means in practice
Structured interview.42.51The best-supported method in the literature, and the only widely used one that rose to the top of the ranking when the estimates were corrected.
Job knowledge test.40.48Strong, but it measures what someone knows now — not what they will decide under pressure.
Biodata (empirically keyed).38.35One of the few methods whose estimate did not fall. Rarely practical at executive level.
Work sample.33.54Fell further than almost anything else. Still the strongest ground to stand on if a decision is challenged.
Cognitive ability.31.51The largest single revision in the re-analysis. For experienced executives, job-specific evidence predicts better.
Integrity test.31.41Useful as a supporting signal; not a basis for a decision.
Assessment center.29.37Expensive, well-liked by candidates, and a more modest predictor than the industry quotes.
Situational judgment test.26—Not estimated in the 1998 table. Cheap to administer, moderate signal.
Conscientiousness.19.31The most-cited personality predictor, roughly halved.
Unstructured interview.19.38The industry default. Less than half the signal of the structured version of the same conversation.
Years of experience.07.18Close to no signal at all — and the thing most job specifications are written around.
Read these honestly

These are correlations, estimated across populations of jobs. A validity of .42 is the best available evidence in a field where nothing approaches certainty — it is not a probability that a given hire works out. The practical consequence is not that assessment is futile. It is that the gap between the best method and the ordinary one is large, nearly free to close, and almost nobody closes it. Our entire argument rests on that gap and on nothing more dramatic.

The four designs

Each answers a different question. None of them answers all of them.

Most searches use one as the spine and a second in support. The scorecard is common to all four — what changes is how evidence against it is gathered.

A

Track record

The default

A scorecard agreed before anyone is approached. A structured interview run identically for every candidate against behaviorally anchored definitions of each attribute. A reference protocol that asks about those same attributes, of people positioned to have observed them, with questions built to surface disconfirming evidence.

What it produces

A record of what the candidate has actually done in conditions comparable to the ones they are being hired into — and what it cost them to do it.

Use it when

  • The role resembles work the candidate has already done at comparable scale or complexity.
  • A track record exists that can be interrogated rather than merely recited.
  • Reference access is normal and the search is not confidential.

Do not use it when

On its own, when the job is a genuine step change — plant to enterprise, founder-led to sponsor-owned, clinician to operator. There is no comparable past behavior to examine, so the method returns a confident answer to the wrong question.

What the evidence says

Structured interviews estimate at .42 operational validity and rank first among widely used methods. The unstructured version of the same conversation estimates at .19.

Cost and burden. Lowest. Included in every engagement we run.

B

Live exercise

When the résumé cannot answer the question

Build the hardest recurring decision of the job and have the candidate make it. An operating review on sanitized real data. A capital request defended under questioning. A ninety-day plan presented to the people who would have to execute it. The content comes from a job analysis, not from a catalogue.

What it produces

Observed behavior in the job being hired for, rather than reported behavior in a previous one.

Use it when

  • The job differs materially from anything on the résumé.
  • Two benches have to be compared on one standard — post-close integration, or an internal candidate against an external one.
  • The decision will have to be defended to a board, a regulator or a court.

Do not use it when

When the pool is thin, because the burden drives scarce candidates away. When the search is confidential or the process is unannounced. When the timeline is compressed — this design is the slowest.

What the evidence says

Work samples estimate at .33 and assessment centers at .29, both revised down sharply from the figures still quoted in this industry. Simulations are not magic. What they are is content-valid: built from a job analysis, they measure the work itself, which is the strongest position to be in if the decision is ever challenged.

Cost and burden. Highest. Scoped and priced per engagement.

C

Staged screening

When the pool is large

Order the assessment so the cheap, highly eliminative screens run first and the expensive ones run only on survivors. Gating facts first — clearance level and currency, licensure, geography, the compensation at which the market will actually clear. Then scorecard screening, then structured interview, then whatever depth the role justifies.

What it produces

The same quality of final assessment at a fraction of the cost, when the population is large.

Use it when

  • The addressable population is large relative to the number of hires.
  • A single gating fact disqualifies most of the market — the common case in cleared and licensed roles.
  • Budget or timeline will not support depth across the whole field.

Do not use it when

When the pool is already small. Sequencing a field of nine people adds process and removes nothing.

What the evidence says

A sequence is not a shortcut. Every screen you add is another point at which someone is cut, and every one of those needs a reason you could defend later. We set them on facts that genuinely gate the job — a clearance level, a licence, a location — never on proxies for ability.

Cost and burden. Lowest per candidate at volume.

D

Panel assessment

When the failure mode is judgment, not skill

Multiple assessors, multiple occasions, multiple methods, integrated against a defined attribute set rather than by one assessor in one sitting. Each assessor rates independently against anchored definitions before the group convenes.

What it produces

Coverage of what a single interview cannot reach: consistency across occasions, behavior when the assessor is not the audience, and the judgment a candidate exercises when nothing appears to be under examination.

Use it when

  • The decision is irreversible, or expensive enough that unwinding it is a material event.
  • The dominant failure mode is judgment, candor or temperament rather than skill or knowledge.
  • Several stakeholders have to own the outcome, and will only own a decision they watched being made.

Do not use it when

When speed matters more than certainty, and when the board would be assembled from people with no basis for judging the attributes in question. A panel of senior people is not the same thing as a panel of qualified assessors.

What the evidence says

Assessing a person across several assessors and several occasions is long-established practice in government and military selection, not a proprietary technique, and we do not represent it as one. What makes this design work or fail is not the panel — it is how the panel combines what it gathered.

Cost and burden. Moderate to high, driven by assessor time.

Choosing one

The conditions that decide it.

These are the questions we work through at scoping, before a role is sourced. Most searches land on a primary design within the first conversation, and we will tell you which one and why.

Condition at scoping ATrack
record		 BLive
exercise		 CStaged
screening		 DPanel
assessment
The candidate has done materially similar work before●··○
The role is a step change in scope or operating model○●·○
The addressable pool is large relative to the number of hires○·●·
One gating fact eliminates most of the market○·●·
The search is confidential or the process is unannounced●—·○
Two benches must be compared on a single standard○●·○
The dominant failure mode is judgment, candor or temperament○··●
The decision is irreversible or very expensive to unwind○○·●
The decision will have to be formally defended○●⚠·
The timeline is compressed●—○·

● Primary design    ○ Supporting    ⚠ Use with caution    — Not appropriate

The part almost everyone gets backwards

Collect richly. Combine mechanically.

A panel that gathers in a room and talks until it agrees feels like rigor. It is usually the weakest step in the whole process.

The best available meta-analysis compared two ways of putting predictor data together: applying agreed weights consistently, against expert judgment integrating the same information holistically. For job performance, the mechanical composite estimated at .44. Expert holistic judgment estimated at .28 — a gap the authors describe as an improvement of more than fifty percent in favour of the mechanical composite. The experts in those studies knew the jobs, knew the organizations, and frequently held more information than the formula did. They still predicted worse.

So: every assessor rates independently against the same anchored definitions. Ratings are recorded before the group convenes. The composite is computed, not negotiated. Discussion exists to surface evidence somebody else missed — not to arrive at a number. A board that talks itself out of its own ratings has quietly reverted to the method that performs worse, and will feel more confident for having done it.

This is the single most consequential item on this page. It costs nothing to implement and it is the step most commonly skipped, because the alternative feels like judgment and this feels like paperwork.

By sector

Where each design tends to land.

Tendencies, not rules. A specific role can override every one of these, and frequently does.

Aerospace & DefenseHurdles first, then behavioral evidence. Clearance and program experience gate the field hard; simulations burn a pool that is already small.
Security Clearance RolesHurdles, unambiguously. Level and currency are gating facts, and the honest version of the search establishes them before sourcing rather than after a slate disappoints.
ManufacturingThe plant-to-enterprise step up is the textbook case for simulation. The résumé cannot distinguish a leader who ran a system from one who built it.
ConstructionBehavioral evidence works unusually well here. EMR, surety history and bid records are independent, checkable, third-party signals — rare in any sector.
HealthcareSimulation plus a whole-person board. A clinician moving into operations is making a career change, and the decision is documented and reviewable.
Engineering & TechnologyBehavioral evidence aimed at decision quality rather than architecture, with a board. A board that cannot judge the technology can still judge the reasoning.
Physical & Cyber SecurityWhole-person board. The job fails on organizational candor — saying no to revenue, reporting honestly to people who want reassurance — not on technical depth.
Waste & EnvironmentalBehavioral evidence, with simulation where an acquisition thesis assumes a step change in integration pace the incumbent has never been asked to deliver.

Limits and sources

What this does not claim.

No design on this page predicts performance with certainty, and the published evidence says plainly that none comes close. Structured selection improves substantially on unstructured judgment and improves considerably on chance. That is the whole claim. The remainder of whether an executive succeeds belongs to the organization they land in, the support they are given, and events no assessment can see from outside.

Everything we conclude is written down against a scorecard agreed in advance, and applied the same way to every candidate. If a decision is ever challenged, that is the record your counsel will want. We are not employment counsel and nothing here is legal advice; where a process carries real exposure, your counsel should see it before it runs.

Validity figures are drawn from Sackett, Zhang, Berry & Lievens (2022), Revisiting meta-analytic estimates of validity in personnel selection, Journal of Applied Psychology 107(11), with the prior figures from Schmidt & Hunter (1998) shown for comparison. Combination figures are from Kuncel, Klieger, Connelly & Ones (2013), Mechanical versus clinical data combination in selection and admissions decisions, Journal of Applied Psychology 98(6). Attribute-based assessment draws on RAND, Assessment and Selection for U.S. Air Force Special Warfare (RR-A549-1, 2022). Published military selection research informs how we think about multi-method assessment; we draw only on what is publicly available, and we do not reproduce or represent any non-public selection content.

Celer Silens Partners

Start a conversation

Tell me the role and what makes it hard.

Twenty minutes. I will tell you which of these four I would use and why, whether the market you want exists at the compensation you have, and what I would do differently. If it is not a fit, I will say so and point you somewhere better.

jj@celersilens.com 704-740-2099