Why AI-assisted hiring loops rank the model instead of the candidate, and how to price the difference
Every hiring process makes one choice, usually without making it deliberately: whether it tests delivery, or tests what a candidate knows and how they think. Ban AI from the room and the loop measures demonstration. Allow it and the loop measures delivery. Most processes inherit their answer from whatever the last company did.
For most of the industry’s history the choice barely mattered, because the two tests pointed at the same person. An engineer who could explain a hash join on a whiteboard could usually also ship the feature; the whiteboard was a demonstration test wearing delivery clothes, and the correlation held well enough that nobody audited it. The long argument between algorithm rounds and take-home projects was an argument about which proxy annoyed candidates less, not about whether the proxies agreed.
AI broke the correlation. A candidate who directs an agent well now delivers things they could not build unaided, and a candidate who whiteboards beautifully may be slower than the agent working alone. The choice between testing delivery and testing thinking is now a real fork, with different hires at the end of each branch.
It is tempting to resolve the fork by fiat: the company is buying results, so test delivery and be done. That position is closer to right than wrong. If someone can command an agentic system to deliver consistently, how they do it does not matter. Decomposing work into pieces an agent can hold, specifying intent precisely enough that output is checkable, verifying what comes back, catching the model when it is confidently wrong: this is not a workaround for engineering skill but the current form of it. Nobody now asks whether an engineer can write assembly.
The Word Doing All the Work
The load sits on one word: consistently. Consistency is a property observed over time and across conditions, and hiring happens before the employer has either.
An interview yields one sample, on a well-scoped problem, in familiar territory for the model. That is precisely where agents differentiate least. A mediocre engineer with an agent and a good one with an agent both ship the tidy CRUD exercise inside the hour; the difference between them is real, but it is off-stage. It appears at the edges: the legacy system with no documentation, the requirement that contradicts itself, the incident at 2am when the agent loops, the security-sensitive change where a passing test suite is not the bar. Survival at those edges is decided by judgment, verification habit, and a working model of what the agent cannot see.
An interview built around a completable task samples none of this. It samples the region where the agent carries any competent operator to the same destination, then applies a rubric built for a world where the destination was the differentiator. A loop that ends in a tie among agent-assisted finalists has not failed to score. It has measured the agent, accurately, once per candidate.
Pricing the Instrument
The economics explain why this matters more than a process nicety. A conventional loop costs 15 to 25 interviewer-hours per candidate, roughly $1,700 to $2,800 at loaded senior rates, and $15,000 to $20,000 per hire across a funnel of eight finalists. The error the loop exists to prevent costs two orders of magnitude more. A mis-hire detected at month nine has consumed most of a year of loaded compensation plus the review time of everyone around them; standard models put the all-in cost at 1.5 to 2 times annual compensation, $250,000 to $350,000 for a senior role. The interview is a $20,000 instrument protecting against a $300,000 error, and its discriminating power has just collapsed in the region where most loops operate.
That asymmetry pays for better instruments. A one-week paid work sample at contractor rates costs about $6,000 per candidate; trialling three finalists costs under $20,000, a fraction of one mis-hire. The case has also been run at full scale: replacing interview funnels with structured trials produces 71% more successful hires per candidate. A work sample measures the thing itself rather than a proxy: real codebase, real ambiguity, real colleagues, delivery observed rather than inferred. A verifiable track record does the same job for free where it exists, meaning checkable artefacts and references who watched the delivery happen, not the CV claim.
Trials carry honest costs of their own. Employed candidates cannot take a week off to audition, so the trial favours people with slack, which is a bias with legal and fairness edges of its own. The compromise is compression: a paid, time-boxed exercise of an evening or a weekend, scoped to fit around a job, priced at a few hundred dollars. Smaller sample, same principle. Where the employer can observe delivery directly, at any scale, the philosophical question dissolves; judge the output.
Where no trial is practical, the interview remains. And since an interview cannot observe consistency, it should probe how the candidate thinks. Not because thinking is a virtue. Because it is the only leading indicator of consistency an interview can produce.
Designing for the Last 20%
Allowing AI in the room is the right call; banning it tests a skill the role no longer exercises daily. The design flaw in most loops is the task, not the tool. A task an agent can complete is a task that measures the agent.
The fix is to build exercises where the model carries a candidate 80% of the way and the remaining 20% requires judgment. Pareto’s principle survives the tooling change in inverted form: the agent now supplies 80% of the work, and the remaining 20% carries all of the signal. The brief contains an ambiguity that forces a clarifying question. The codebase contains a trap the agent will step into. Somewhere in the flow sits a plausible but wrong suggestion, planted deliberately, because production plants them daily. None of this is adversarial theatre; it is the job, compressed.
Scoring then shifts from the artefact to behaviour. Whether the candidate tests. Whether they read the diff before accepting it. Whether they push back on the model, or on the interviewer. Whether they know when to stop prompting and read the code. These behaviours are cheap to observe and hard to fake for two hours, which is the property a discriminating instrument needs.
The soft-skills round comes free. How a candidate asks questions, narrates trade-offs and explains decisions while working is the communication assessment, observed in flight rather than performed in a separate hour of rehearsed stories. Teams that score it this way can usually drop a round from the loop and recover some of the interviewer-hours the new design costs.
A short round without AI still earns a place, for the same reason pilots still fly some hours without autopilot: fundamentals decay silently, and a company wants to know the floor. It should be the minority of the signal, not the spine of the loop.
The Calibration Test
A company can audit its own loop in one afternoon. Run the next interview with the 80/20 design and score only the last 20%: the clarifying questions, the trap, the planted suggestion, the verification behaviour. Then score the same session the old way, on the artefact alone. If the two rankings differ, the loop has been measuring the agent, and every hire it has ranked since the tools arrived deserves a second look.
Delivery is what the company is buying. Judgment is how it forecasts delivery. The instrument should be chosen by which of the two it can measure, and priced against the $300,000 error it exists to prevent. Most loops today are precise instruments pointed at the wrong object, returning clean, confident readings of the model everyone in the funnel is using.