Are Coding Tests Necessary? Three Formats Still Work, Two Waste Everyone's Time

Yes, but only one kind of them, and probably not the kind you are running today. A coding test earns its place when it shows you how a candidate handles a problem they have never seen, in the language they will actually ship in, with you in the room. It stops earning its place the moment the question has a public answer. In a 2024 experiment by interviewing.io, candidates who were given an AI assistant passed 73 percent of verbatim LeetCode-style questions against 53 percent for an unaided control group. On custom, unpublished problems, those same assisted candidates passed just 25 percent. The test did not get harder. It started measuring something real.

This guide ranks the five screening formats US teams actually use, prices one round of each with numbers you can replace with your own, and lays out the 90-minute version that still works when the person on the other side has an assistant open in the next window.

A coding test has exactly one job

The job is to tell you whether this person can turn an unclear request into a working, explainable change in a codebase like yours, at a pace you can afford. That is it. Filtering volume, standardizing the funnel and producing something to show a skeptical exec are all reasonable wants, but they are administration dressed up as evidence, and they are the reason most hiring processes keep a test that stopped telling them anything two years ago.

Hold a format against that one job and the weak ones fall away fast. A timed algorithm puzzle contains nothing unclear: the problem is fully specified before the candidate reads it, so the only open question is whether they remember the trick. A four-hour take-home is unclear in the wrong way, because the candidate resolves every ambiguity alone and you never see which choice they agonized over and which they made in four seconds.

There is a blunt test for your test. If it cannot produce a no for a candidate you were otherwise ready to hire, it is not a screen, it is a ritual. The specific failure it should be catching is the one in our piece on senior on paper, junior in practice: a person who is excellent with a clean ticket and lost without one. No auto-graded exercise has ever caught that, because an auto-graded exercise is a clean ticket.

The experiment that broke the auto-graded screen

The clearest public evidence comes from interviewing.io, which ran mock technical interviews where the candidates were told to use ChatGPT and the interviewers had no idea the study was about AI at all. The questions were split three ways: LeetCode problems used verbatim, the same problems modified, and custom problems written for the experiment.

• Verbatim questions: 73 percent passed, against 53 percent for the unaided control group. A public question measures retrieval, not engineering.
• Modified questions: about 67 percent passed. Editing the wording of a known problem buys you very little, because the underlying shape is still in the training data.
• Custom questions: about 25 percent passed. A problem nobody has published does not merely resist an assistant, it inverts the advantage. Assisted candidates landed well below the control group's 53 percent, though the comparison is not perfectly like for like, since the control was not restricted to custom problems.

Two honest caveats before you build a policy on those numbers. The study ran in 2024 and the assistants have improved since, so treat the gap on verbatim questions as a floor rather than a ceiling. And the candidate pool was deliberately stacked with people comfortable using AI under interview pressure, which is the right design for the question being asked and the wrong one for estimating how often it happens in your pipeline. Assessment vendors publish their own flag rates, some of them alarming, but those vendors also sell detection, so take the direction and leave the decimal.

What survives all the caveats is simple. The expensive part of a coding test is senior review time, and a publicly answerable question spends that time to learn whether the candidate found the answer site.

Five formats, and what each one still predicts

Below are the five screening formats in common use on US engineering teams, scored on what each one tells you now rather than what it told you in 2019. Three still earn their place and two do not. Candidate time is what you are asking a stranger to donate; your time is what the round costs in senior attention.

The code review exercise is the most underused of the five. You hand over a real pull request from your own repository, roughly 200 to 300 lines, and ask what the candidate would block, what they would let through and what they would ask the author. It is fast, it is almost impossible to prepare for, and it surfaces the standards a person carries around, which is exactly what you are buying at senior level. Our walkthrough of how to review a pull request doubles as the grading rubric.

The live pair is the format most teams already own and run badly. The six probes in our guide to running a technical interview cover the mechanics: pick a problem smaller than the job, state the rules of the room, then change a requirement halfway and watch what the candidate does with the cost of the change.

What one round costs, and what skipping it costs

Arguments about screening go in circles until somebody writes the arithmetic down, so here is a worked example rather than a survey. Assume a shortlist of ten candidates and one reviewer whose loaded time is worth $95 an hour, the bottom of the US senior band on our software developer rate breakdown. Replace both inputs with yours and the ranking barely moves.

• Auto-graded screen: 12 minutes per candidate to read results and shortlist. Two hours, $190.
• Four-hour take-home: 30 minutes of real review each. Five hours, $475.
• Code review exercise: 45 minutes live plus nine minutes of notes. Nine hours, $855.
• Live pair on a custom problem: 60 minutes plus 15 of prep and write-up. Twelve and a half hours, $1,188.

The whole spread between the cheapest format and the most useful one is $998. Now price the error it exists to prevent. One month of a senior hire who cannot do the job, at a $160,000 base, is $13,333 in payroll before benefits, equipment or the hours your team spends covering. The best screen on the list costs under nine percent of that single month, and you run it once per shortlist, not once per hire.

There is a second cost that never appears in a spreadsheet. Every unpaid hour you ask for is a filter on availability rather than ability, and it bites hardest on the senior people who already have work and no reason to audition. A 90-minute ceiling on unpaid candidate time is not generosity, it is self-interest.

When you can skip the test entirely

Screening is not free, and there are four situations where running one anyway is theater.

• You have already paid them. A completed contract, however small, outranks any exercise. Nothing you can run in 90 minutes beats a month of shipped work.
• The work is public and attributable. Not a GitHub profile with green squares, but commits you can read, in a project you can run, where the person argued with a maintainer in a thread and you can see how they held the argument.
• The role is reversible. Short engagement, small blast radius, output visible inside a week. Hire, watch the first pull request, decide. That is cheaper and more accurate than any test.
• Somebody competent already screened them and stands behind it. A vetted marketplace or a staffing partner that replaces the person at its own cost has priced the risk for you. That guarantee is the product. Ours is described on our technical interview services page, and it exists precisely so clients can stop running first-round screens.

Three situations run the other way, where skipping the test is the expensive choice: the first engineer on a product, anyone touching money or health data, and anyone senior enough that the patterns they write will be copied by everyone hired after them. In those three, run the full round even when the candidate comes recommended.

The 90-minute screen that survives an AI assistant

Build it from your own repository, not from a question bank. The whole point is that the problem has never been published, which is the one property the 2024 numbers say actually matters.

• Start from a real pull request merged in the last quarter. Strip it to 200 to 300 lines and keep the part where something went wrong.
• Put back the two problems you actually hit, the ones that cost your team an afternoon. Real bugs have a texture that invented ones never do.
• Tell the candidate the assistant is allowed and should be open. Banning it is both unenforceable and a lie about the job, since they will use one on day one.
• Change one requirement at minute 60 and ask what the change costs. This is the part no assistant can answer for them, because the answer lives in the code they just wrote.

Grade the conversation, not the diff. Three signals carry almost all of the predictive weight: the questions asked before any code is written, whether the candidate verifies what the assistant handed back instead of pasting it, and what happens to their tone when the requirement moves. A candidate who notices that the new rule breaks something they wrote at minute 20, and can say out loud what fixing it will cost, has told you more than any score out of 100.

Where to start this week

Open whatever screen you are running today and ask one question of it: when did it last produce a no for somebody you wanted to hire, and a yes for somebody you almost passed on? If neither has happened in a year, the round is a cost center.

Then do three things. Pull a real merged pull request from last quarter and strip it into a 200-line exercise. Cut any unpaid candidate task over 90 minutes, and pay for anything longer. Tell your next five candidates the assistant is welcome, and write down what you learn from watching them use it, because that is the skill you are hiring for now.

If you would rather not build the round at all, that is a legitimate answer too. A vetted marketplace runs the technical screen before you ever see a profile and carries the replacement risk if the match is wrong, which turns a $1,188 round and three weeks of calendar into a shortlist you can interview for fit.