Opus-level results at 43% of the price: pay the cloud for judgment, not labour

Open-weight models on owned hardware cost nothing per call. Claude Opus is billed per token and is better at almost everything. The obvious plan: let the local model do the work, and call Opus only when the local model is about to get something wrong. Cloud quality, mostly at local prices.

The whole plan hangs on one question: how do you know when the local model is wrong? Get that right and you pay for Opus only on the handful of tasks that need it. Get it wrong in one direction and you've paid for the cloud on work the free model had already done. Get it wrong in the other and you've shipped the free model's mistakes.

It was measured over a few weeks, on six benchmarks, climbing from multiple-choice trivia to real GitHub issues, with twenty different ways of splitting the work. Each round sharpened the question, and the last one settled it.

The setup

Every setup in this post is built from the same three parts.

The worker goes first on everything. The judge then decides, one task at a time, whether to escalate: hand that task to Opus and pay. That single decision is the whole subject of the post.

Here is one task going through it. A maths word problem comes in. The worker, a 7B Qwen model, answers 42. The judge, a 3.8B Phi model, is shown the problem and the answer and asked one question, "Is this answer correct?", and says yes with only 30% confidence. That is doubt enough, so the problem goes to Opus, which answers 48, the right answer, and bills about half a cent. Had Phi said 95%, the 42 would have stood, right or wrong, at no cost.

How a judge is scored. A judge is good if, when you sort the worker's answers by its doubt, the wrong ones come out on top. The score used runs from 0.5 to 1.0. Pick one wrong answer and one right answer at random: the score is the chance the judge is more doubtful about the wrong one. A coin flip scores 0.5. A judge that ranks every wrong answer above every right one scores 1.0.

How a setup is scored. Accuracy, or issues fixed, against what was spent on Opus. Spend is written as a share of what Opus-on-everything costs, so the worker alone sits at 0% and Opus alone at 100%. Sending a random share of tasks to Opus gets you the straight line between those two points. A setup is worth having only if it lands above that line.

The tasks climb from cheap to real:

Qwen2.5-7B is a general chat model. Qwen3.8-27B is a popular local coding model, and on the last row it runs as a plain command-line agent: it reads files, edits them, runs tests and submits a patch when it thinks it is done, up to 100 steps.

The judges come in three kinds:

• Small ordinary models. Phi-4-mini (3.8B), Llama-3.2-3B and Llama-3.1-8B, asked "Is this correct?" and read for their yes/no probability.
• The worker itself, two ways: the confidence it had while answering, which is free because it's already there, or shown its own answer and asked the same question.
• A decision model. Clef is not a chat model. It is a 27B model built to take a situation plus a yes/no question and return a calibrated probability, nothing else. Cloudflare hosts it. It turns up in every section.

The cloud is Claude Opus 5.5 throughout, with Claude Haiku, its cheap sibling, tried once near the end. Every cost in the post is what they billed.

1. Who should decide "this answer is probably wrong"?

The local model is decent and Opus is better: 71.5% against 93.0% on trivia, 91.5% against 98.5% on maths. Opus on all 1,200 questions costs about $4. Four free ways of spotting the local model's mistakes were tried:

To compare them: what share of questions do you have to send to Opus before the combined system is as accurate as Opus alone? Lower is better.

Among the ordinary models, the smallest was the best judge, and the worker judging itself was the worst. A model checking its own work seems to share the blind spot that caused the mistake. The decision model beat all of them by a mile. On maths it was 33% with Clef against 52% for the best ordinary judge.

Even 44% isn't as bad as it sounds. "As accurate as Opus" is only a yardstick: a judge that doubts answers that were actually right sends them to Opus for nothing, and has to send many of those before the last few wrong ones get caught. The yardstick is also beatable, because sometimes the local model is right and Opus is wrong (8 of 600 trivia questions). A perfect judge would send 21.5% and score 94.3%, above Opus. Clef gets some of that: 93.2% on trivia for 68% of the money, 98.7% on maths for 62%. The margin over Opus is tiny. The saving is not.

Then code was tried, and the ordinary-model result flipped. New worker, the 27B coding model (92.6% on HumanEval+ and 76.4% on MBPP+, against Opus's 95.1% and 89.4%):

The small judge was a coin flip, and worse than random once money was involved: on MBPP+ with half of Opus's money, the worker's self-check reached 86.7%, random picks 82.9%, the 3B judge 81.4%. On trivia the judge was half the worker's size. Here it's a ninth, and judging code means reading code, which a 27B coding model does far better than a 3B generalist. Even the decision model, so strong on trivia, is a little behind the worker here.

So "small and different" was never the rule. The judge has to be capable enough to understand the work, and built to give honest probabilities. A decision model the size of the worker is the best default found here. For code, add the worker's self-check. And it's cheap: all three Clef models together judged about 8,000 answers for well under a dollar.

2. Don't set a cutoff. Set a budget and pace it

Once you can rank answers by doubt, the question becomes what's the best accuracy for a given amount of money? The simplest rule: sort every local answer from most doubtful to least, send them to Opus in that order until the money runs out, keep the rest. With Phi-4-mini judging:

On maths, half the money gets you within 0.2 points of Opus.

That table cheats, though. It sorts all 600 questions first, which you can only do if you have them all up front. Real work arrives one task at a time, and you decide right then whether to pay. So 1,050 questions were replayed as a stream, trivia and maths mixed, with a hard budget of 25% of Opus-for-everything, and two ways to decide were compared:

The fixed threshold left 71% of its money unspent. Its cutoff was tuned on trivia, and the judge is far more confident about maths, so on the maths half of the stream it almost never fired. The money was there, and the rule had no way of noticing. The pacer spent the whole budget and landed within a point of perfect hindsight, and it held up in four different stream orders, including one that switches from all trivia to all maths halfway through.

On code the fixed threshold failed the opposite way. Tuned on HumanEval+, where the worker is confident, it escalated too eagerly on MBPP+ and ran out of money two-thirds of the way through the stream, every time. The pacer scored 85.8–86.0% in every order, against 86.1% for hindsight. It can't hoard and it can't over-spend early, because all it watches is the bill.

Two refinements that didn't help: preferring cheap questions (most doubt per dollar) was 0.3–0.8 points worse than the plain pacer, and a classic online-knapsack rule under-spent like the threshold. Keep it simple.

3. Watching an agent work

Multiple choice is a toy. The work worth splitting is multi-step: browsing, coding, using tools. So the next stop was the customer-service agent of τ²-bench. Qwen3-32B passed 15 of 40 tasks, Qwen3-8B passed 7. Plenty of room for help.

Before any Opus was paid for, it was checked whether free warning signs predict a failing run. "Free" means read off the run itself, without asking any model: did a tool call return an error, did the agent repeat the exact same call. Then each recorded run was stopped halfway and three-quarters through, and two small models were asked, "Is the agent off track?"

• Repeated calls catch the weak model (0.82 for the 8B agent): when it's lost, it loops. The strong model fails quietly. It rarely loops, and tool errors show up in its good runs too.
• Readers help, but modestly. Nothing got much past 0.7. Later is easier: failing runs look more like failure the longer they go.
• The raw numbers lie, the ranking doesn't. Phi said "off track" with 45–65% confidence on average, Qwen3-8B with 6–13%, for the same runs. Neither number means anything on its own, but both rank failing runs above passing ones, which is all a pacer needs.

(The simulated customer here is a local 32B, not the stronger model the official leaderboard uses, so these pass rates can't be compared with published ones.)

The overseer, live on real bugs

Now the real test: a live agent fixing the 50 real GitHub issues of SWE-bench Verified mini, each checked by the project's own tests.

The overseer is everything above, applied mid-run:

1. The local model works as usual.
2. From its third step on, before every step, a gate looks at the run. Two free checks first: an exact repeated command, or three failed commands in a row. Then a judge reads the issue plus the last ten messages and answers "Is the agent on track?" as a probability. One version uses Llama-3.2-3B for this, the other Clef.
3. A pacer turns that into a decision. It ranks the score against every score seen so far in the batch, works out what share of the remaining checks it can still afford to escalate, and escalates only when this one falls in that worst share.
4. On escalation, Opus takes over the same conversation for 3 steps, then hands back.
5. The budget for the batch was 25% of what Opus-only cost on the same issues.

That looks like a clean win: nine more issues than local for a fifth of Opus's cost. Then it was checked which of the Clef overseer's 11 wins over local actually involved Opus doing anything:

Most of the gain was luck. The local model, run again on the same issue, changed its outcome on roughly one issue in four, even at temperature 0: hosted inference isn't perfectly repeatable, and a 50-step agent amplifies tiny differences. Three local-only runs fixed 29, 29 and 34. Both overseers got 3–4 genuine wins from Opus.

Two things follow from that. A single run per setup can't rank setups on 50 tasks: differences of two or three issues are noise, and every table below should be read with that in mind. And count wins by cause: "fixed because Opus stepped in" and "fixed anyway" have to be logged separately, or the overseer gets credit for luck.

4. Don't watch the work. Check the result

Judging a run in progress is hard because a run that's going badly often looks like one that's going fine, until it ends. So the order was flipped. Let the local model finish every task, then ask one question about the finished patch, "Is this a correct and complete fix for the issue?", and let Opus redo only the doubtful ones.

That's 0.89, against 0.6–0.7 for watching mid-run. Judging finished work is much easier than judging work in progress. Then the money question, with Opus redoing the most doubtful tasks until the budget runs out:

45 of 50 for half of Opus's cost, against 48 for Opus alone and 29 for local alone. It's a replay (real local runs and real Opus redos, combined after the fact), but it was the first setup that clearly beat the straight line between local and Opus.

5. Who should go first?

Both setups so far let the local model start and brought Opus in later. There's a mirror image: let Opus go first. It can go first in two very different ways, and they came out on opposite ends of the table.

Scout first: local explores, Opus finishes. The local model takes 10 steps (find the files, reproduce the bug), then Opus continues in the same conversation. It fixed 48, as many as Opus alone, and every win involved Opus. But it cost 110% of Opus alone: the local model finished inside its 10 steps on only 6 issues, and picking up a conversation that already holds 10 steps of someone else's work costs more than starting fresh. Handing over a short brief instead of the transcript (the files found, how to reproduce the bug, what was tried) brought it to 46 for 98%. A brief clearly beats forwarding a transcript, but here it only broke even, because the cloud agent was already lean at about 9¢ an issue. There wasn't much searching left to save.

Plan first: Opus thinks, local builds. The reverse:

1. Opus reads the code, without changing anything, and writes a short plan: which file, which function, the exact change, and how to test it.
2. A fresh local model gets the issue plus the plan, nothing else, and does all the editing and testing.
3. Opus never touches the code.

A plan for a Django bug about two-digit years in HTTP dates looked like this:

File: django/utils/http.py, function parse_http_date. Change: replace the hard-coded rule ("under 70 means 20xx, otherwise 19xx") with this code [about ten lines, given in full], which reads a two-digit year as the past if it would otherwise land more than 50 years in the future. Check: parse two sample dates and expect 1994 and 2070, then run tests/runtests.py utils_tests.test_http.

Why it works: thinking is the cheap part of an agent. The expensive part is the long tail of editing, running tests, reading errors and trying again. Here Opus does only the thinking, in a few short steps, and the free model does the long tail. It was run with Opus capped at 8 reading steps and with no cap. Opus never used more than 4 either way, so the gap between 44 and 47 is run-to-run noise. A cousin, where Opus splits the issue into up to three subtasks and reviews the result once, fixed 44 for 40%. Three live runs of "Opus thinks, local types", three results between 44 and 47, all at 40–44% of the cost.

One surprise: checking doesn't stack on planning. The check from section 4 was expected to stack on top. But Clef couldn't tell good plan-built patches from bad ones (0.53, a coin flip, against 0.89 on patches the local model wrote alone). With three failures in 50 there was little to find, and the failures looked as convincing as the successes.

6. Everything else that was tried

A local model and Opus are a team with a big skill gap, and people have been splitting work across gaps like that forever: surgeons and assistants, help desks with tiers, two radiologists reading every scan. The obvious arrangements were turned into setups too. Everything on the same 50 issues, sorted by issues fixed. Cost is the share of what Opus-for-everything cost ($4.35):

A few of them deserve a line each.

• Pair programming was the most expensive setup of all. Opus looked over the local model's shoulder every 5 steps, 298 reviews in total. Twice the cost of Opus doing everything, and fewer fixed. Watching costs more than working.
• The junior never asked. Allowed to ask Opus up to three questions per issue, the local model asked twice in 50 issues. It doesn't know when it's stuck, so you get the local model alone.
• Three tries barely beat one. Clef picking the best of three local patches: 33, against 30.7 for picking one at random. Even a perfect picker would reach only 36. Most issues the local model fails, it fails every time.
• A middle tier helps a little, at best. Claude Haiku alone fixed 33 for 3% of Opus's price. Putting it between local and Opus, with Clef scoring the finished patch at each tier and escalating below a cutoff, did 1–4 issues better than local-then-Opus at the same price. But both cutoffs were chosen after the fact on these same 50 issues, so the real gain is probably smaller.
• A second reader added nothing. Clef and the local model disagreed on only 8 of 50 patches, so the Opus tie-breaker had little to do. Clef alone, given the same money, also fixed 46.
• Seeing the work beats predicting it. Guessing from the issue text alone was decent (0.81), but fixed 42 against 45 for checking the finished patch at the same price.
• The surgical team had nothing to do. Only 7 of 50 issues produced a long output for the local model to summarize. It saved 16%.

Judgment, not labour

Sort that table by what Opus was actually paid to do, and one pattern covers all of it:

The cloud model is worth paying for its judgment, not its labour. Where Opus thought for a few steps and left, the free model did the rest nearly as well as Opus does it, for under half the price. Where Opus stayed to work or to watch, the bill climbed back to Opus's own, or past it, because the long tail of an agent's work (edit, run, read the error, retry) is the expensive part no matter who does the thinking. And where the decision to call Opus was left to the run itself, mid-flight, it barely got called at all: a struggling run looks fine from the inside.

What should be built

For coding:

For work without a plan, quick answers or tasks where planning isn't worth it:

Leave out: a cloud model watching every few steps, letting the local model decide when to ask for help, and running the local model several times. If the cloud model ever has to take over mid-task, hand it a short brief, never the transcript.

The scorecard

Every question asked along the way, and what the data said:

The short version

A handful of models and six benchmarks don't make a law. But for anyone splitting work between a model on their desk and one in a datacenter, four defaults are the place to start. Pay the cloud model for judgment, not labour: a short plan from the expensive model makes the free one far more capable, and keeping it around afterwards mostly adds cost. Let the local model finish, then check the result, rather than trying to spot trouble along the way. Let whoever best understands the work do the checking: a decision model for simple answers, the worker itself for code. And let a paced budget decide how much doubt you can afford.

A plug

Does your local model live on a Mac at home while you work from somewhere else? Do you want to watch the local agent build, see the moment Claude is called, and nudge either of them from your phone or any browser? Every run in this post was driven that way, from a Mac in another room. Try reminal.