JevAgentHarness: 81 to 89% cheaper agents, but a decider with no threshold cost 19 points

Between October 13 and 16, 2026, three pieces of research landed on "System One" decision models: an open 35B MoE decider, a decision-native architecture from NUS, and JevAgentHarness from UIUC, the first paper I've seen that hands these models control of a running agent and measures what breaks. TypeSafe's Jev 1.14.0 also shipped on October 14.

The vendor launches are in the decision-model market wave; this piece is about the evidence. The headline result is that small deciders made agents 81.1 to 89.0% cheaper. The more useful result is that delegating tool choices without a fitted threshold cut success by 19.0 points.

What actually shipped

JEV-35B (AutoTrust AI Lab). An Apache 2.0 fine-tune of Qwen3.5-35B-A3B: 35B total parameters, about 3B active. It answers in one pass through a 264-slot decision head and returns choice, score or noul probabilities. Its self-reported Decision Index 0.3 score is 59.73, against 53.64 for JEV-27B-VL. The card lists weak spots honestly: GPQA 0.360, HLE 0.000, and computer use falling from 95% with element text to 38% with numbered boxes only, where 24 of 37 failures were declaring the task done too early. Median latency is 204 ms on one B200; NVFP4 fits a 32 GB GPU.

Jeva / Jeva-4B (Qi Li, Xinchao Wang; NUS; arXiv 2610.18773). A decision-native architecture. The request and each option are encoded separately, a small decision stack (about 13.4M parameters on Qwen3.5-4B with LoRA) refines only the request, and aggregation is symmetric. It has three properties existing deciders lack. The first is permutation invariance: zero flips across 600 Choice examples under reordered options, where Jev-1.13.0 flipped 17.00% of the time. The second is explicit null detection, with 98.66% AUROC on a BANKING77 answer-absent set, against 79.44% for JevK5. The third is cacheable option encodings: p50 latency on MLDR fell from 2,102 ms to 75 ms with no decision changes.

It trails InternDecision-4B on Typed Decisions (78.30 vs 78.70), and accuracy falls from 86.90% to 69.88% as candidates grow from 32 to 2,048. I found no license for the weights.

JevAgentHarness (Zhu et al., UIUC; arXiv 2610.21412). A proxy that sends agent tool choices to a decision model and escalates to the LLM below a per-decision-point confidence threshold. It comes with AgentDecisionBench, which has three levels: isolated question, recorded step, and live task control. It also includes AgentDecision-4B, trained on states an agent reaches after its own mistakes, with Jev's answers as labels.

Numbers that matter

From the abstract and repo: the decider with the best accuracy on isolated questions solved only 69.5% of agent tasks, so benchmark accuracy didn't predict in-agent results. Delegating tool choices with no fitted threshold cost 19.0 points of success (−19.0). On tasks unlike the training data, AgentDecision-4B solved 91.5% and Jev 82.0%. On tasks like the training data, routing was 81.1 to 89.0% cheaper and up to 2.5x faster, losing fewer than 3 points.

The coding results are more modest. The setup used Gemini 4 Argon as the strong tier, Gemini 3.5 Flash-Lite as the fast tier, and Jev 1.13 as the decider, over 74 held-out Terminal-Bench 2.0 tasks:

Illustrative arithmetic (repo's mini-swe-agent row). $0.91 − $0.33 = $0.58 saved per task; at 10,000 tasks a month, $5,800 for 2.7 points of success, about 270 extra failures. At 20 human minutes per failure (my assumption) that's 90 engineer-hours. Your failure cost decides, not your token bill.

Delegating tool choices without a fitted threshold cut success by 19.0 points.

Two caveats from the authors deserve more prominence than the headline. Savings depend on the price gap: with Gemini 3.8 Flash as the fast tier, about 2.7x cheaper than the strong model, savings fell to 7 to 13%. And about half of steps stayed on the strong model in every run.

The AgentDecision models carry a CC BY-NC 4.0 license and were trained on customer-service states, not coding traces.

How to evaluate a decider on your own traces

The 69.5% result means a leaderboard can't answer this question for you. Here's what I'd measure, in order. Start by extracting decision points from logged traces: state, option set, frontier choice, and whether the outcome proved it wrong. The frontier choice is a label, not ground truth. Next, plot accuracy vs coverage by sweeping the threshold; one accuracy number hides the only trade-off you control. Then check calibration (Brier or ECE) per decision point. Strands' ≥0.9-confidence-is-95%-right claim holds only for short classification tasks.

After that, run a permutation test: shuffle the option order and count flips. A 17% flip rate is a bug in your router, not noise. Include null cases, states where no option is correct. Oversample post-error states, the steps after a failed tool call, where off-the-shelf deciders have no training signal. Last comes an end-to-end A/B on live tasks.

A minimal harness

def decide(point, state, options):
    if point in ALWAYS_STRONG:            # first step, "task done?"
        return strong_llm(state, options)
    try:
        r = decider(state, questions=[{"type": "choice", "options": options}])
    except DeciderError:
        log(point, "decision_error"); return strong_llm(state, options)
    p = max(r.probabilities)
    log_shadow(point, state, r)           # for refitting
    if p >= THRESHOLD[point] and not after_tool_error(state):
        return r.choice
    return strong_llm(state, options)

Fit one threshold per decision point from shadow logs (the lowest whose accepted decisions match the strong model on outcome), freeze it, validate on fresh tasks: the paper's two-stage protocol. Keep the first step and the completion decision on the strong model, as JevAgentHarness does. JEV-35B's top computer-use failure was declaring done too early, which shows why. Refit thresholds when the decider, prompts or tool set change. Verification is the other half of this loop; see verification is the new bottleneck and the threshold discipline in RRSI's regularized harness.

What I'd do

I'd run shadow mode for two weeks before any control. I'd gate per decision point, starting with tool and argument selection, which passed the paper's stage 2. I wouldn't delegate stop/done decisions. And I'd expect little saving if the fast tier is within about 3x of the strong model's price.

What I couldn't verify

The arXiv PDF of 2610.21412 was rate-limited. I relied on the abstract, HF page, project page and README, and couldn't confirm which setting produced the 81.1 to 89.0% figure beyond "familiar tasks." I suspect it's DecisionGym customer service, but that's a guess. Neither the JEV-35B release date nor the Jeva-4B license is stated. All JEV-35B scores are self-reported, partly on the authors' own rebuilds of public datasets. Jev 1.14.0 is real (Oct 14, jev-preview, new box type), but its charts give no numbers in text.

I'll revisit once AgentDecisionBench has third-party submissions on coding traces.

Sources

Primary: Hugging Face: autotrust/JEV-35B; Hugging Face Papers: Jeva (2610.18773); arXiv PDF: Jeva (2610.18773); Hugging Face Papers: JevAgentHarness (2610.21412); arXiv: JevAgentHarness (2610.21412); JevAgentHarness project page; GitHub: JevAgentHarness/JevAgentHarness. Related: TypeSafe blog: Jev 1.14; GitHub: strands-labs/strands-decider.