Self-evolving agent harnesses: one survey with a PAC admission gate, one method with +11.5 points

On 16 October 2026 Self-Evolving Agent Harness (arXiv 2610.18813, v1 posted 13 October) showed up on Hugging Face Papers. It comes with a repository called Harness-RSI. The pitch is recursive self-improvement without touching the weights. The agent rewrites its own prompts, memory, workflows, tools and plugins, and the accepted revisions carry over to later runs. A week earlier, a Google team posted Harness²: Recursive Agent Harnessing for an Open World (arXiv 2610.14224, 9 October), which does that in practice and reports numbers.

Read them together. The first is a 66-page survey and theory paper with no new experiments. The second is a method with benchmark gains and almost no cost reporting. If you maintain an agent harness in production, the useful part is the overlap between them: an explicit admission gate for harness changes.

What actually shipped

Harness-RSI (Wei, Guan, Zhu, Zhou, He, Zhou, Chen, Shen) is a taxonomy and a theory. Interpreted updates change text that a fixed host reads: prompts and instructions, memory and skillbooks, workflow specifications. Executable updates change code the host runs: tools, repository and runtime logic, plugin composition. Search over revisions is framed as evolutionary optimization, and reliability is analysed with continual PAC-learning. Under stated assumptions the authors derive a generalization bound, a retention bound tied to a historical regression budget, and a high-probability admission rule on fresh, independent validation data.

The repo audits 93 primary comparisons from prior work. Its main finding is that many reports never confirm that an accepted update actually runs, or separate its effect from run-to-run variance and extra compute. License: the repo contents are CC BY 4.0. There's no runnable framework.

Harness² (Xu, Chen, CuiZhu, Dalvi Mishra, Ming, Yu, Han, Lee, Pfister) recurses at two levels. At the task level it runs Propose → Probe → Compose: it proposes an invariant edit set and a domain-specific one, runs each candidate harness against a reference run, then a composer keeps only the edits the execution evidence supports. At the harnessing level, lessons are distilled into a shared improvement library (harness files plus an EXPERIENCE.md) that feeds future tasks. Every new task restarts from the base harness H₀.

The editable surface is the system prompt, guardrails, memory, skills, tool guidance, scripts, subagents and plugins. Weights stay frozen. The default editor is Claude Opus 4.8. The executors are Gemini 3.5 Flash and GLM-5, running under OpenCode or Codex.

Numbers that matter

Harness², self-reported, base → one step (k=1):

Going to parallel recursion at k=5 adds roughly another +2 to +7 points per cell. Baselines don't hold up as well: ReasoningBank falls up to 6.6 points below base on coding tasks, and "Harness Scaling" underperforms base in 7 of 12 pairs.

Illustrative cost (my arithmetic, not the paper's): the paper reports steps per task, not tokens or dollars. The pipeline it shows for k=1 is one reference run, two probes and one final run, so 4 executor runs instead of 1, plus Opus 4.8 editor and composer calls. For GLM-5 + OpenCode on WorkBuddy, steps per run are about 13.5 (base) and 13.7 (Harness²). That's roughly 13.5 + 2×13.5 + 13.7 ≈ 54 executor steps per task, against 13.5, so about 4× before counting the frontier-model editor. That buys +11.5 points. k=5 multiplies it again. Whether that's worth it depends entirely on whether the library amortizes across tasks, and the paper doesn't quantify that.

Technical meaning and trade-offs

The weak-model gains are partly a strong-model gain. The biggest jumps come from GLM-5, with Opus 4.8 doing the editing. Some of that is distillation of a frontier model's judgment into the harness. That's useful, but it's not "self"-improvement in the strict sense. The paper mentions same-model editor settings, but I couldn't see those results.

No verifier means no correctness gate. Harness² is explicit that its probes "inform subsequent composition rather than verify task correctness". Its defences are blind selection of anonymized deliverables when k>1, edits tied to concrete observations, and evidence-gated memory. None of these is an explicit regression check. Reward hacking isn't discussed by name.

When your evaluator is weak, the harness learns to satisfy the evaluator.

That's the failure mode I covered in ImpossibleRubrics.

The survey's gate is the missing piece. Accept a candidate only if its estimated loss beats the incumbent's by more than a confidence term, on fresh tasks it never saw while it was being proposed. Repeated rollouts on the same task reduce noise but don't count as independent samples. Retention is bounded by an explicit regression budget. It's the same instinct as RRSI: regularize the self-improvement loop, or it drifts.

Executable updates are a privilege escalation path. An agent that can rewrite its own tools and plugins can widen its own permissions. The survey recommends independent safeguards for learning, runtime privileges and consequential actions. I agree, and I'd enforce them outside the harness the agent can edit.

What I'd do

I'd start by versioning the harness like code. Prompts, skills, tool specs and plugin config go in git, and every agent run records the harness commit hash; without that, nothing below is measurable. Then I'd add an admission gate in CI. Any harness change, whether a human or an agent made it, runs on a held-out task set the proposer never touched, and merges only on improvement beyond noise and zero regressions on a pinned "must not break" split.

I'd split interpreted from executable changes. Agents can propose prompt, memory and skill edits automatically behind the gate. Tool, code and plugin edits need human review, and the agent never gets to edit permissions. Following the survey's audit finding, I'd log activation: check that an accepted update actually fires in the traces before you credit it with a gain.

I'd measure cost per point, tracking tokens and wall-clock alongside score. If one Harness²-style step quadruples executor runs, you want that on the dashboard. And I'd try Harness² on a weak, cheap executor first, because that's where the reported headroom is largest.

What I couldn't verify

All Harness² numbers are self-reported, and I found no independent replication. Harness² doesn't report token, dollar or wall-clock costs; my 4× figure is derived from the illustrated pipeline and the step counts, not measured. I didn't see the same-model editor results, the ablations, or Harness²'s limitations appendix (Appendix F). The 18.6k GitHub stars shown for the Harness² repo on the HF page is unverified and irrelevant to quality. Harness-RSI's bounds rest on assumptions (independent task draws, losses in [0,1], a fixed model and evaluator) that production systems rarely satisfy.

I'll revisit when someone publishes Harness² results with a same-model editor and full cost accounting.

Sources

Primary: Self-Evolving Agent Harness (Hugging Face Papers); arXiv 2610.18813; Harness-RSI repository (zgcagi); Harness²: Recursive Agent Harnessing for an Open World (Hugging Face Papers); Harness² on arXiv (HTML). Project: Harness² project page.