Self-improving systems / verifier quality
The feedback is
the research.
A technical note on what we are reading, what the current Rust implementation actually provides, and the experiment we want to run next.
Source boundary Upstream: SIA paper + Rust port. Proposed here: verifier holdout and Goodhart-gap instrument.
01 / Starting point
SIA turns improvement into a loop.
SIA: Self-Improving AI with Harness & Weight Updates describes a system with three roles: a Meta-Agent writes or revises a task harness, a Target-Agent runs the task, and a Feedback-Agent reads the resulting trajectory and chooses the next change. The paper studies two levers: changing the harness around a model, or updating model weights. Both are judged by the same verifier.
That shared verifier is the useful pressure point. The paper names a possible “coupled co-evolutionary Goodhart” failure: the loop can become strong on the score it sees while becoming fragile under a changed evaluation. The paper identifies the risk; the question remains empirical.
02 / The Rust port
A faithful base for inspection.
The SIA Rust port is an active engineering implementation of that framework. Its deterministic core, run format, context handling, and orchestration preserve the task contract while generated target agents continue to run through a Python subprocess. Optional native Rust runners cover several LLM paths, and the repository includes an Axum runs visualizer.
The repository’s own paper draft is careful about scope: its evidence is byte-parity testing, deterministic microbenchmarks, and offline tests. It does not claim a live end-to-end self-improvement accuracy result. That distinction is part of the research surface.
03 / Open question
When a score rises, what improved?
A higher training score can mean a better harness, a better model, or a strategy that learned the evaluator’s surface. We want to measure those possibilities separately. The first test is deliberately narrow: does an improving SIA loop retain its gains when the verifier sees disjoint examples and a controlled perturbation?
04 / Proposed instrument
Expose the wedge before trusting the curve.
Our proposed SuperRadiant instrument keeps the upstream training loop intact, then evaluates each generation against two views:
The verifier the loop can optimize against.
A disjoint and perturbed verifier the loop never sees.
The Goodhart gap is score(Vtrain) − score(Vholdout). A widening gap is a signal to inspect, not proof of failure. The planned interface would show both curves, log the alert, and make any intervention legible. Rotating the holdout or using a verifier ensemble are proposed mitigations; this page does not claim those guardrails or results exist yet.