EnvHarness Lets an LLM Rewrite Your Benchmark, But Never the Grader

Google Research's new Apache-2.0 layer wraps a frozen environment in generated Python so it can teach an agent its own weaknesses. The one thing the generator is forbidden to write is the code that decides whether the agent won.

An LLM writes Python. That Python changes which actions an agent is allowed to take, what it observes when it takes them, and what state it wakes up in. The code is compiled and executed in a subprocess on every episode. It was not reviewed by a human.

And it cannot touch the verifier.

That last sentence is the whole design of EnvHarness, released by Google Research on August 21, the day after arXiv 2608.19880 went up. The paper's framing is neat: an agent harness makes a frozen model capable through plug-in components, so do the same thing to a frozen environment. The engineering underneath it is more interesting than the framing, because somebody sat down and drew a line between the parts of a benchmark a machine may rewrite and the part it may not.

Why static environments became the bottleneck

Agents stopped learning mainly from curated text a while ago. They learn from interacting with environments, and environments are expensive to build. ALFWorld, WebArena, SWE-bench Verified: each one is months of human work encoding tasks, dynamics, and grading.

Once built, they sit still. The README puts the problem in one line: environments "behave identically no matter which agent interacts with them or how much it has improved, so they can neither target a particular agent's weaknesses nor keep teaching once its tasks are solved."

You have felt this if you have ever run the same eval suite against three model versions in a row. The suite tells you the score went up. It tells you nothing about what the agent still cannot do, because it was never designed to probe this agent in particular. The obvious fix is to generate more environments, and the field has tried it, mostly per-domain: SWE-smith for code, GenEnv and VeriEnv for other slices. Each one is its own pipeline with its own grading, and each new generated task brings a new question about whether the grading is any good.

EnvHarness takes the opposite bet. Do not generate environments. Reshape the ones you trust, and keep their graders.

Three components, one interface, one thing off limits

EnvHarness operates strictly at the standard reset and step interface. Nothing in it reaches into environment-specific code, which is why the same machinery works across a text adventure, a browser, and a Docker container running a Python repo.

Three plug-in components do the reshaping. Setup replays a fixed action list on every reset, so the agent starts somewhere other than the task's canonical beginning. Rules hooks each step, changing which actions do what and what the agent observes coming back. Link composes another environment's tasks into the episode.

They stack, and they stack arbitrarily, because an EnvHarness is an ActionableEnv that wraps one. That single recursive definition is why nothing above or below in the stack needs to know how many layers exist.

Here is the part I would put on a slide. The three components "reshape only what the agent observes, what it may do, and where it starts; the goal predicate that decides success is left untouched." A reshaped ALFWorld is still graded by ALFWorld's human-written verifier. So when the paper reports a number, the number came from the original benchmark's own grading, not from a grader the same system invented.

Anyone who has reviewed a self-generated eval knows why that matters. The failure mode of automated environment generation is not bad tasks. It is a generator that produces tasks and a grader that agrees with them, which is a closed loop that reliably reports success. EnvHarness cuts the loop by refusing to hand the generator a pen for that part.

The designer writes real code, and that is a deliberate risk

The loop is driven by an LLM designer agent. It reads the policy's trajectories, diagnoses a specific weakness, writes components targeting it, runs the policy in the reshaped environment, and revises. EnvRigger, in the paper's naming, treats the target policy as a black box and validates every proposal with fresh rollouts.

The designer emits real Python, specifically a _Rules(Rules) subclass, and the README is direct about why: it is "not a selection from a fixed menu." A menu of parameterized knobs would be safer and would also cap what the system can invent.

The safety comes from where that code runs. It "is compiled and executed in an isolated subprocess, so a bad mutation becomes a recorded trace instead of a dead run." One design consequence falls straight out of that: get_env_state() "must carry data, never handles." Put a Docker client or a browser page in there and it breaks, because the generated hooks receive that state across a process boundary.

I keep coming back to this pattern, because it is the same instinct that showed up in the crates.io compromise earlier the same week, where three malicious packages were live for under two hours. A system that writes and runs its own code needs the failure to be recorded rather than silent. EnvHarness treats a bad generated component as a logged trace by construction. That should be table stakes for every self-modifying agent stack shipping right now, and it mostly is not.

What the numbers say, and what they do not

All results are the mean over three independent runs with standard deviations reported, comparing three skill sources: no skills, skills induced from the original environments, and skills induced from EnvHarness-reshaped environments. All skills are evaluated back on the untouched benchmark.

On ALFWorld out-of-distribution tasks, EnvHarness skills score 70.4 against 61.4 for original-environment skills and 60.7 for no skills. That +9.0 gap is the headline, and the error bars (±2.3 versus ±4.3) leave it standing.

On SWE-bench Verified, success rate goes 47.67 (no skills) to 49.88 (original envs) to 52.58 (EnvHarness). Agent steps drop from 55.01 to 49.61, which is where the widely repeated "9.8% fewer steps" figure comes from. It is one benchmark's step count, not a cross-benchmark average, and it is worth knowing that before you quote it.

OfficeQA exact match moves 54.40 to 56.20. SpreadsheetBench Pass@1 moves 45.88 to 49.15.

Now the parts nobody is quoting. On WebArena, the aggregate gain is +3.1, and the per-site numbers are messier than that suggests: on Reddit, EnvHarness scores 40.6 ± 4.7 against 38.7 ± 9.7 for original-environment skills, which is a gap sitting well inside the noise. Induced skills of either kind barely move that site: no skills at all scores 39.6 there. On GitLab, EnvHarness scores 37.7 and loses to VeriEnv's 38.9. The README prints that loss in its own table, in bold, for the competitor. I respect that more than I would respect a cleaner sweep.

And every number in both tables was produced with Gemini, while the shipped configs default to openai/gpt-4.1-mini. The README says so and gives the override. Absolute numbers move with the model; what the tables compare is skill sources at a fixed model.

Put this into practice

The repo went public yesterday and the on-ramp is better than that suggests. Budget an hour for the first two steps.

1. Run the toy environment before anything else. Every benchmark folder follows the same shape:

python scripts/check_env.py <benchmark>           # preflight
bash experiments/<benchmark>/reproduce_smoke.sh   # same stages, fewer tasks
python experiments/<benchmark>/reproduce.py       # the full protocol

Start with experiments/toy24. It exercises the whole pipeline (corpus generation, skill induction, evaluation) without Docker, a browser, or a benchmark download. If the smoke script passes there, your credentials and your config plumbing are correct, and any later failure is about the benchmark rather than about you.

2. Set one env var, not five. Provider selection is a single model string that reaches every stage:

MODEL=gemini/gemini-3.5-flash python experiments/swebench/reproduce.py

MODEL overrides whatever the YAML names for corpus policy, harness agent, skill induction, and evaluation at once, which exists specifically so a run cannot end up split across two providers. Leave it unset only when you deliberately want the policy and the designer agent on different models, which the config supports:

policy:
  model: openai/gpt-4.1-mini
mutator:
  type: llm
  model: vertex_ai/claude-sonnet-4-6

3. Know the three things that will bite you. Concurrency defaults assume a modest quota, and this is the trap worth memorizing: exceeding your tokens-per-minute tier produces 429s that truncate episodes mid-task, "which shows up as unexpectedly low success rates rather than as an error." You will think the method failed. On Gemini, spread workers across several keys with GEMINI_API_KEYS, since that quota meters per key. Second, skill retrieval uses an embedding model paired to your provider, and Bank.retrieve rejects a query vector of a different width, so switching providers means rebuilding the bank. Third, if you want the published tables, point the configs back at Gemini.

4. Only then think about your own benchmark. Adding one means implementing a single interface, ActionableEnv, with seven methods (reset, step, observe, evaluate, get_env_state, save_state, from_state), plus an env_state_schema() string that gets injected verbatim into the designer's prompt. That schema is the only thing telling the designer which fields its generated hooks may read, so vague field names there produce vague components. Copy tests/test_actionable_env_toy24.py, which walks the whole contract including a save/load round-trip and needs no GPU, Docker, or API key.

Where it falls short

This is research code with four commits and no tagged release. It is also moving fast enough that any number I print here is wrong by the time you read it: the repo page served 11 stars and 1 fork when I opened it on the morning of August 22, and 159 stars by that evening. Anything you build on it this month will break.

The license notice is doing more work than the license. Apache-2.0 on the code, and then this at the bottom of the README: "This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program." Read that alongside a system whose core mechanism is generating and executing Python. Google is explicitly declining to run a security bounty on a code-execution surface. The subprocess isolation is a correctness boundary for bad mutations, and treating it as a security boundary is a decision you are making, not one Google made for you.

Extending it past the three shipped components is more work than the docs make it sound. A Candidate carries exactly two levers, rules_code and in_env_actions, so the episode runner composes Setup and Rules and nothing else. Writing your own layer, say for reward shaping or budget accounting, gets you something usable by hand or from a checkpoint. Getting the designer to emit it means extending Candidate, build_env_stack in the orchestration runner, and the propose schema in the harness agent. Three files in three subsystems, in a codebase that went public yesterday.

Checkpointing is honest about being partial. For an in-memory benchmark you snapshot everything. For a container or a browser you store the reset seed and options and let from_state re-run reset, which the README notes is "valid at episode boundaries, which is where checkpoints are taken." Fine for this loop. Not a general resume.

And the memory design is borrowed. EnvHarness adopts ReasoningBank for its skill bank, which is a reasonable choice and also means the skill store inherits whatever is true about that design. Given what MemTrapBench published the same week (every tested memory framework scoring below a no-memory baseline), a skill bank is a component I would want a control run against, not a component I would assume is additive. The paper does report the no-skill baseline, which is exactly that control, and EnvHarness clears it on all five benchmarks. Good. That is the shape of evidence to ask for.

The question this leaves open

Seven other 2026 papers show up in the recommendations attached to this one, all circling the same idea of a runtime that edits itself. When that many independent groups converge, the abstraction is usually real.

What none of them has settled is who reads the generated code. EnvHarness gets the hardest part right by refusing to let the designer near the grader, which means the score you get back is still anchored to something a human wrote. But the environment your agent spent ten thousand episodes inside was authored by a model, revised by a model, and validated by rollouts of a model. If that agent later fails in a way you cannot explain, the explanation may be sitting in a _Rules subclass nobody ever opened.

Clone it, run toy24, and then go read one of the generated components end to end. I would be curious whether you can tell, from the code alone, what weakness it was written to attack.

Sources: google-research/envharness, EnvHarness: Awakening Static Worlds for Agent Learning (arXiv 2608.19880), project page, Hugging Face paper page.