ALTK-Evolve Lets Your AI Agent Write Its Own Rulebook
IBM's open-source tool turns an agent's past mistakes into short prompt rules. With an older model, it topped a public benchmark. Here's how it works, what it costs, and how to try it.
Most AI agents make the same mistake twice. They forget to check a customer's identity before changing an order, or they confirm a refund before reading the policy, and the next conversation starts from zero with no memory that it ever went wrong.
The usual fix is a bigger model. IBM Research just published numbers suggesting a cheaper one: let the agent write down what it learned, in plain sentences, and paste those sentences into its own instructions.
That tool is ALTK-Evolve, and in an October 20 post on the Hugging Face blog, IBM reports that it took gpt-4.1, a model that's no longer anyone's frontier, to first place on the retail section of a public agent benchmark. Ahead of every frontier entry. No retraining. The only thing that changed was a block of text in the system prompt.
This means the next upgrade to your agent might be a paragraph it wrote about itself.
What ALTK-Evolve actually does
The idea fits in one sentence: run the agent on practice tasks, look at what worked and what didn't, and distill that into short rules.
IBM calls the rules guidelines. Its examples are rules about how to authenticate a customer or how to confirm a transaction: the steps a careful human would never skip. The post describes it this way: Evolve analyzes the agent's attempts on training tasks and uses the model itself to turn successes and failures into rules. Those rules get appended to the prompt for later conversations.
Think of it as the notes a new support hire keeps on a sticky pad during their first month. Nobody retrained their brain. They just stopped making the same three mistakes.
Each guideline also carries a support count: the number of different training tasks it came from. A rule that showed up in five separate tasks is probably a real pattern. A rule from one task might be a fluke. You can keep only the rules with high support, which gives you a shorter, stricter rulebook.
What doesn't change matters as much as what does. IBM says the approach needs no weight updates and no human labeling, and the agent's tools, policy and scaffolding stay stock. Everything Evolve adds is visible text you can read.
The numbers, in plain terms
The test bed is τ²-bench (tau-two bench), a benchmark where an agent handles simulated customer conversations for a retailer or an airline, using tools and following company policy. IBM reports a strict score called Pass^4: the agent has to get the same task right on all four tries. One lucky run doesn't count.
On 40 held-out retail tasks with gpt-4.1:
- Without guidelines, Pass^4 was 57.5%.
- With the full guideline set, it was 67.5%.
That's 10 points, which on 40 tasks means four more tasks solved reliably. Mean accuracy rose from 78.1% to 82.5%.
On the benchmark's full retail split, IBM reports 67.5%, ranked first on the leaderboard snapshot of October 12, above RAFT-30B-A3B, a model fine-tuned on synthetic retail data (61.4%), and above all frontier entries. With gpt-5.5 on the airline section, it reports 82.0%, also first, against 66 to 72% for frontier entries.
Two more details are worth slowing down for.
First, more rules weren't always better. With gpt-4.1 on airline tasks, a compact set of 14 high-support rules lifted Pass^4 from 35% to 45%. The full set reached only 40% on that measure. The best amount of guidance, IBM says, varies by domain.
Second, the rules made conversations a little shorter, not longer. On matched runs, IBM saw guidelines slightly shorten the back-and-forth.
What it costs
The rules aren't free. They ride along in every prompt.
IBM's own benchmark figures (they include the simulated customer on the other side of the chat, so treat them as relative, not as a production price): the full retail guideline set raised cost per conversation from $0.084 to $0.139, a 65% jump. On airline tasks it went from $0.081 to $0.107, up 32%. The post notes that prompt caching can absorb much of that, since the guideline block is the same on every call, though how much depends on your provider's caching rules.
Whether that's a good deal depends on what a failed conversation costs you. If a botched refund means a human cleans it up, five cents is cheap. If you run millions of low-stakes chats, it isn't.
Why this is a shift
For two years the default answer to "my agent keeps getting X wrong" has been one of three things: write a better prompt by hand, switch to a bigger model, or fine-tune. Evolve is a fourth option that sits between the first and the last. It writes the prompt for you, from evidence, and you can still read every line.
The interesting part isn't the benchmark win. It's where the improvement lives. Weight changes are a black box. A guideline block is a text file. You can diff it, review it, and roll it back. That makes learned behavior something you can manage like code.
How this differs from the instruction file you already have
If you use Claude Code or Codex, you probably keep a CLAUDE.md or AGENTS.md file: a list of rules the agent reads at the start of every session. Evolve's guidelines look a lot like that. So why bother?
The difference is where the rules come from. Your instruction file holds the rules you thought of. Most of them get written after something goes wrong and you happen to notice. Evolve writes rules from every run you feed it, including the failures nobody caught, and it tells you how many separate tasks backed each rule. A rule you wrote at 11 p.m. after one bad session has a support count of one. You just never saw the number.
The other difference is retrieval. A CLAUDE.md loads in full every time. Evolve's get_guidelines tool returns the rules relevant to the task in front of the agent, so a rulebook can grow without every rule landing in every prompt.
They work well together. Let Evolve propose rules, then promote the ones that hold up into the file you already maintain by hand. Your instruction file stays short and human-reviewed, and the rules in it come from evidence instead of memory.
One uncomfortable fact stays true either way. The rules are written by the same model that made the mistakes. Nothing in the method guarantees a rule is right; support counts just tell you how often it came up.
Put this into practice
ALTK-Evolve is open source under Apache-2.0. The latest tag is v1.7.1 (October 15, 2026), and it runs as an MCP server, so any agent that speaks MCP (the Model Context Protocol, the standard plug-in format for agent tools) can use it. The README lists quick-start guides for Claude Code, Codex, IBM Bob and Hermes.
1. Install it. You need Python 3.12 or newer. From PyPI:
pip install altk-evolve
Or from source, as the README shows:
git clone https://github.com/agenttoolkit/altk-evolve.git
cd altk-evolve
uv venv --python=3.12 && source .venv/bin/activate
uv sync --package altk-evolve
cd frontend/ui && npm ci && npm run build && cd ../..
The last line builds the web dashboard, so the source route also needs Node and npm.
2. Give it a model. The README's example uses an OpenAI key (export OPENAI_API_KEY=...), or a LiteLLM proxy (an open-source router that puts many model providers behind one OpenAI-style API) for other providers. EVOLVE_MODEL_NAME sets a fallback model.
3. Start the server. From the source checkout:
uv run evolve-mcp
When it runs over stdio (the default, where your agent launches it as a local subprocess), that also serves a web dashboard and REST API at http://127.0.0.1:8000/ui/. In the v1.7.1 release there's no switch to turn that off. Newer, unreleased code on the main branch adds --no-dashboard and an EVOLVE_DASHBOARD=false setting, so check which version you installed before reaching for them.
4. Feed it runs and pull rules back. Two MCP tools do the work. save_trajectory stores a finished agent run (a trajectory is the full record of what the agent saw and did) and generates new guidelines from it. get_guidelines(task) returns the rules relevant to a new task, so your agent can read them before it starts.
5. Start small and read everything. Pick one workflow your agent repeats daily, such as triaging issues or answering a support category. Save 20 or 30 runs. Then open the dashboard and read every guideline before you let the agent use them. Delete the ones that are wrong. Keep the ones you'd have told a new teammate.
6. Measure the cost. Compare token use per run before and after. If it climbs, try keeping only the high-support rules, which is what helped on IBM's airline tasks.
Honest limitations
The test sets are small. IBM's held-out results come from 40 retail and 20 airline tasks. Four tasks is the whole retail gain. The leaderboard figures use the full task set and differ from the held-out numbers, as the post itself notes.
Rule quality varies by model. With gpt-5.5, no guidelines reached a support count of three, so IBM lowered the bar to two. Stronger models may produce fewer repeated mistakes to learn from.
The local API has no login. In v1.7.1 the dashboard and REST API listen on 127.0.0.1 with no authentication, and the current README on main says it outright: "The API has no authentication, so any local process can read and change memories through it." Anything that can change the rules can change your agent's behavior. Run it on a machine you control, and don't expose the port.
It ships telemetry settings you should check. The README says Evolve's OpenTelemetry export (usage traces and metrics) points at localhost:4318 by default; set EVOLVE_OTEL_ENABLED=false if you don't run a collector.
Your runs go to a model provider. Evolve uses the model to turn trajectories into rules, so whatever your agent saw (customer messages, file contents) travels to whichever provider you configured. Pick one your data is allowed to reach.
The default search is plain text matching. Semantic search needs an extra backend (uv sync --extra pgvector or --extra milvus).
It's young. The repo had about 130 stars on shields.io as of October 21, 2026, so expect rough edges and changing APIs.
What you do with this
Evolve won't make a weak model brilliant. What it does is narrower and more useful: it catches the mistakes your agent repeats and writes them down where you can see them.
Pick one workflow your agent runs every day. Feed it a few weeks of real runs. Then read the rulebook it writes. If the rules are things you'd have told a new hire, keep them. If they're nonsense, you've learned something about your agent too.
Sources: IBM Research, "ALTK-Evolve" on the Hugging Face blog (Oct 20, 2026); AgentToolkit/altk-evolve README (reviewed at the v1.7.1 tag and current main, read Oct 21, 2026); LICENSE.