AKV Lets AI Agents Forget Old Tool Output Instead of Summarizing It

A new paper and repo drop stale tool results from the model's memory directly. The authors report 27.7% lower average cost and a slightly higher average score.

Long-running agents have a strange way of saving money. When the conversation gets too long, they pay a model to rewrite their own history into a summary, then pay again to read it back in from scratch. A new paper says the cheaper move is to let the agent forget.

Most of what gets summarized is junk the agent stopped needing hours ago: a search result it already used, a directory listing from step 3, a 40-line stack trace it already fixed. The usual tools treat that pile as text to compress, and compressing it breaks the one thing that makes long contexts affordable.

A paper called "Cache Engineering for Agents", submitted October 20 and among the top daily papers on Hugging Face Papers on October 22, proposes something blunter. Do not rewrite the history. Let the agent tell the server to forget specific pieces of it. The code is public as thu-nics/AKV, and in the authors' tests a simple rule (keep the latest 12 tool responses, forget the rest) scored slightly higher than summarization while costing 27.7% less on average.

The shift is small to say and large in practice. Agent memory has been treated as a writing problem. AKV treats it as a deletion problem.

First, what the KV cache is

When a language model reads your prompt, it turns every token into internal numbers it will need again for the next token. Those numbers are the KV cache (short for key-value cache). Think of it as the model's working notes on everything it has read so far.

Building those notes is called prefill, and it is the expensive part of reading a long context. Serving systems save money by reusing the notes. If your next request starts with exactly the same tokens as the last one, the server reuses the cached notes for that shared beginning (the prefix) and only processes what is new. That is why hosted APIs charge less for cached input.

Here is the catch. The reuse only works while the beginning stays identical. Change anything early in the conversation and every token after it has to be prefilled again.

Why summarizing history is expensive

Now look at what a typical agent harness does when its context fills up. It takes the older turns, asks a model to summarize them, and replaces them with the summary. The paper's baseline does exactly this, triggering a summary at a fixed size such as 96,000 tokens.

That edit lands near the start of the conversation, so the server's cached notes for everything after it are useless. The agent pays for the summary call, then pays to re-read the whole retained context. The paper puts it simply: the earlier the edit, the more re-prefilling you pay for.

Summaries also lose things. A model deciding what matters in a log it no longer understands will sometimes throw away the line you needed.

What AKV does instead

AKV keeps the text of the conversation exactly as it was. Instead of editing the words, it edits the cache. The paper defines three operations:

• Append: the normal case, add new tokens.
• Drop: exclude chosen messages' cached notes from all future attention, so the model stops "seeing" them, while every other cached note stays valid and reusable.
• Reposition (repos): close the gaps left by dropped messages by giving the remaining tokens new, compact position numbers.

An analogy helps. Summarization is like retyping a long document in shorter form and then reading it again from page one. Dropping is like taking a black marker to the pages you no longer need. The rest of the document stays where it is, and you do not reread it.

There is a hard part, and it is why this needs a server change rather than a prompt trick. After drops, the same text can sit on top of different cache states depending on which messages were dropped and when. The paper calls this history dependence. AKV's serving engine, built on SGLang (an open-source model server), tracks the operation history alongside the tokens so it never reuses the wrong notes, and frees dropped entries first when memory runs short.

The rule that did the work

The policy behind the headline number is almost boring. The tool-response rolling drop keeps the cache for the latest 12 tool responses, drops older tool-response cache, and keeps the model's reasoning. Repositioning kicks in at the same size where the baseline would have summarized.

The authors tested it on four agent benchmarks (BrowseComp-Plus, xbench-DeepSearch, a text-only GAIA set and SWE-Verified) with three open models: MiniMax-M2.7, GPT-OSS-120B and MiroThinker-1.7-Mini. Average scores across the benchmarks went up a little for each model, for example from 51.0 to 53.1 for GPT-OSS-120B; the paper has the per-benchmark breakdown if you want to see where it gained and where it did not. Average cost per task fell 27.7% compared with the summarization baseline. That cost is modeled from token counts at provider rates, with repositioned tokens charged at the cache-hit rate (the cheaper price for reused input), not taken from a cloud invoice. Against a second baseline that uses the same summarization at a lower 48,000-token threshold, the drop policy was still 21.9% cheaper.

On the serving side, the paper reports up to 1.2x output throughput and a 1.5x speedup in P95 latency for the AKV engine. It also tried letting a model iteratively improve the drop policy, which cut cost a further 7.1% over the hand-designed rules.

These are the authors' own measurements on their own setup. They are worth testing, not quoting as settled.

What the repo gives you

AKV has two halves, per its README:

• akv.agent, a client-side library that decides which messages to drop and when to reposition. You build a CachePolicy from three kinds of parts: a trigger (when to act), a selector (which messages), and a reposition rule. Each comes in turn-based, token-based and model-based versions, such as TurnTrigger, TokenSelection or ModelReposition. Custom parts are plain Python functions.
• akv.system, the SGLang-based server that carries the operations out behind an OpenAI-compatible /v1/chat/completions endpoint.

The interface is two extra fields in the request body. In the README's example, the policy adds:

"drop_message": {"4": [2]},
"reposition": [4]

Read it as: after message 4 is processed, forget message 2's cached notes, then compact positions. You always send the full message history with its original numbering; the server, not your code, does the forgetting.

There is also a DropContextTool that lets the model itself schedule drops, which the repo's demo uses for a multi-turn delivery-records agent.

When summarizing still makes sense

Dropping is not a universal win, and it helps to know where it loses.

A summary keeps the gist of something you threw away. A drop keeps nothing. If your agent reads a long spec early in a run and needs to remember its main rules three hours later, a short summary of that spec may be worth paying for. Dropping it means the agent either forgets the rules or calls the tool again.

So treat the two as different tools. Drop what is stale and cheap to fetch again, like search results, listings and logs. Summarize, or simply keep, what is expensive to recreate. AKV's model-based parts (ModelTrigger, ModelSelection and the DropContextTool) exist for exactly this case, letting the model itself choose what to forget instead of a fixed rule.

Put this into practice

1. Measure before you change anything. Before trying AKV, log what fills your agent's context. In many agent setups the answer is tool output: file reads, search results, command logs. If tool responses are a small share of your tokens, this approach will not do much for you.

2. If you self-host a supported model, try the server. AKV supports Qwen-family, GPT-OSS and MiniMax-M2 text architectures. Setup per the README:

git clone https://github.com/thu-nics/AKV.git
cd AKV
uv venv --python 3.12 .venv && source .venv/bin/activate
uv pip install -e .

You also need Rust and Cargo, and the system README asks for a working C++20 compiler and a writable compilation cache, because drop and reposition compile CPU kernels at startup. The README's quick start launches the server with GPT-OSS-120B, tensor parallelism of 2 (the model split across two GPUs) and port 8000, through the akv.system.launch_server module; check the system README for the exact flags before you run it.

3. Start with the paper's rule. Keep the last 12 tool responses, drop older ones, keep reasoning. It is the policy the authors measured, and it is easy to explain to a teammate when a run goes wrong.

4. Verify the policy fired. Responses report cached_tokens, drop_skipped_tokens and repos_tokens under usage.prompt_tokens_details. The docs do not define these as a pass or fail signal, so to confirm the policy actually fired, check the drop_message operations it adds to each request and the operation history policy.process returns.

5. If you use a hosted API, steal the principle, not the code. You cannot drop cache entries on someone else's server. You can still stop treating history as something to rewrite. Keep the early part of your context stable so prefix caching keeps working, cap how much of each tool response you append in the first place, and when you must trim, trim the oldest tool output rather than summarizing everything at once.

Honest limitations

This is not a drop-in fix for most agent builders. It needs your own GPU server running a supported open model. Hosted models from OpenAI, Anthropic or Google cannot use it.

The quick start uses two GPUs for a 120B model. Smaller supported models may need less, but the README does not promise a single-GPU path.

It is young. The repo has no tagged releases and a few dozen stars as of October 23, 2026. Its LICENSE file is Apache-2.0 text with a copyright line naming the SGLang team, which looks inherited from the SGLang code it builds on, and the README does not state a license.

Some serving features are off the table for these requests: the system README says pipeline and context parallelism (other ways of splitting work across GPUs), speculative decoding, LoRA, HiCache and beam search are rejected. The README also does not say whether stock OpenAI SDK clients pass the extra drop_message and reposition fields through. The documented workflow sends raw JSON over HTTP.

And dropping is forgetting. If the agent later needs a tool result you dropped, it has to call the tool again. The paper's scores suggest that rarely hurts on its benchmarks; your workload may differ.

The bigger idea

Most agent memory advice today is about writing better summaries. This paper argues the cheaper move is often to stop writing and start deleting, and to delete at the layer where deletion is cheap.

Whether or not you ever run AKV, look at your own agent's context this week and count how much of it is tool output nobody will read again. That number is your cost problem, and now there is a measured way to cut it.

Sources: Fu et al., "Cache Engineering for Agents," arXiv 2610.28577 (submitted October 20, 2026) · thu-nics/AKV README · Hugging Face Papers. Reviewed October 23, 2026 on the main branch; AKV had no tagged releases.