VitaBench 2.0 Ran Three Agent Memory Architectures Against the Same Tasks, and Agentic Memory Won Half of Them

Meituan's MIT-licensed personalization benchmark has 42 stars and a May 2026 leaderboard that contradicts how most teams pick a memory backend. The model names are dated. The comparison isn't. Here's how to run it against your own agent.

The received wisdom about agent memory says that dumping the full history into context is the ceiling nobody can afford, and every memory system you actually ship is a lossy approximation of it. VitaBench 2.0 tested that directly, running identical tasks against Full Context, agentic memory, and RAG retrieval on 27 model configurations.

Agentic memory beat full context in 14 of them.

Not by rounding error, either. o4-mini goes from 0.210 Avg@4 under full context to 0.270 under agentic memory, a 29 percent relative gain. GPT-3.5-Turbo goes from 0.140 to 0.231, a 65 percent gain. Gemini-2.5-Pro picks up 0.047. Meanwhile Claude-Opus-4.6, sitting at the top of the board, loses ground under both memory settings, and so does every other model above roughly 0.43.

Before you check those model names against your own stack: that run is from May 2026 and Meituan has not refreshed it. Claude-Opus-4.6 tops the board because Anthropic has shipped four models in under two months since (Mythos 5, Fable 5, Sonnet 5, Opus 5) and none of them are on it. So read the rankings as a historical artifact and ignore the ordering.

What survives the date is the comparison, because every row tests one model against itself across three memory backends on identical tasks. Opus 5 existing does not change what Opus 4.6 did when you took its raw history away.

That is a different finding from "memory systems are lossy," and it is the reason this repo is worth an afternoon even though it has 42 stars and 4 forks.

What the benchmark actually measures

VitaBench 2.0 comes from Meituan's LongCat team with co-authors at the National University of Singapore and USTC, published as arXiv:2605.27141 with the code under MIT and the tasks on Hugging Face.

The setup is a simulated long-term relationship rather than a set of one-shot prompts. Each of 56 users gets a profile and a set of fine-grained preferences that the agent never sees stated outright. Preferences live inside fragmented dialogue and behavior logs (browsing, ordering, reviewing, searching) mixed with noise that is irrelevant or misleading. Each user has a temporally ordered sequence of 10 to 20 tasks across delivery, in-store consumption, and online travel booking, wired to 66 executable tools. Between tasks, new interaction history arrives, and preferences drift: some get added, some go stale, some change.

The evaluation is rubric-based against the final environment state, not against generated text. Each task carries atomic constraints (price ranges, item attributes, timing, context) checked against both the trajectory and the outcome. So a run only counts if the agent booked the right thing for the right reason.

The piece that makes it useful as a design tool is the memory interface. Two operations, UPDATE(M, H) and RETRIEVE(M, q), sit between the agent and the history. With memory on, the agent cannot see raw history at all. It only sees what the memory module hands back. That lets you swap architectures under identical tasks, tools, and rubrics:

Full Context dumps everything into the prompt.
Agentic Memory lets the model decide what to keep, update, and discard after each batch of new history. The reference implementation is based on MemAgent.
RAG Memory chunks and embeds records, then does similarity retrieval per query.

Three metrics per setting. Avg@4 is the mean success rate over four rollouts. Pass@4 is best-of-four. Pass^4 is the fraction solved in all four, which is the number that tells you whether the thing works or just got lucky.

The result that should change a design doc

Sort the leaderboard by full-context score and the pattern falls out cleanly.

Every entry above 0.43 under full context lost ground when you swapped in agentic memory. Claude-Opus-4.6 drops from 0.503 to 0.454. Doubao-Seed-2.0-pro drops from 0.474 to 0.428. DeepSeek-V4-Pro drops from 0.472 to 0.449. GPT-5 drops from 0.441 to 0.421.

Below 0.40, it inverts. Thirteen of the eighteen entries in that range go up with agentic memory instead of down.

Treat 0.43 as a property of this particular field of models rather than a constant of nature. What transfers is the direction, not the decimal.

The mechanism makes sense once you see it. Memory compression is a filter, and a filter costs you information while saving you attention. If the model is strong enough to sift a long noisy history on its own, the filter is pure loss. If it isn't, the filter is doing work the model can't do, and it earns its place. Across this set of models the two effects cancel somewhere around 0.42. On a board with Opus 5 and GPT-5.6 Sol on it, that line would sit somewhere else.

Which means "should we use agentic memory or full context?" is a question with no general answer, and any design doc that answers it without naming a model is answering the wrong question.

The consistency numbers add a second wrinkle I did not expect. For Claude-Opus-4.6, agentic memory scores higher on Avg@4 than RAG does (0.454 against 0.430), but RAG scores higher on Pass^4 (0.299 against 0.259). So agentic memory gets more tasks right on average while RAG gets more tasks right every single time. If your product breaks when the agent is inconsistent rather than when it's wrong, those two columns point at different backends.

And the ceiling itself is low. The best score on the board is 0.503, and the best Pass^4 anywhere is 0.337. The authors also report that even when they hand the model ground-truth preferences directly, top models only reach moderate scores, which locates the bottleneck in applying preferences rather than extracting them.

Put this into practice

You can get a signal out of this in an evening. A full run is expensive, so scope it deliberately.

Clone and install. git clone https://github.com/meituan-longcat/VitaBench-2.0.git, then cd VitaBench-2.0 && pip install -e .. That gives you the vita CLI. Python only, six commits, no packaging surprises.
Pull the tasks from Hugging Face. huggingface-cli download meituan-longcat/VitaBench-2.0 --repo-type dataset --local-dir data/vita/domains/personalization. You end up with a tasks.json.
Point it at whatever endpoint you already run. Copy src/vita/models.yaml.example to models.yaml and change default.base_url. Any OpenAI-compatible endpoint works, including vLLM, llama.cpp, Together, and Azure. The YAML expands ${VAR} from your shell, so keys stay out of the file. For RAG you also want VITA_EMBEDDING_MODEL, which defaults to text-embedding-3-large.
Run one task first, not the suite. vita run --domain personalization --memory-type rewrite --agent-llm gpt-4.1 --user-llm gpt-4.1 --evaluator-llm gpt-4.1 --num-tasks 1 --max-steps 50. This is the step that tells you what a rollout costs you before you commit to hundreds of them. Add --save-to run1.json to keep the trace.
Then run the comparison that matters. bash scripts/run_memory_benchmark.sh full_context rewrite rag runs your model against all three architectures on the same tasks. There are six backends available, and two of them are worth knowing about beyond the headline three: null gives you a no-memory floor, and groundtruth injects the canonical preferences directly for a true upper bound. Running null and groundtruth alongside your real backend is how you find out whether your memory system is closer to nothing or closer to perfect.
Read the delta, not the absolute number. Your score against Meituan's food-delivery and travel-booking tasks is not your score on your product. The gap between your backend and full_context for your specific model is the number that transfers.

The rag_cache backend with scripts/precompute_rag_cache.py is worth using if you plan more than one pass, since re-embedding the same history on every run is a bill you can avoid.

Honest limitations

The released task set is Chinese-language. The README says an English version is coming, with no date attached. If your evaluation depends on English-language preference inference, you are waiting.

The numbers in the repo and the numbers on the project site disagree about the size of the benchmark. The README's quickstart says the download gives you 56 users and 771 subtasks. The paper and the project page both say 819 subtasks across the same 56 users. That's a 48-subtask gap between what's documented as shipping and what's documented as evaluated, and I couldn't resolve it from the published material. Check your own tasks.json before you cite either figure.

The leaderboard is stale, and it's worth being blunt about how stale. It says "Last update: May 2026." A model four Anthropic releases back sits at the top of it. Opus 5, GPT-5.6 Sol, and everything else from the past two months are absent, and Meituan has not published a refresh. So nobody has run the current frontier through this memory interface, which means the most interesting version of this experiment has not been done yet. If you run it, publish the numbers.

The benchmark is also not news. The paper went up in May, the repo has six commits, and it landed on my radar because it surfaced on a trending board this week. Treat it as a two-month-old result you're seeing late rather than a fresh release.

The user simulator and the rubric judge are both gpt-4.1-2025-04-14. Using one model family to simulate the user and grade the result introduces a bias the authors don't quantify, and it means your own scores are partly a measurement of GPT-4.1's judgment.

The cost is real. Four rollouts across hundreds of subtasks with a frontier model, plus a simulator model and a judge model on every turn, is a budget line rather than a smoke test. This is why you run one task first.

And one thing the benchmark cannot tell you: whether the agentic-memory result generalizes past MemAgent. "Agentic memory" here means one reference implementation. Your memory layer might be better or worse than the thing that got measured, and the leaderboard gives you no way to know which.

What I'd do with this

If you have a memory design doc that says summarize-and-store gets you close to full context, this repo turns that from an assumption into a testable claim, and the answer probably depends on which model you're routing to. That's an uncomfortable finding for anyone who wants one memory architecture across a model-agnostic product.

The thing I'd watch is whether the crossover point moves. If it holds at roughly 0.42 as models get stronger, then agentic memory is a scaffolding for weak models and the frontier will keep pushing toward larger raw context. If the crossover climbs as context windows grow and attention degrades, the opposite is true and memory compression becomes more necessary, not less.

VitaBench 2.0 has 42 stars. The desktop app for running five coding agents at once has 16,400. If you install one agent-related repo this week, I'd argue for the one that tells you when your design is wrong.

Sources: meituan-longcat/VitaBench-2.0 on GitHub, VitaBench 2.0 project page and leaderboard, arXiv:2605.27141, VitaBench 2.0 dataset on Hugging Face.