Reef Retrains Your Agent's Harness Instead of Its Weights
The continual-learning repo sitting at #9 on Trendshift has two ideas worth taking even if you never install it. One is a receipt header on every response. The other is an update path that writes a skill file instead of a weight delta.
Every response Reef serves comes back carrying an extra HTTP header. It is called x-reef-agent-record-id, and the README describes it as a receipt: a string that "a later report uses to identify this interaction." That looks like observability boilerplate, the kind of trace ID every serving stack has shipped since 2015.
It is the whole architecture.
Reef is continual-learning infrastructure from Human-Agent-Society, Apache-2.0, at about 5.7k stars on a cache-busted shields read, with v0.1.1 tagged on September 25. It sat at #9 on the Trendshift momentum board when I looked at 15:12 ET on September 26. The pitch is the one every agent-tooling repo makes right now, that your agent should get better from use. The reason this one is worth reading is that it answers a question the pitch usually skips, which is how you know which interaction the feedback is about.
The gap between a log and a training set
Think about what you have after a week of running an agent in production. You have traces. Lots of them. You also have feedback, and the feedback arrives on a completely different clock: a thumbs-down twenty minutes later, a support ticket the next morning, a code review three days out, a test that fails in CI on Thursday about a change the agent made on Monday.
Those two streams are not joined. That is the actual reason most teams never train on their own agent traffic, and it is a plumbing problem wearing the costume of a machine learning problem. You can have a hundred thousand traces and zero training examples, because nothing connects "this was bad" to "this specific generation, with this specific context, produced it."
Reef's four-step loop puts the join first. The README lists it as: "Serve agent requests and record interactions," then "Match feedback to recorded interactions," then "Produce an update from eligible records," and finally "Apply the configured selection policy and publish accepted updates."
The receipt is step one's output and step two's input. You serve a request through Reef's OpenAI-compatible or Anthropic-compatible endpoints, at /v1/chat/completions or /v1/messages, and the response header hands you an ID. Whenever your feedback shows up, from whatever system, on whatever delay, you POST it to /reef/report with a score, some text or structured detail, and the receipt. The server validates the report schema and attaches it to the record.
That is it. That is the mechanism. It is not clever and it is not new as an idea, and that is exactly why it is worth copying: the reason your agent traffic is not a training set is almost never that the algorithm is hard.
The second idea: the harness is a training target
Here is where Reef stops being a well-organized trace store and starts being interesting.
It ships recipes, which are the policies that turn matched records into an update. Two of them show the split clearly.
The SAO recipe does what you expect. It runs on the Slime framework, accumulates feedback, and when there is enough, "the recipe runs a training step and synchronizes the updated weights to the serving runtime. Later inference requests use the current version without restarting Reef." Weight updates, hot-swapped into a live SGLang serving path. That is a real capability and it wants a real GPU stack, which the README says up front.
The Reefine recipe does something else entirely. It operates on "prompts, rules, and skills." It "uses a model API instead of GPUs." And when it decides on a change, it "writes the change as a skill, a rules entry, an agent command, or a pi extension."
Read that again with your own setup in mind. The thing being trained is your CLAUDE.md. Your skill files. Your slash commands. The artifacts you have been hand-editing for six months based on vibes and the last thing that annoyed you.
The startup command is not exotic:
reef serve --recipe reefine \
--inference.upstream-url http://127.0.0.1:11434 \
--inference.upstream-model gemma4:26b
That is an Ollama endpoint on the default port. The whole harness-evolution path runs against an upstream model API, which means a laptop with Ollama on it is a sufficient training rig, because the expensive part of the loop is a language model reading records and proposing a prompt change, not gradient descent.
I have been arguing for months that the harness is where most agent quality actually lives, and the usual counterargument is that harness changes are unmeasurable, so you tune them by feel. Reef's answer to that is the same machinery it uses for weights: a proposer generates a candidate, train/evaluation/ scores it before anything ships, a configured selection policy decides whether to accept, and artifact/ keeps the version history. A prompt change goes through the same gate a weight update does.
Turning "edit the prompt and see if it feels better" into "propose, evaluate against held-out records, accept or reject, version the result" is the part of this repo I would steal first, whether or not I ran their code.
Put this into practice
The lowest-friction path here does not start with Reef.
Start by emitting a receipt. Today, in whatever you already run, put a stable ID on every agent response and return it to the caller. One header. Then make every feedback surface you have carry that ID back: your thumbs-down button, your ticket template, your CI annotation, your code-review bot. This costs an afternoon and it is the prerequisite for everything else on this list. Without it you are collecting two piles of data that can never be joined.
Then look at your last month of feedback and ask which half is a harness problem. My rough split, and yours will differ: most agent failures I see are not the model being incapable. They are the model not knowing a project convention, calling the wrong tool because the description was vague, or repeating a mistake nobody wrote down. Every one of those is a prompt, rule or skill fix. None of them needs a GPU.
If you want to try Reef itself, installation is uv pip install reef-infra, and the simplest server is one line:
uv run reef serve --inference.model-path Qwen/Qwen2.5-1.5B-Instruct
Point a throwaway agent at it, send ten requests, capture the receipts, POST five reports to /reef/report with real scores, and then inspect what came back through /reef/scenarios and the version-history endpoints. That round trip takes under an hour and it tells you whether the receipt-and-report shape fits how your feedback actually arrives, which is the only question that matters before you commit to any of this.
Do the Reefine path second, not first, and do it against a scenario you can score automatically. A recipe that proposes prompt changes is only as good as the evaluation gating it, and if your score is a human opinion arriving twice a week, the loop will not turn often enough to tell you anything.
What I would want to know before trusting it
The benchmark results are all in the repository. AIME 2025 under recipes/gepa/examples/aime/, IMOAnswerBench and CEO-Bench under recipes/sao/examples/, Terminal-Bench under recipes/meta_harness/examples/, TriMul and circle packing under recipes/tttd/examples/, and preliminary WildClawBench numbers for GLM-53-Flash under recipes/skillclaw/. Those are the project measuring its own recipes. There is no independent reproduction cited for any of them, and "preliminary" is the repo's own word on the last one. Treat the catalog as a set of worked examples, which is genuinely useful, rather than as evidence that the approach beats an alternative.
The license file says Copyright 2025 Zhipu AI. The repository sits under an org called Human-Agent-Society, and the README does not mention Zhipu anywhere I could find. The Apache-2.0 grant is clean either way, so this is not a legal problem. It is a disclosure one, and it matters because one of the benchmark recipes is measuring a GLM model, which is Zhipu's own family. Knowing who is behind a benchmark changes how you read it.
The operational requirements are real and easy to miss. Artifacts need git-lfs installed as a system package. Weight training needs a GPU stack matching the dependency guides. And harness sandbox isolation wants Linux with bwrap and pasta for non-root users, with the documented escape hatch being REEF_PROPOSER_SANDBOX=none. That last one deserves a hard look before anyone sets it, because the thing being sandboxed is a model-driven proposer that writes files into your agent's configuration. Switching off isolation on a component whose job is to edit your rules is a decision, not a convenience flag.
Version v0.1.1 is also exactly what it sounds like. A long alphabetical contributor list in the README and a serious design do not make a 0.1.x tag a stable dependency. Pin it, and expect the interfaces to move.
The part that outlasts the repo
Reef may or may not be the tool you end up running. The two ideas inside it are not going anywhere.
The first is that feedback is only training data if something joins it to the generation that caused it, and that join has to survive a delay measured in days and a hop into a different system. A receipt on the response is the cheapest join there is.
The second is that "continual learning" does not have to mean weights. When the update path can write a skill file, the loop runs on a laptop, the change is readable, and a human can review the diff before it ships. That is a better property than a weight delta has, and almost nobody is treating it as the primary path.
Go find the last five things your agent got wrong. Ask, for each one, whether a sentence in a rules file would have prevented it. If the answer is yes more than twice, the training run you need is not a training run.
Sources: Human-Agent-Society/reef README · reef releases (v0.1.1, 2026-09-25) · Trendshift daily board, read 15:12 ET 2026-09-26. Star count from a cache-busted shields.io read; api.github.com was not reachable for a cross-check.
Medium metadata
Title: Reef Retrains Your Agent's Harness Instead of Its Weights Subtitle: A receipt header on every response, and an update path that writes a skill file instead of a weight delta. Tags: AI Agents, Machine Learning, Open Source, LLM, Developer Tools