billion-context Lets Your AI Coding Agent Decide What to Forget

A free proxy folds old conversation into summaries the model can reopen. Here's how it works, how to try it, and the one test to run before a long session.

Every AI coding agent eventually hits the same wall. The conversation gets too long for the model's context window (the amount of text it can read at once), and something has to go.

Most tools handle this the blunt way. They chop off or squash the oldest part of the conversation when you near the limit, whether or not it held the one detail you needed. You notice it as an agent that suddenly forgets the database schema it read an hour ago, or re-breaks the bug it already fixed.

billion-context, an open-source project climbing GitHub's trending boards this week, tries something different. It lets the model itself decide what to fold away, writes a summary in its place, and gives the model a way to unfold it again later. The README's pitch is long sessions on small windows: "100K is enough," and "month-long single sessions."

That's a real shift in how agent memory works. It also puts a new piece of software between your agent and your model provider. Both halves deserve a close look.

What billion-context is

billion-context is a proxy, a small program that sits in the middle of a connection and passes traffic along. In this case it sits between your coding agent (Claude Code, Codex, Gemini CLI, OpenCode, Aider and many more, per the README) and the model API the agent talks to. Its command-line tool is called bili.

Your agent thinks it's talking straight to Anthropic or OpenAI. Every request actually passes through bili first, and bili decides what the model gets to see.

It runs entirely on your machine. By default it listens only on your own computer (127.0.0.1) and by default stores its state in a local SQLite database under ~/.local/share/billion-context/, with a log in ~/.local/state/billion-context/.

How the folding works

Picture a long document where you can collapse any section into a sticky note. The sticky note says what was in the section. If you need the exact wording again, you peel the note off and the section is still there.

billion-context gives the model that ability through four tools it injects into the conversation:

compress folds a range of messages into "a detailed summary."
decompress restores a folded range "when exact details are needed again."
search_context runs a keyword search over the summaries and the messages still visible.
acp_status reports how full the context is and which ranges can still be folded.

The README states the core idea directly: the model decides when and what to compress, rather than hitting "a hard truncation limit." The proxy nudges the model to fold roughly every 50,000 tokens of growth (a setting called compress.nudgeGrowthTokens), and has a backstop if the input alone would overflow the window.

Two design choices stand out. First, your agent's own saved history is never rewritten; the README says "bili never rewrites it." Only the copy sent to the model shrinks. Second, the folds are kept in small ranges so the cached beginning of the conversation stays the same, which matters because providers charge less for text they have already cached.

Why this is a different idea

Most context tools today treat forgetting as a disaster to delay. billion-context treats it as an editing job: the model picks what to fold, writes a summary of it, and can open the fold again if it needs the original.

That moves the risk. With truncation, you lose whatever happened to be oldest. With folding, you lose whatever the model judged unimportant at the time, and you get it back only if the model later realizes it needs it and asks.

Most of the time that's an upgrade. A model is better at judging what matters than a byte counter is. But it puts a new question at the center of your setup: does the detail you care about survive a fold? That's testable, and the test is below.

How to try it

The README's install is one npm command. It installs into ~/.local so it doesn't need admin rights:

npm install -g billion-context --prefix=~/.local

(On a Node install from nvm or Homebrew, a plain npm install -g billion-context works, per the README. Its Docker notes say the storage engine needs Node 22 or newer.)

There are three ways to put it in front of an agent:

Launcher. bili claude, bili codex, bili gemini and similar start the agent with the proxy in front of it. The README says "No real config file is ever edited."
Native plugin. bili plugin install claude installs it as a plugin for supported clients; bili plugin remove claude takes it out.
Standalone proxy. Run bili, which listens on port 8787, then point any client's base URL (the address it sends API requests to) at it by adding a prefix, for example http://localhost:8787/bili/https://api.openai.com/v1.

Check that it's alive and see what it's doing:

curl -s http://localhost:8787/__bili/health
curl -s http://localhost:8787/__bili/stats

The health check should return a small JSON object with "ok":true and the upstream API it forwards to. The stats endpoint is where you watch compression happen.

Run the canary test first

Before you trust any memory system with a long session, plant a canary: a specific, made-up detail the agent will need later.

Early in a session, tell the agent something precise and checkable: "Our staging database runs on port 6544, never 5432."
Work normally until the conversation has grown well past the first fold. With the default settings, that's somewhere past 50,000 tokens of new text. Call acp_status through the agent, or watch the stats endpoint, to confirm a fold has happened.
Ask a question that depends on the canary, without hinting: "Write the connection string for staging."

If the agent gets 6544, the detail survived, either in the summary or through a decompress. If it writes 5432, the fold lost it, and you have learned exactly how this tool fails before it cost you anything. Run it two or three times with different kinds of details (a number, a rule, a file path). Then repeat it with a canary the agent has no reason to think matters, like a passing remark about a deadline, because those are exactly the details a summary is likely to drop. Summaries tend to keep the gist and drop exact values, so numbers are the most useful canaries. The project's own benchmark backs this up: on a test of pulling an exact fact out of a huge pile of text, billion-context's score fell to 27.0%, against 69.0% for sending the full context, while it was the only setup that survived the longest task at all (benchmark README).

The middleman question

Here is the part a fast-trending repo can make easy to skip. billion-context sits in your model API path. For every model call routed through it, your prompts, any file contents the agent sends to the model, and your provider's authorization headers pass through it.

Three details in the README are worth knowing before you install.

It updates itself. The README describes an update check every three minutes with a staggered rollout. For software that sees your agent traffic, consider pinning a version. Running with --no-auto-update (or setting ACP_AUTO_UPDATE=0) stops routine upgrades, but the README describes a separate critical-advisory check that can still force an update; to stop that too, also set BILI_ADVISORY_CHECK=0. Then update on purpose.

Some modes intercept encrypted traffic. For clients that honor proxy settings, the launcher uses what the README calls certificate-based interception: a local certificate that lets it read traffic your agent believes is going straight to the provider. That's how it works without editing your config. It's also the kind of access you should grant knowingly.

Network exposure has no password. The default is safe, listening only on your own machine. But --host 0.0.0.0 and the Docker image both listen on the network, and the README is explicit that there is "no authentication." Only expose it on a network you trust completely.

None of this is a red flag on its own. It's simply what this kind of tool has to do to work. Treat it like any other program with access to your keys.

Honest limitations

Its own status line says "Early." The README's status section reads: "Protocol handling and compression work against mock tests (500+ passing). Real-model integration testing is the next milestone. Expect rough edges." The same README also cites a self-authored study covering 174,327 real model calls. Both can be true (a long personal deployment and an unfinished test suite), but take the status line seriously.

The benchmarks are the project's own. The README headlines results under "Independent Third-Party Benchmarks," including first place on ContextBench at about 58% lower cost than sending full context, and 79.20% against 62.60% for the best retrieval setup on LongMemEval-S. Those numbers live in the project's own repository, and the benchmark page names no outside evaluator. The "+16.6%" label is 16.6 percentage points, not a relative gain. Treat them as a starting claim, then run your own canary.

Not every surface goes through it. The README notes that Claude Desktop sessions bypass the proxy entirely, and that Claude Code's Remote Control command disappears while bili routes traffic.

The license has an extra term. It's MIT plus an "Attribution on Product Surfaces" clause: if you ship it inside a product people can see, you must credit billion-context and link the repo on your home page, docs or credits page. That's fine for personal use and easy to miss for a product.

It moves fast. The latest release, v0.1.213, shipped November 7, and hundreds of commits had landed on the master branch since then when we checked on November 10. The README itself warns that the development branch "can be broken at any moment."

Should you use it?

Skip it for short tasks. If your sessions end before the first fold, the proxy adds a moving part and buys you nothing. It earns its place on the long ones: a multi-day refactor, a migration that touches hundreds of files, an agent you leave running against a backlog.

If your agent sessions regularly hit the context wall, billion-context is worth an afternoon. The fold-and-reopen idea is a better default than blind truncation, and the tools it gives the model are simple enough to reason about.

Install it on your own machine, pin the version, and run the canary test with the kinds of details your work depends on. If your canaries survive, you've bought longer sessions for the price of one proxy. If they don't, you found out on a made-up port number instead of a production database.

Sources: billion-context README and LICENSE on the master branch, reviewed November 10, 2026; release v0.1.213.