NVIDIA NeMo Switchyard Cuts Agent Costs 74 Percent. Its Known-Issues File Says the Meter Is Broken.

A Rust proxy that routes your coding agent between a cheap model and an expensive one. The published savings are real. The instrument you would use to confirm them is on the known-issues page.

NVIDIA's NeMo Switchyard is sold as a cost lever, and it has numbers. Routing between a small model and a frontier model cut spend 74 percent on one partner benchmark. Then you open the known-issues page and read the five entries filed against version 0.2.0. The first one says buffered upstream work continues after the client disconnects, so a cancelled request can still incur provider cost. The second says routing-tier attribution is missing from GET /v1/stats and /metrics for escalation decisions, stage-router fallbacks, and classifier judge failures.

Read those together. A tool whose entire value proposition is spending less money currently cannot always tell you which tier served a request, and can charge you for requests you cancelled.

That is not a reason to skip it. It is the reason to run it the specific way I am going to describe, and to not report a savings number to your finance team this month.

Why routing became a budget line

Model routing was a research topic for two years and became a procurement conversation this summer. NVIDIA published the numbers on August 11.

LangChain benchmarked Switchyard against its internal deep-agents evaluation suite: 145 multi-turn agentic tasks shaped like production work, including customer support under policy constraints and on-call incident investigation. Routing between NVIDIA Nemotron 3.5 Lightning and Claude Opus 4.8 with the escalation router, across five runs, produced a 74 percent cost reduction against a frontier-only baseline while sending only 7 percent of calls to the frontier model. Accuracy dropped about six points.

Cognition implemented the staged-routing method in Devin Desktop and deployed it to NVIDIA internal users. On FrontierCode Main, routing between Opus 5 and Kimi K2.7 landed 50.6 percent accuracy at a $3.11 mean cost, within 2.8 percentage points of Opus 5 at roughly 28 percent lower mean cost.

Both sets of numbers come from NVIDIA's own blog describing partner results. Take them as directionally honest and specifically unaudited.

My position: the cost story is the least interesting thing here. Six accuracy points is a real tax, and whether 74 percent savings justifies it depends entirely on what your agent does. The durable value is that provider choice stops being an architectural commitment. That matters more this month than last, because token prices have started moving in both directions and teams who sized an agent fleet against one price list are finding out how fast a price list gets withdrawn.

What Switchyard actually is

Switchyard is a Rust proxy and library. This is worth stating plainly because the project was rewritten from Python and a lot of secondary coverage, including trend boards and a stale copy of its own README that GitHub served me during research, still describes it as a Python proxy. The documentation site's own page metadata still reads "for Python." The current docs open by saying Switchyard has three native Rust execution paths.

The launcher path is the one to start with. You install a Python-distributed CLI that wraps a packaged PyO3 Rust extension, then launch a coding agent through it:

uv tool install --python 3.10 "nemo-switchyard[cli]"
export OPENROUTER_API_KEY="your-openrouter-key"
switchyard launch claude --model switchyard

codex and openclaw use the same packaged deployment. The launcher starts the native server, points the agent at it, and tears it down on exit. It does not install the standalone server binary.

The server path is the standalone proxy. It comes from crates.io with cargo install --locked switchyard-server and reads an explicit TOML file. The config has three layers: llm_clients define an upstream base URL, wire format, and the environment variable holding the credential; targets name one upstream model each; routes expose one client-visible model ID and the algorithm that picks between targets. Secrets stay out of the file by design, since api_key_env names the variable rather than holding the key.

The library path embeds routing in your own Rust application through switchyard-libsy, which never calls a model itself. The algorithm yields a stream of steps, each Step::CallModel is a call your host makes over its own transport, and the run ends with Step::Done. That is the right shape for anyone who already owns their HTTP stack, retries, and credentials.

Translation is the piece that makes any of it work. switchyard-protocol defines provider-neutral requests, responses, tool calls, and streaming events, and switchyard-translation converts between openai_chat, openai_responses, and anthropic_messages. Your client keeps speaking its native dialect while a different backend serves the request.

The routers, and one that is commonly described wrong

The standalone server supports four primary route types: passthrough sends everything to one target, random splits by weight for A/B work, llm_classifier asks a classifier target to choose between weak and strong, and stage_router reads tool results and progress signals to pick an efficient or capable target.

Escalation is not a fifth router. It is llm_classifier with mode = "escalation", and its behavior is more aggressive than most write-ups suggest. It does not start a session cheap and promote it when things get hard. Every turn runs on the efficient target first, and a judge reads that answer to decide whether to re-send the same request to the capable target. That is per-turn re-dispatch, which is why it can produce large savings and also why it puts a judge in the path of every single turn.

The roles matter more than the names. Strong, weak, capable, and efficient are positions inside an algorithm, not properties of a model, and the same upstream model can hold different roles in different routes.

Put this into practice

The lowest-friction version takes about fifteen minutes and involves no routing at all.

1. Install the launcher and run the packaged deployment. uv tool install --python 3.10 "nemo-switchyard[cli]", export an OpenRouter key, then switchyard launch claude --model switchyard. You are proving translation works before adding a routing decision on top of it.

2. Make your first task tool-heavy, not conversational. Ask the agent to read three files, make an edit, and run a test. A chat request exercises none of the machinery that breaks. Known issue 2 from 0.1.0 is still listed: tool-bearing Codex requests may fail when the upstream accepts only a fixed set of tool names or schemas. The launcher guide documents the concrete version of this, where Bedrock's 64-character toolSpec.name cap collides with long auto-injected MCP tool names and returns a 400.

3. Move to the server path with --dry-run before you bind a socket. switchyard-server --config routes.toml --dry-run validates schema, environment lookup, target references, and route construction without starting anything. Then run it on 127.0.0.1 and check /health and /v1/models.

4. Treat /v1/stats as a hint, not a bill. Compare it against your provider's own billing for a full day before you believe any savings figure. Two separate known issues bear on this, and both push the numbers in the flattering direction.

5. Keep passthrough one config change away. A passthrough route back to your original provider is the fastest rollback you have.

Honest limitations

The accounting is the weak part, and it is the part being sold. Version 0.2.0 lists five known issues and four of them touch measurement: cancelled requests can still cost money, routing-tier attribution is missing from stats and metrics for exactly the interesting decisions, the retry recovery counter stays at zero after a successful retry, and x-switchyard-session-id is not recorded in native session stats. Version 0.1.0's zero-token-usage bug for completed Codex Responses tasks is still on the page.

The docs describe a header the server does not send. Known issue 5 says the native server does not send the documented X-Switchyard-Version header upstream. Small thing on its own, useful signal about how fast the Rust rewrite is moving relative to its documentation.

The benchmarks come from interested parties. LangChain and Cognition are both partners, and the results reach you through NVIDIA's blog. To LangChain's credit the roughly six-point accuracy tradeoff is stated plainly rather than buried.

Star counts are unreliable right now. The repository page showed 342 stars and 51 forks, while GitHub's cached file pages served me materially lower figures during the same research session. If you are sizing adoption, load the repository root and ignore anything quoted from a file view.

A judge in every turn is a latency story nobody has published. Escalation routing re-dispatches the same request to a stronger model after a judge reads the weak answer. The cost numbers are public. The wall-clock cost of that pattern is not, at least not in anything I could find.

It is early. The project self-describes as pre-alpha with the API and algorithms expected to change before 1.0, and the version numbering has already moved from 0.1.0 to 0.2.0 with a growing known-issues list. Run it in a branch.

What this is actually good for

Switchyard's real contribution is that a semantic target name sits between your agent and a provider endpoint. Change the endpoint later and the routing contract holds. That is a smaller claim than "74 percent cheaper" and a more durable one.

So try the launcher path on a tool-heavy task and watch whether the edits land. If they do, you have an exit from whichever provider just repriced you, and you can spend the next month figuring out what routing is actually worth on your workload. Just do the arithmetic against your provider's invoice rather than the proxy's own counters, at least until that page gets shorter.

Sources: Switchyard documentation, Getting Started; Switchyard Core Concepts; Switchyard Known Issues; NVIDIA Technical Blog, "Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard," August 11, 2026; NVIDIA-NeMo/Switchyard on GitHub.