Independent AI intelligence Two editions daily · ET
Fervor AI
Menu

Analysis · November 1, 2026 · concept

Nolan Lawson agent framework benchmarkclaude-codeai-skillsagent-harness

Claude Built the Same App in 12 JavaScript Frameworks. The Bill Nearly Doubled.

When an AI agent writes your code, your framework comes with a bill. One developer priced it, and the reasons change how you should pick a stack.

Framework debates have always been argued on taste. Bundle size, developer experience, hiring pools, whatever the loudest person on the team used at their last job. The cost of the person writing the code rarely entered the math, because that person was on salary and the hours disappeared into a sprint.

That stopped being true the moment an AI agent started doing the typing. An agent bills by the token. Every file it reads, every doc page it pulls in, every line it writes and rereads on the next edit shows up as money or as usage against a plan.

So here is a question almost nobody had measured until last week: does the framework you pick change what the agent costs?

Nolan Lawson, a longtime web performance engineer, ran the experiment. He had Claude build the same app in 12 frameworks, several times each, and wrote down the cost, the tokens, the time and how fast the result loaded. The cheapest framework came in at $2.67 a run. The most expensive came in at $4.97.

The result is simple to state. When an agent writes your code, the framework is a recurring line item, not a one-time preference. And the gap is big enough to matter.

What the test actually did

The setup is worth understanding, because it is what makes the numbers fair.

Lawson wrote one spec: a small Pokémon browser (a Pokédex) with search and filtering, backed by a local copy of the public PokéAPI. Then he gave the same prompt to Claude Opus 5.5, running at high effort inside Claude Code, Anthropic's terminal coding agent. The prompt told the agent to build the app in a named framework and keep working until the test suite passed. Each agent ran alone in a fresh container, so no run could borrow from another (benchmark repo).

The 12 contenders were Angular, Astro, Gracile with Lit, Hono with htmx, Next.js, Nuxt, Qwik, React Router, SolidStart, SvelteKit, TanStack Start and plain Vite with no framework as a baseline. Each framework was attempted five times, and the post reports the median of the successful runs.

Two details keep this honest. Where a framework publishes official AI tooling, such as an MCP server (a plug-in that gives the agent live access to docs or tools) or agent skills (instruction files the agent can load), Lawson installed it. And the agents were not told to optimize for speed. They did not know a performance test was coming.

The cost figures are the API prices Claude Code reports for each session, for model tokens only, from runs in October 2026. Lawson is on a Max plan, so, in his words, the numbers do not "represent how much money I spent." Think of them as a meter reading, not a receipt.

The numbers that change a decision

Here is the spread, cheapest to most expensive, using Lawson's medians:

  • Hono with htmx: $2.67, 8.6 minutes. Hono is a small server framework; htmx is a library that lets plain HTML request page fragments from the server, so there is very little client-side JavaScript.
  • Astro: $2.76, 8.2 minutes, the fastest build. Astro is a site framework that ships zero JavaScript to the browser unless you ask for it.
  • Nuxt: $2.82, 9.0 minutes.
  • Next.js: $3.14, 10.9 minutes.
  • SvelteKit: $4.28, 10.8 minutes.
  • Gracile with Lit: $4.65, 12.1 minutes.
  • Angular: $4.97, 17.2 minutes, the slowest and most expensive.

The other five landed in between. The full table is in Lawson's post.

Angular cost about 86% more than Hono with htmx for the same app, and took twice as long. Run that difference across every feature, bug fix and refactor an agent touches in a year and it stops being a rounding error.

One thing surprised me reading the table. Tokens and dollars do not line up perfectly. Nuxt used the fewest tokens of any framework (3.85 million), yet Hono with htmx was cheaper while using more (4.26 million). Anthropic prices input, output and cached tokens (text the model has seen recently and can reread at a discount) differently (pricing), so the mix most likely matters as much as the total. If you only watch the token counter, you can draw the wrong conclusion.

Why some stacks cost more

Lawson's explanation has two parts, and both are worth carrying into your own decisions.

More code means more tokens, now and later. The Hono and htmx apps came out at about 1,370 lines of code. Angular's came out at about 2,009. An agent does not write code once and walk away. It reads files to understand them before it changes them. Lawson's point is that "the code size ultimately impacts the number of tokens, including in future edits." His benchmark only measured the first build, so treat the maintenance cost as his reasoned inference rather than a measured result. It is a reasonable one: a heavier codebase gives the agent more to read every time it opens the project.

This is the part that matters most. A framework that generates more boilerplate used to cost you some reading time. Now it likely costs you a little on every agent session for the life of the codebase.

Unfamiliar docs cost money. Lawson links some of the higher costs to agents reading documentation for newer or less familiar framework versions. When the model's training does not match the current version, the agent pays to learn it, run after run.

The surprise: official AI tooling often went unused

You would expect a framework's own MCP server or skill files to make the agent cheaper and faster. That is the pitch.

The per-framework notes tell a messier story. Hono's skills got used in all five runs, and so did React Router's. SvelteKit's plugin was used through its MCP in all five. But Nuxt's MCP server was never used once. Astro's MCP got used in two of five runs. Next.js agents read the dev-loop skill file in just one of five runs.

Lawson goes further than I would on this data: "I have a strong suspicion, for example, that the current crop of MCPs and skills may be doing more harm than good." His evidence does not prove harm. It does prove something simpler. Installing AI tooling is not the same as the agent using it. If you are paying for a skill to load into context, check that it actually loads.

What this test does not prove

Lawson is careful about the limits, and you should be too before you rewrite anything.

It is one app, and a specific kind: content-heavy, closer to a website than a dashboard. Apps that live in the browser all day, such as an editor or an admin panel, might favor different frameworks. It is one model at one effort level, through one agent. A different model, or Claude at a lower effort setting, could reshuffle the order.

The sample is small. Five attempts per framework is enough to see a pattern, not enough to call a $0.10 difference real. Angular, for example, ended up with only three usable runs after two runs built an app that refused requests from anywhere but localhost, which Lawson counted as failures to meet the spec.

And the performance side of the test leans one way. Lighthouse, Google's page-speed tool, rewards sites that ship less JavaScript, which favors multi-page apps (MPAs, where each click loads a new page from the server) over single-page apps (SPAs, where the browser runs the whole app). Astro shipped no JavaScript at all; Next.js shipped 142 KiB. Accessibility scored 100 for every framework, so that axis told you nothing.

Treat the ranking as a strong hint, not a verdict. The method is the valuable part.

Put this into practice

You do not need to switch frameworks to use this. You need to start measuring what your current stack costs when an agent works in it.

1. Run the benchmark yourself. The code is public. You need Linux, Podman 4.3 or later (a container tool similar to Docker), Node 20 or later (nvm, the Node version manager, picks the pinned version for you), about 15 GB of free disk and a Claude credential (setup):

git clone https://github.com/nolanlawson/agent-framework-benchmark
cd agent-framework-benchmark
nvm install && nvm use
export CLAUDE_CODE_OAUTH_TOKEN=...   # or ANTHROPIC_API_KEY=...
node bench/run.mjs --frameworks=vanilla --iterations=1

Start with one framework and one iteration, which keeps the first run cheap. Results land in runs/<timestamp>/, with a report.md comparison and a full agent transcript (a .jsonl log, one event per line) for every run. The README asks you to read bench/run.mjs before running it, since that is the only script that touches your machine; the agents work inside containers. Anything running in those containers can see the credential you pass in. An API key with a spending limit caps the damage; a Claude plan token draws on your plan's usage instead. The benchmark also installs current framework versions rather than pinned ones, so your numbers will not exactly match the October snapshot, and the repo shows no license file, so check before reusing its code. To reproduce the published numbers, the README gives --frameworks=all --iterations=5 --effort=high, which is dozens of full agent sessions, so budget for that.

2. Try your own stack against the spec. Framework keys and prompt labels live in bench/frameworks.mjs, and per-framework AI tooling gets installed by bench/container/setup-tooling.sh. If your team's stack is missing, those are the two files to look at.

3. Check whether your AI tooling gets read. Open a few agent transcripts from real work and search for your skill files or MCP calls. For a single transcript, the repo includes node bench/tokens.mjs <path-to-transcript.jsonl> to total its token use. If the skill never appears, you are paying to install something the agent ignores.

4. Track lines of code as a cost signal. In Lawson's test, the stacks that produced less code also tended to cost less to build, and a smaller codebase gives an agent less to read on later edits. Maintenance was not tested, so measure it on your own repo. When you compare two approaches to a feature, the shorter one now has a likely dollar argument behind it, not only a style argument.

5. Price tasks, not tokens. Watch cost per finished task (a feature shipped, a bug closed) rather than raw token counts, for the reason the Nuxt and Hono numbers show.

The bigger shift

For years, the advice was to pick the framework your team knows. That advice still holds. But "your team" now includes an agent with its own learning curve, and that agent sends a bill.

Lawson's test will not settle which framework is best. It was never meant to. What it shows is that the question can now be answered with measurements instead of arguments, and that his own conclusion was the modest one: apply "additional scrutiny to your choice of JavaScript framework."

Run the benchmark on the stack you already have. Then decide whether the number you get is one you are happy paying every time an agent opens the repo.

Sources: Nolan Lawson, "The AI cost of JavaScript frameworks" (Oct 28, 2026); agent-framework-benchmark README, default branch as of Nov 1, 2026; results repository; Anthropic pricing.