CUA-S1 Ships a Model Card That Disqualifies Its Own Launch Benchmark

Cua open-sourced a 706,000-parameter form specialist and a document explaining why you should not believe its headline number. The document is the better artifact.

A launch post and a repository can disagree, and usually the repository is the one telling the truth. Cua put CUA-S1 on Hacker News on September 19 with a set of numbers that read like a rout: 99.7% correct against a hosted general decision model's 83.6% on a full form-filling decision set, 100% versus 74% on the subset where the correct move was leaving an already-filled field alone, 7 to 9 milliseconds of local scoring against 260 to 280 milliseconds per hosted call. Then you open libs/cua-s1 in the repo that post links to and the first section says the release "does not include or download model weights, datasets, demo binaries, or recordings," followed by a flat sentence: "No checkpoint performance claim is established by this source-only release."

The model card goes further. Under Evaluation: "Comparisons are meaningful only when task definitions, environments, scoring methods, and model-selection procedures are compatible."

That is a project publishing, in its own documentation, the criterion by which its own launch comparison does not qualify.

I want to be careful about what kind of story this is, because the obvious reading is the wrong one. This is not a company caught lying. The Show HN, which opens "We're Dillon and Francesco from Cua" and was posted from Francesco Bonacci's account, flags the problem itself, noting that the hosted model "has not been fine-tuned for it, so this is an experiment in scoped specialization." Everybody involved said the true thing. The gap is between the true thing and the number that travels, and Cua shipped the rare release where the documentation is built to close that gap rather than paper over it. That documentation is worth more to a working builder than the model is.

What CUA-S1 actually is

Strip the benchmark and here is the object.

cua-s1-form-v0 is described in the Show HN as a 706,000-parameter checkpoint at 2.8 MB, trained in under 30 minutes on synthetic data. The model card describes the reference tinyx configuration as a byte-level transformer encoder with an option-attention classification head, and adds that "its size depends on the checked-in configuration used for a training run," which means the parameter count is a property of a config, not of the family.

Given a structured set of interface elements and values extracted from a source document, the model picks one option per element from a fixed four: fill the element with an extracted value, check it, click it, or skip it. It does not generate field values. It scores elements independently, and plain code turns the selected options into an execution order.

Two limits are stated in the model card that the launch numbers do not carry. First, the document parser "only extracts values represented as Label: value pairs," which is a narrow slice of real documents. Second, the training generator produces fictional form episodes using reserved phone numbers, .invalid domains and invented brands, and the card says outright that "generated data is not a substitute for evaluation on real interface variation."

So the shape is: a very small classifier, trained on synthetic forms, evaluated on synthetic forms, scoring one decision per field. The 99.7% figure describes that closed loop.

The part worth stealing

Here is where the release earns its attention, and it has nothing to do with the checkpoint.

Most agent evaluation reduces everything to one number: did it do the right thing. CUA-S1's offline evaluation utilities do not. The model card lists what they report separately: abstention, coverage, selective accuracy, wrong actions, wrong targets, and unsafe actions.

Read that list slowly, because six categories are doing work that one number erases.

Wrong action and wrong target are different failures. Clicking the right checkbox for the wrong reason and clicking the wrong checkbox correctly are both "incorrect," and they need different fixes. Abstention and coverage together tell you how often the model declined and how much of the job it attempted, which is the pair that separates a cautious model from a useless one. Selective accuracy asks how good the model is on the subset it chose to act on, which is the only accuracy number that means anything once a model is allowed to skip. And unsafe actions is broken out from wrongness entirely, because an action that cannot be undone is a different category of problem than one that can.

Almost no agent eval I have run into measures the action-taken-when-abstention-was-correct case at all. It is the failure mode that hurts most in production, and it is invisible in aggregate accuracy, because a model that acts on everything and is right 95% of the time scores better than a model that abstains on 20% and is right on 99% of the rest, right up until the moment the first one fills a field it should have left alone in someone's live account.

The second stealable artifact is the release checklist. The model card says a checkpoint release must report six things: the exact checkpoint and code revisions; the task set, environment, applications and operating-system configuration; the action space, observation method, stopping rules and retry policy; success criteria and independent outcome verification; aggregate results with representative failure categories; and known exclusions and material differences from real-world deployment.

That is a usable template for any narrow model you train and hand to a team. Print it. The fourth item, independent outcome verification, is the one most internal model reports skip and the one that catches the most embarrassing failures.

The safety boundary is real engineering

One more thing in the repo deserves credit, because it is the opposite of the usual demo-first posture.

The runtime defaults to a dry run. execute and submit are independent opt-ins, so turning on execution does not turn on submission. It requires one unambiguous target window, uses snapshot-bound element tokens, and reobserves the window after every mutation. Submission is deliberately narrow: submit=true permits at most one high-confidence Button or AXButton whose normalized label is exactly Submit or Submit Form, and other click decisions are omitted.

The checkbox handling is the detail that shows someone thought about state rather than actions. The runtime refuses checkbox mutations when the role or checked state is unknown, skips a checkbox that is already checked, and verifies the checked postcondition afterward. The model card then says a future runtime "must preserve an equivalent fail-closed boundary rather than assume that actions are idempotent," which is a sentence I wish more agent frameworks had written two years ago.

The observed failure modes are named too, and they are the mundane ones: window-title changes, and concept collisions like email versus street address, or state versus an organization name. Those are the confusions a form filler will actually hit, and naming them beats a robustness claim.

Put this into practice

You cannot reproduce Cua's numbers, because the weights are not distributed. The checkpoint table lists cua-s1-form-v0 as "Profile defined; weights not distributed." So the practical value is in the pipeline and the process, and both are reachable today.

Read the model card before the README. Five minutes. It is the most honest document in the repo and it tells you the scope, the failure modes and the licensing posture up front.

Pull the metric set into your own evaluation harness first. This is the highest-value, lowest-effort move and it does not require touching Cua's code at all. Wherever you currently log a single correctness boolean for an agent decision, split it into: did the model act or abstain, was the action type right, was the target right, was the action reversible, and what is accuracy on the acted-upon subset only. You will find failures your current dashboard is averaging away. Do this even if you never run a specialist model.

Then stand the source up locally. The component installs standalone:

uv sync --project libs/cua-s1/python --extra pdf --group test
uv run --project libs/cua-s1/python pytest libs/cua-s1/python/tests

The package is cua-s1, imported as cua_s1, and it is not published to a package index, so the repo is the only source. Loading a checkpoint requires a local safetensors file and matching JSON config; pickle-based PyTorch checkpoints are rejected outright, which is the correct default.

If you wire up the MCP server, read the environment contract carefully. cua-s1-mcp needs CUA_S1_PLANNER_FACTORY in module:attribute form, and the docs are blunt that importing the factory "executes code with the server process's privileges, so do not point it at untrusted modules." CUA_S1_ALLOWED_PDF_ROOTS bounds what the server may read, and if it is unset the server falls back to its current working directory, which is almost never what you want. Set a dedicated least-privilege directory. The docs also warn that MCP tool results and stdio logs are sensitive, since they can carry values pulled out of PDFs along with form labels and window metadata.

Expect fill execution to fail closed. The portable Cua Driver contract does not currently expose set_value, so filling fails unless your connected runtime advertises compatible token-based value mutation. Planning works regardless. That is a feature, not a bug, but it will look like breakage on first run if you do not know.

What this release does not give you

No weights. That is the headline limitation and everything else follows from it. You can read the architecture, generate synthetic data, run the training loop and evaluate your own checkpoint, but you cannot check Cua's.

No untouched holdout. The README says a future checkpoint release "must add an untouched holdout, artifact hashes, exact environment details, and independently reproducible results," which is a precise description of what the current evaluation lacks. The synthetic splits are disjoint by form signature and model selection uses validation rather than test results, which is better practice than most shipped projects manage, but a synthetic holdout is still synthetic.

A licensing tail worth watching. The source is MIT, and the README states plainly that the license "does not apply to future official model weights, datasets, hosted services, or Cua trademarks," while the model card adds that an official checkpoint "may use separate terms that permit research and evaluation while requiring a commercial license for production, hosted inference, resale, or commercial redistribution." If your plan is to ship a product on a future CUA-S1 checkpoint, that plan currently has an unpriced line item in it. The card is clear that this is a possibility rather than a commitment, but the possibility is disclosed for a reason.

And the specialist framing itself has an edge the card names better than the launch post did: "A specialist checkpoint may also overfit its evaluation distribution and may not recognize when a task has moved outside that distribution." A 706k-parameter model scoring a form it has never seen the shape of will still return confident-looking probabilities. Nothing in the architecture tells you when it has left its training distribution.

What to do with this

There is a version of this release that gets covered as "tiny model beats big model," and that version will be wrong in a way nobody can correct once it is moving. The comparison was specialist against generalist on the specialist's own convention, on synthetic forms, with weights nobody else can load, and the people who ran it said so three separate times in three separate documents.

The version worth carrying is different. Cua shipped a model card that names its scope, enumerates its failure modes, separates six kinds of wrong, and publishes the standard by which its own numbers should be judged as preliminary. That is a higher bar than most model releases clear, including releases from labs with a thousand times the headcount.

So take the six-category metric set and put it in your harness this week. Take the six-item release checklist and make it the template for the next internal model report you write. Those two lists cost nothing, apply to work that has nothing to do with computer use, and will outlive whatever the eventual checkpoint scores.

The benchmark is a number. The documentation is a practice. Take the practice.

Sources: cua-s1 README · cua-s1 MODEL_CARD.md · Show HN: CUA-S1, September 19, 2026 · trycua/cua

Medium metadata

Title: CUA-S1 Ships a Model Card That Disqualifies Its Own Launch Benchmark
Subtitle: Cua open-sourced a 706,000-parameter form specialist and a document explaining why you should not believe its headline number. The document is the better artifact.
Tags: AI Agents, Machine Learning, Open Source, Computer Use, Software Engineering
Canonical: import from the fervorai.dev URL