The Typed Decision Leaderboard Put a Length-and-Formatting Baseline Above Eight of Thirteen Answer Verifiers
What a shared test set of 2,018 items says about the judge sitting in your pipeline, and why AUC is the wrong number to deploy on
A logistic model that reads nothing about an answer except how long it is, how many digits it contains, and how it is formatted scored 0.7036 AUC on a test set of 2,018 question-and-answer pairs. Eight of the thirteen real answer-verification systems measured on that same test set scored lower.
Not lower by a rounding error. Patronus Lynx 8B scored 0.5179. Convai's Laya, in its two configurations, scored 0.5144 and 0.4796, against a line where 0.5 is a coin flip. Asking GPT-4o-mini directly scored 0.5878.
That result comes from a leaderboard published on Hugging Face on September 20, 2026 by an author writing as Proto_AGI, who ran thirteen systems against one set of items with identical labels and published the grading code. The category these systems belong to has a name now. A typed decision model takes a structured question and returns a structured verdict, a boolean or a choice or an ordinal score, in a single forward pass with zero generated tokens. That is what makes them one to three orders of magnitude cheaper than asking a frontier model the same question, and it is why they have spread fast enough that a shared test set was overdue.
What the shared test set found is that most of them are detecting shape.
Why this lands harder than another benchmark result
Everybody has a judge in their pipeline now. It might be a hosted verification API, an open model behind a local server, or a prompt that asks a cheap model whether an answer looks right. Whatever it is, its job is the same: read an answer, decide whether it is good enough, and either let it through or send it back for a retry.
The reason that job matters is that it is load-bearing in a way most components are not. A bad retriever returns worse context and you notice. A bad verifier lets bad answers through and blocks good ones, and both failures look like the system working. The gate fires. The retries happen. Latency and cost go up in the way you budgeted for. The dashboard is green.
Here is the position I will defend for the rest of this piece: if your verifier does not clearly beat a surface-feature baseline on your own labeled data, you do not have a verifier. You have an expensive length detector, and the number the vendor published cannot tell you which one you have.
The mechanism, not the ranking
The leaderboard's top of table is almost beside the point. VIDRAFT's ZTC at 397B scored 0.7364 and TypeSafe's Jev scored 0.7350, a gap of 0.0014 with a 95% paired-bootstrap interval running from -0.019 to +0.032. The board's own rule is that an interval containing zero produces no rank, so it declines to separate them and says so above the table rather than in a footnote. Good.
The part that should change how you work is what happened when the same systems were used as actual gates.
The setup was simple. Score every answer with a verifier. Send the lowest-scoring 20% to a stronger model to be re-answered. Keep the rest. Every arm drew from one shared pool of re-answers, so no system got luckier retries, and ties were broken randomly and averaged over 200 seeds. Against a no-gate baseline of 74.83% accuracy:
| Gate | Final accuracy | vs. no gate |
|---|---|---|
| ZTC | 76.16% | +1.34 pp |
| JEV | 74.76% | -0.07 pp |
| Random | 74.58% | -0.25 pp |
Two systems separated by 0.0014 AUC are separated by 1.4 percentage points of end-to-end accuracy. One of them is worth the retry budget. The other performs about as well as routing at random.
The reason is the single most useful number in the whole post, and it explains the entire category:
wrong answers sent back -> 38% get fixed
right answers sent back -> 30% get broken
Re-answering is not a repair operation. It is a coin flip weighted slightly toward repair. So the value of a gate is set almost entirely by how many of the items it flags were actually wrong, which is precision. Recall barely enters into it, because an item you failed to flag simply stays as it was, and an item you flagged wrongly has a 30% chance of being actively damaged.
At a 20% budget, Jev routed 403 items, of which 216 were already correct. ZTC routed 403 items, of which 195 were already correct. Twenty-one items of difference in what got flagged, and the whole 1.4-point swing lives there.
AUC cannot see any of this. AUC asks how well a scorer sorts wrong answers toward the bottom across the entire distribution. It says nothing about the composition of the specific slice you decided to act on. Two scorers with identical AUC can have completely different precision at the threshold you happen to deploy at, and the threshold is a business decision made by your retry budget, not by the model.
Two more findings worth carrying around
Bigger is not better on this axis. Within one family, the 397B model beat the 27B overall, and then reversed hard on scientific reasoning: 27B scored 0.7410, 397B scored 0.6287. A model fourteen times larger scored 0.11 lower on that domain. The post pairs this with a separate observation of a 180B-class model at 0.7146 landing below a 4B at 0.7284, and draws the conclusion I think is right: verification quality tracks representation geometry, not parameter count. If you have been assuming the bigger judge is the safer default, stop.
How you aggregate changes the number more than most model choices do. Pooling all 2,018 items into one AUC gives systematically higher figures than averaging per domain first, because score scales differ between domains and pooling rewards that difference rather than discrimination. On the author's own system the pooled figure reads 0.80 where the per-domain mean reads 0.73. They published the lower one. Seven points of apparent performance sitting in an aggregation choice is enough to explain most of the gap between any two vendor charts you have compared this year.
Put this into practice
The genuinely useful thing here is not the ranking, which will be stale in a month. It is that the grading code is published even though the source items are not, because their licences forbid redistribution. That means the protocol is runnable against your own private set, and a number you compute that way is comparable to the board's.
Here is the lowest-friction version, and it fits in an afternoon.
Get a hundred to five hundred labeled items. Real questions from your system, real answers your system produced, and a human label for whether each answer was acceptable. Five hundred is comfortable; a hundred will tell you something. You almost certainly have these sitting in a support queue or an eval spreadsheet already.
Build the baseline first, before you touch your verifier. Fit a logistic regression on surface features of the answer only: character length, token count, digit count, number of newlines, whether it contains a bullet or numbered list, whether it contains a code block. Ten features is plenty. This takes about fifteen lines of scikit-learn. Score its AUC on held-out items. That number is your bar.
Then score the same items with whatever verifier you are paying for. If it does not clearly clear the baseline, you have your answer, and it cost you an afternoon instead of a quarter.
Finally, and this is the step people skip, simulate the gate. Pick your actual retry budget, say the bottom 15% or 20% of scores. Count how many items in that slice were already correct by your labels. That fraction is the thing that determines whether the gate helps, and you can compute it without running a single retry. If most of what you are about to re-answer was already fine, you are paying to break things.
If you want the full version, take the re-answer step too: send the flagged slice to your stronger model, label the new answers, and measure your own fix rate and break rate. The 38/30 split is one team's number on one item set. Yours will differ, and the direction of the difference tells you whether escalation is even the right remediation for your domain.
Honest limitations
This is one test set and one study, and the case against it is real.
There is an affiliation question the post does not answer, and I want to be precise about what is and is not established. Two of the three systems clearing 0.70 are ZTC at 397B and ZTC at 27B, both credited in the table to a vendor called VIDRAFT, and the underlying model repositories sit under a third name, FINAL-Bench. The account VIDRAFT-lab appears in the row of contributor avatars under the article's byline. The post itself never states an affiliation between its author and VIDRAFT, never calls ZTC its own entry, and the one place it writes "our own system" is about its aggregation method rather than about owning a row in the table. So this is not a disclosed conflict, and I am not claiming it is one. It is a question the page leaves open while ranking VIDRAFT first, and open questions about who built the winner are the reason to treat the protocol as the contribution and the ranking as a claim.
The gating experiment is narrower than the leaderboard. One escalation target, one item set, ties averaged over 200 seeds. The post labels it "a mechanism, not a universal constant," and that hedge should survive being quoted, which is why I am quoting it.
The axis is specific and it disqualifies some entrants unfairly. This test set supplies no grounding document, so it measures factual verification from the answer alone. Vectara's HHEM-2.1 scored 0.4852 and decides whether an answer follows from a supplied document, which is a different job. Laya's own README publishes results beating Jev on Convai's English typed-decision benchmark; here Laya scores near chance. Both can be true, because they are different tests, and that gap is the whole argument for having a shared one rather than a verdict on Laya.
Four published reproductions would not run from their released artifacts, with a missing classifier head, unsupported architectures, an incomplete tokenizer. They appear on the board under "did not run" with links, which I think is the right call and which also means the board is not a complete census of the category.
And the items themselves are not redistributable. You cannot reproduce the exact table. You can only reproduce the method, which is the part worth having.
What to do with this
The thing I would not do is treat this as a shopping list. The ranking will move, the entrants will change, the top of the table is a statistical tie between two systems that most readers cannot deploy interchangeably, and the affiliations behind the leading entry are not spelled out.
The thing I would do is stop accepting "we added a verifier" as an architectural statement. It is a measurable claim, and now there is a published protocol for measuring it and a baseline that costs fifteen lines of code to build. A gate that does not clear answer length is not filtering for correctness, and a gate that clears it might still be routing mostly-correct items into a process that breaks three of every ten.
Both of those are things you can find out about your own system this week. Neither requires anyone's leaderboard.
Sources: Inside the JEV Ecosystem: 13 Answer Verifiers on One Test Set, Hugging Face community post published 2026-09-20. Typed Decision Leaderboard Space. All figures quoted above were verified line by line against the post's own tables.
Medium metadata
- Title: The Typed Decision Leaderboard Put a Length-and-Formatting Baseline Above Eight of Thirteen Answer Verifiers
- Subtitle: What a shared test set of 2,018 items says about the judge sitting in your pipeline, and why AUC is the wrong number to deploy on
- Tags: AI Agents, LLM Evaluation, Machine Learning, AI Engineering, Benchmarking
- Suggested reading time: 8 minutes
- Canonical: import from the fervorai.dev URL