Jev as a Judge: The Most Repeatable LLM in LangChain's Benchmark Was Also the Least Accurate
LangChain measured four agent evaluators on the same five frozen runs. The judge that gave the steadiest answers got one in five of them wrong.
Buried in the results table of LangChain's agent-eval benchmark, published September 20, is a pairing nobody has pointed at yet. Claude Sonnet 4.6 produced the most consistent quality scores of the three LLM judges tested, varying 92x more than the decision model rather than the 433x and 913x of the two GPT-5.6 variants. It also agreed with the human oracle on 80.0% of pass-or-fail decisions, the worst binary accuracy in the field by sixteen points. GPT-5.6 Terra ran the numbers in reverse: the highest variance of any judge in the test, 913x, and 99.8% agreement with the human.
Two properties that agent teams routinely treat as the same thing moved in opposite directions on the same five inputs.
That is a small result on a small corpus, and the authors say so in the first paragraph after the accuracy table. But it is the part of the benchmark that transfers, because it invalidates the mental shortcut most teams are about to take. The pitch arriving with this generation of eval tooling is that a cheaper, steadier judge is a better judge. Steadier and better are separate axes, and LangChain's own data shows them coming apart.
What was actually measured
The setup, from Daniel Shea and Seán Roche at LangChain, is deliberately small and stated plainly.
They built a weather agent on Deep Agents 0.7.15 with Tavily web search, then defined five cases in a LangSmith dataset: current conditions in Seattle, a weekend forecast for Austin, an umbrella decision for Dublin, an extended forecast for Tokyo, and a request for "Springfield" with no unique place attached. They ran the agent once per case and froze the complete output, including the final answer, evidence and tool calls. Freezing matters more than it sounds: with the agent runs fixed, the only thing that can vary between repetitions is the judge.
Then four evaluators scored those five frozen runs 100 times each, on two signals. quality is a continuous score from 0 to 1 covering grounding, search behavior and usefulness. does_pass is a binary success decision. A human reviewer labeled each response against the same rubric, and those labels became the oracle for binary accuracy.
The four judges: TypeSafe AI's Jev, a decision model that returns typed answers with probabilities rather than generating text, plus GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 running through LangSmith Gateway.
Here is the whole result set in one place, which is worth seeing together because the published write-ups separate the tables.
| Judge | Pass-or-fail accuracy | Mean quality variance | Cost per call | Latency |
|---|---|---|---|---|
| Jev | 100.0% | 0.0000149 (1x) | $0.00035 | 0.44 s |
| GPT-5.6 Terra | 99.8% | 0.01364 (913x) | $0.00289 | 2.83 s |
| GPT-5.6 Luna | 96.4% | 0.00647 (433x) | $0.00039 | 2.50 s |
| Claude Sonnet 4.6 | 80.0% | 0.00137 (92x) | $0.02811 | 2.16 s |
Sort that table by variance and you get a ranking. Sort it by accuracy and you get a different one. Among the three LLMs, the two orderings are close to inverted.
Why the inversion matters more than the winner
The headline everyone will carry away is that Jev swept: perfect binary accuracy across all 500 repeated decisions, variance two to three orders of magnitude below the field, the lowest cost, the lowest latency. Fine. It is also a five-case test run by a company co-hosting a webinar with the model's vendor two days later, and the README is candid about it: "This is a small corpus with five agent runs and one human reviewer. The result describes this experiment; it is not a general ranking of judge accuracy."
The inversion is the part that survives the sample size, because it is a claim about what variance can and cannot tell you, not a claim about which model is best.
A judge that returns 0.72 every single time on the same input has perfect repeatability. If the right answer is 0.30, it has perfect repeatability and is perfectly wrong. Variance measures only whether a judge argues with itself. It carries no information about whether the judge and reality agree. LangChain states this directly, and then the data hands over a live example: Claude Sonnet 4.6, the steadiest LLM in the test, disagreed with the human on one decision in five.
The practical failure this sets up is specific. A team picks a judge by running it a few times on the same trace and eyeballing whether the score wobbles. Wobble is visible without labels. Accuracy is not visible without labels, because measuring it means someone has to sit down and decide what the right answer was. So the property that is cheap to observe gets used as a proxy for the property that matters, and on this data that proxy would have picked the worst judge in the field.
The cost story is narrower than it looks
There is a second number worth pulling apart, because it is going to get repeated in a shape it does not support.
TypeSafe markets Jev at up to 200x faster inference and 400x lower cost than comparable LLMs on classification tasks. LangChain attributes that figure correctly, as the vendor's claim. Now look at what the actual run measured: Jev at $0.00035 per call against GPT-5.6 Luna at $0.00039. That is an 11% difference. Across the full 500-call benchmark, Jev cost $0.34 and Luna cost $0.39. Five cents.
The 400x-shaped gap in this experiment exists only against Claude Sonnet 4.6, at $0.02811 per call and $28.17 total, which is roughly 80x. Against the cheapest LLM judge tested, the cost argument for a decision model is close to noise.
What is not noise against Luna is latency, 0.44 seconds versus 2.50, and variance, 433x. Those are the real differences, and they are the ones worth designing around. If you are running online evaluation across a meaningful share of production traces, a judge that answers in under half a second and gives the same answer twice changes what your feedback loop can look like. The bill was never the binding constraint against a cheap LLM judge; the wait and the wobble were.
I think the cost framing is doing real damage here, because it is the easiest number to repeat and the least useful one. The README itself adds the qualifier that these costs "depend on the prompts, inputs, and provider pricing at the time of the run." Provider pricing moves monthly. The architectural difference does not.
Put this into practice
The genuinely useful thing LangChain shipped is not the conclusion. It is the reproduction repo, published alongside the post at danielgshea/jev-as-a-judge, with the frozen cases, the human oracle labels, and the analysis code all committed.
Here is the lowest-friction path to a number that belongs to you rather than to a vendor.
Start by building the oracle, not the judge. Pick five real traces out of your own product. Not synthetic ones, not the five happy paths, and include at least one genuinely ambiguous case the way LangChain included "Springfield" with no state attached. Freeze the complete agent output for each: final answer, evidence, tool calls. Then sit down and label them yourself against a written rubric. This is the unglamorous hour that makes everything downstream mean something, and it is the step people skip.
Then clone and swap. The repo needs Python 3.13+ and four keys: TAVILY_API_KEY, TYPESAFE_API_KEY, LANGSMITH_API_KEY, and LS_LLM_GATEWAY_KEY for Gateway model invocation.
cp .env.example .env
uv sync
uv run python src/evals/judge_reliability.py --local
The --local flag runs the repeated-judge benchmark without uploading an experiment, which is what you want while you are still swapping in your own dataset. The dataset definition lives in src/evals/dataset.py and the judges in src/evals/judges/. Replace the five weather cases with your five, point the oracle at your labels, and run it.
Report both axes, always together. Whatever judge you end up with, publish its accuracy against your oracle and its variance across repetitions as a pair. A judge at 94% accuracy with moderate variance is a usable instrument. A judge at 94% accuracy with unknown variance is a coin you have not counted. A judge with low variance and unknown accuracy is the dangerous one, because it looks trustworthy on the dashboard.
Spend the cost savings on repetition, not breadth. This is the recommendation I would push hardest. When judge calls get cheap, the tempting move is to score more dimensions: add a helpfulness judge, a tone judge, a safety judge, a citation judge. Every one of those is a new unvalidated instrument. The better purchase is running the judges you already validated more times, on more traces, so you catch a regression the day it lands instead of the week it ships.
What this does not tell you
The limits here are real and the authors name most of them.
Five cases and one human reviewer is a pilot. One reviewer means there is no inter-rater agreement figure, so the oracle itself has unmeasured error, and every accuracy number in that table inherits it. A second labeler disagreeing on one of five cases would move the numbers by 20 points.
The LLM judges ran with no temperature, top-p, seed, or max tokens set, so provider and gateway defaults applied. Several of those judges would look more repeatable at temperature 0, and the benchmark does not test that. This is a fair comparison of default-configured judges, not of best-configured ones, and teams running LLM judges in production usually pin temperature.
The experiment metadata did not capture the hosted Jev service version. That means the run is not exactly reproducible: you can rerun the code against a moving endpoint and get a different model. For a benchmark whose selling point is repeatability, that is an awkward gap, and it is one the README discloses rather than hides.
And the variance result is explicitly not causal. LangChain offers the hypothesis that a model trained to return bounded decisions fits this task better than one trained for token-by-token generation, then declines to claim it: "The result is observational, not evidence that the model architecture caused the lower variance." There is a plausible alternative nobody has ruled out, which is that Jev's output is quantized to a coarser set of values than a continuous LLM score, and coarse outputs vary less by construction.
The repo has no LICENSE file, which for a project meant to be cloned and adapted is worth knowing before you build on it.
The part you can act on today
The inversion in that table is a warning about instruments, and it generalizes past this benchmark and past this model class. Any time a measurement tool is easy to test for one property and hard to test for another, the easy property becomes the proxy, and the proxy becomes the decision. It happens with test suites that measure coverage because coverage is countable. It is about to happen with agent judges that measure variance because variance is countable.
The fix is not clever. It is five traces, one rubric, and an hour of labeling them yourself. After that you have an oracle, and an oracle turns every judge comparison from a vibe into a measurement. LangChain's whole run cost $31.80 across four judges. Yours will cost less, because you only need to find out about the one you are actually going to use.
Go build the oracle first. Everything else in this conversation is downstream of whether you did.
Sources: LangChain, "Jev-as-a-Judge for Agent Evals," September 20, 2026 · danielgshea/jev-as-a-judge · TypeSafe AI docs · LangSmith LLM-as-a-judge
Medium metadata
- Title: Jev as a Judge: The Most Repeatable LLM in LangChain's Benchmark Was Also the Least Accurate
- Subtitle: LangChain measured four agent evaluators on the same five frozen runs. The judge that gave the steadiest answers got one in five of them wrong.
- Tags: AI Agents, LLM Evaluation, Machine Learning, Software Engineering, AI Tools
- Canonical: import from the fervorai.dev URL