The Cheap AI Judge Has a Retention Bill: What LangSmith's Jev Integration Actually Turns On
LangChain shipped TypeSafe as a model provider on September 21. The per-call price dropped by two orders of magnitude. Two retention defaults came along with it, and neither is in the announcement's headline.
The pitch for a typed decision model as an eval judge is almost too clean to argue with. LangChain ran the comparison last week and published it: Jev matched a human reviewer on every decision in the test, averaged 0.44 seconds per call against 2.16 to 2.83 seconds for three LLM judges, and cost $0.34 to run the full judgment set where Claude Sonnet 4.6 cost $28.17. On September 21 the integration landed, TypeSafe became a selectable model provider inside LangSmith, and the whole thing became six clicks.
Somewhere in those six clicks, two defaults get set. One lives in LangSmith and changes your bill. One lives at the judge vendor and changes where your production traffic goes. Neither appears in the announcement's cost comparison, and one of them has a different answer depending on which page you read.
This is the part worth slowing down for, because the shape of it generalizes well past this one integration.
What actually happens when you click Apply
LangSmith's own documentation is unambiguous about the first default. From the online evaluators guide: "When an online evaluator runs on any run within a trace, the trace will be auto-upgraded to extended data retention. This upgrade will impact trace pricing."
The usage and billing page gives you the numbers behind that sentence. LangSmith has two retention tiers. Base traces live 14 days. Extended traces live 180 days, a maximum that changed for SaaS customers on September 14, 2026. And the invoice does not show you base traces and extended traces as two counts. It shows "LangSmith Traces (Base Charge)," which includes everything, and "LangSmith Traces (Extended Data Retention Upgrades)," which counts the upgrades separately, because an upgrade can happen in a later billing period than the trace it applies to.
So the accounting works like this. You score a trace with a judge that costs $0.00035. The scoring act promotes that trace from 14-day to 180-day retention, and the promotion is its own billable event on its own line. The judge is cheap. The judging is not free.
LangChain is not hiding this. The docs state plainly that "Retention extension is enabled by default for new online evaluators and automation rules" and that you can opt out when configuring each evaluator, with the caveat that the opt-out is only available when the project's default retention is the base tier. They even publish their reasoning: traces that got evaluated are "fundamentally more interesting than other traces," and they would rather charge an order of magnitude less for traces nobody meaningfully interacts with.
That is a defensible pricing philosophy. It was also built for a world where evaluating a trace was expensive enough that you sampled. The entire argument for a typed judge is that you stop sampling. LangChain says so in the post: "With Jev-as-a-judge, you can score every trace instead of a sample of them."
Score every trace, and by default upgrade every trace. The feature that made evaluation cheap interacts with a pricing model that assumed evaluation was selective, and the interaction runs in exactly the wrong direction.
The second default has two different answers
Step three of LangChain's setup walkthrough contains this sentence, written by Winston Huynh: "Note that TypeSafe does not currently offer zero data retention, so prompts and outputs sent for evaluation may be retained by the provider."
I went to check it, because zero data retention is the kind of thing that is usually a sales conversation rather than a product state. TypeSafe's own legal index page says something different: "We also offer zero data retention (ZDR) for enterprise customers. Contact privacy@typesafe.ai to learn more."
Both statements can be honest. LangChain's integration ships with self-serve API keys, and whatever an enterprise ZDR arrangement is, it is not something you get by pasting a key into a provider secrets field. The accurate version is probably "not available on the path you are about to take, available on a path involving a contract." Which is a materially different fact from "does not currently offer," and it is the difference between a blocker and a phone call.
The useful takeaway is not that anybody got it wrong. It is that the retention posture of a subprocessor arrived in your stack as a sentence inside a third party's tutorial, and you can only find out how accurate that sentence is by reading the subprocessor's own legal index. That is the diligence step, and it took me one fetch.
State for an eval is not a small thing either. TypeSafe's docs describe state as "the content you ask a System One model to evaluate," and recommend a JSON object so each part has a descriptive name. In LangSmith's flow, you build that object by mapping in run or thread variables. For an agent trace, that is user input, tool calls, tool results and final output. The whole conversation, shaped nicely and labeled clearly, leaving for a provider.
My position
Use the typed judge. The measurement is good, the latency is real, and 80x on judge cost is not a rounding error.
But stop calling it a swap. Swapping a judge implies the only thing that changes is the model behind a scoring function. What actually changes here is retention on both sides of the wire: how long LangSmith keeps the trace and at what price, and how long a new subprocessor keeps a copy of your production conversations. Those are the two decisions. The per-call price is the thing that prompted them, not the thing you are deciding.
And the general form is worth naming, because this will keep happening. When a capability gets dramatically cheaper, the systems around it are still priced and configured for the old cost. Every default in that surrounding system was tuned for scarcity. Cheap judging meets a pricing model built for expensive judging, and the collision shows up on your invoice under a line item you have never read.
Put this into practice
Four steps, in order, all doable this week.
Start on a dataset you own, not on live traffic. Create the evaluator against an existing dataset rather than a tracing project. You get the real latency and cost profile with no production traces involved. Be honest that this does not test the thing you would actually buy, since the 0.44-second number exists specifically for scoring live traffic, but it tells you whether the typed questions survive contact with your data before anything leaves.
Open the evaluator's Advanced section and look at retention extension before you save. LangSmith lets you opt out per evaluator, but only when the project's default retention is the base tier. Check which tier your project defaults to first, because if it is already extended, the opt-out will not be there and the decision was made for you upstream. LangChain's docs on this are at langsmith/evaluators under "Manage evaluator trace retention."
Set the two limits that actually cap this. There is a per-evaluator spend limit under Advanced, defaulting to the organization-wide value, which pauses the evaluator on that project or dataset when weekly spend hits it and resets Monday at 12AM UTC. Separately, and more usefully, Settings has a monthly usage limit specifically on extended data retention traces. That second one has a documented side effect worth understanding: once you hit it, LangSmith blocks evaluators that extend retention from running at all. Evaluators with retention extension disabled keep working. So the limit that protects your bill also silently changes which evaluators run, and you want to know that before it happens rather than during an incident.
Send one email about ZDR before you scale up. privacy@typesafe.ai, per TypeSafe's legal index. Ask specifically whether ZDR is available on the plan you are on, not the plan you might be on. Do this while your volume is small enough that the answer is still a design input rather than a migration.
One more, if you want the version of this that is actually cheap: use the sampling rate. LangSmith lets you filter which runs trigger an evaluator and set a sampling rate to control cost. A filter that only scores runs where a user left negative feedback, or runs that invoked a specific tool, gets you most of the signal at a fraction of both bills. The fact that you can now afford to score everything does not mean everything is worth scoring.
What I could not verify, and where this gets thin
I did not run this integration. Everything above is a reading of LangChain's announcement, LangSmith's published docs, and TypeSafe's legal index, all fetched on September 21, 2026. I have not confirmed in the UI that the retention opt-out appears where the docs say it does, and I have not seen an invoice with both line items on it.
The benchmark underneath the cost claim is LangChain's, on LangChain's product, using one agent and five frozen test cases with a single human as the oracle. LangChain says so themselves: "This was one test on one agent." The $28.17 figure for Claude Sonnet 4.6 is the full judgment set at that judge's rate, not a per-call comparison, and the ratio between judges will move with the size of your traces.
The ZDR conflict is a conflict I found, not one I resolved. I have not contacted TypeSafe, I do not know what tier "enterprise" means there, and it is possible that LangChain's sentence is precisely accurate about what a self-serve LangSmith integration can get you today. Read both sources before you repeat either.
I also have not tested the claim that matters most and that nobody has published: whether a typed judge with hand-written questions catches the failure modes your agent actually has. Jev answers questions whose answer sets you enumerate in advance. The pricing and retention analysis above is all downstream of that question being answered yes, and for your system, I have no idea whether it is.
Finally, retention tier pricing itself is a number I deliberately did not state, because LangSmith's docs point to the pricing page rather than publishing the multiplier inline. Go look it up for your plan. The structure is the point; the multiple is yours to find.
Where this leaves you
You do not need to have an opinion about System One models to get value out of the last twenty minutes of reading. You need to open two pages before you turn on any evaluator: the docs page describing what the feature does to your data retention, and the legal index of whoever is about to receive your traces. Both existed before the announcement. Neither was in it.
That is the habit worth keeping, and it generalizes past judges, past LangSmith, and past this week. The announcement tells you what got cheaper. The docs tell you what that costs.
Sources: LangChain, "Jev is now available in LangSmith Evals" (September 21, 2026); LangChain, "Jev-as-a-Judge for Agent Evals" (September 20, 2026); LangSmith docs, "Set up LLM-as-a-judge online evaluators"; LangSmith docs, "Usage and billing"; TypeSafe docs, "Legal"; TypeSafe docs, "State". All fetched 2026-09-21.
Medium metadata
- Title: The Cheap AI Judge Has a Retention Bill: What LangSmith's Jev Integration Actually Turns On
- Subtitle: LangChain shipped TypeSafe as a model provider on September 21. The per-call price dropped by two orders of magnitude. Two retention defaults came along with it.
- Tags: LangSmith, AI Agents, Data Retention, LLM Evaluation, AI Engineering
- Canonical: fervorai.dev