LangSmith Trajectories and the Tool Set Your Trace Export Forgot to Record

Turning agent traces into fine-tuning data breaks on a detail almost no observability format stores, and the replay evaluation that comes back measures something narrower than it sounds

LangChain shipped five things on September 24, and buried in the fine-tuning announcement is a sentence that indicts most agent observability setups. Describing why a naive export of an agent session is not good enough for supervised fine-tuning, the post notes that tool availability often changes as the agent works, cites deferred tool loading as the example, and says that exporting the final message list loses that nuance.

Read that twice, because it is a claim about your existing logs. If your agent uses on-demand tool discovery, the list of tools the model could call in turn three is not the list it could call in turn eleven. A flat message transcript records what was said. It does not record what was possible. And supervised fine-tuning needs the second one, because you are teaching a student model to make the choice the teacher made, from the options the teacher actually had.

Most people will never notice this, because most people never try to train on their traces. That changed this week.

The thing that is easy to miss

Anthropic's advanced tool use post from November 2025 laid out the mechanism. You hand the API your full tool library, mark the ones you do not need up front with defer_loading: true, and those definitions stay out of the model's context entirely until it searches for them. The savings are real: Anthropic reported tool definitions consuming 134K tokens before optimization internally, and a five-server MCP setup burning roughly 55K tokens on definitions before a conversation starts. With the Tool Search Tool, only the search tool plus three to five discovered tools land in context, and their internal MCP evaluations moved Opus 4 from 49% to 74% and Opus 4.5 from 79.5% to 88.1%.

Every one of those numbers is a good argument for deferred loading. None of them is the point here.

The point is what deferred loading does to the record. A session that starts with one tool in context and ends with nine is nine different decision problems, and a transcript that lists all nine at the top, or lists none of them, describes neither. If you fine-tune on that, you are teaching a model to pick GitHub's createPullRequest in a turn where, at the time, it could not see that the tool existed. You have taught it to guess, and rewarded the guess.

This is the class of bug that never shows up as an error. The pipeline runs. The loss goes down. The model gets worse at something you were not measuring.

What a trajectory is, precisely

LangSmith's documentation is unusually clean on the data model, so it is worth quoting the shape rather than paraphrasing it. A run is one unit of work, equivalent to an OpenTelemetry span. A trace is the collection of runs for one operation, capped at 25,000 runs. A thread is a sequence of traces making up a multi-turn session, grouped by a thread_id. And a trajectory is not a fourth recording. It is a projection over the traces in a thread: the human, AI and tool messages, each appearing once, in the order it first appeared, with the nesting of runs removed.

The docs put the tradeoff in a table, and the row worth memorizing is the last one. Reach for a trace when you are debugging why one operation failed or ran slow. Reach for a thread when you want timing and nesting intact across turns. Reach for a trajectory when you are reading what was exchanged, without the execution detail.

So the flattening is deliberate, and it is not lossy in the direction people assume. The runs are still there. The trajectory is a view, and you can drop back into the tree for any step. What the projection buys you is a format where each message exists exactly once, which turns out to be exactly what you need for two jobs that have nothing to do with debugging: handing a session to a domain expert who should not be reading retry metadata, and building a training set.

Anyone who has asked a domain expert to evaluate agent behavior from a run tree knows how that goes. The tree is the right artifact for the engineer and the wrong artifact for the person who actually knows whether the agent asked the right follow-up question.

Where the tool set gets preserved

The smithtune CLI is where the abstract claim becomes a checkable behavior. Its README states that preparation checks each trajectory and the tools available at each assistant turn, and that each supported assistant answer becomes one training target, with its preceding context and the tools available at that call.

That is the whole argument, stated as an implementation detail. One training example is not "message, response." It is "message, response, and the set of options that existed at that moment."

A few other details in that README are worth your attention before you run it:

Recorded system messages are preserved. Reasoning is omitted by default.
Trajectories that are unsupported or too long are excluded without truncation, and listed in rejected.json. Read that file. It is the honest record of what your pipeline could not use.
Preparation assigns roughly 80% to training, 10% validation, 10% test, and each source trajectory stays inside one split.
The council that votes on which trajectories to keep runs on DeepSeek V4.1 Flash and GLM-5.3-Flash on Baseten by default, and smithtune ships no default rubric at all. triage requires you to supply one.

That last item is a design decision I respect. A tool that shipped a default "keep the good ones" rubric would get used with the default, and the default would be wrong for everybody. Making you write the rubric after reading a varied sample of your own pulled trajectories forces the one step where the human judgment actually lives.

The results, and what they are results of

LangChain ran this on two of its own agents and published numbers. On a stripped-down version of an Engine agent, measured on a subset of their internal IssueBench: GPT-5.6 Sol at 87.0, Kimi K3 at 90.0, Kimi K3 plus SFT at 96.0. On code review with Qwen-3.8-27B, F1 moved from 48.9% to 53.7%, precision from 62.9% to 81.5%, recall flat at 40.0%, with 29.8% fewer model calls and 29.4% fewer tool requests per review.

The recall number staying flat while precision jumps is the interesting part. The tuned model did not find more bugs. It stopped flagging things that were not bugs, and it did so in about two-thirds the calls. If your review agent is expensive and noisy, that is the exact shape of win you want. If your problem is that it misses things, this did not fix that.

And the detail LangChain kept in the post, which they did not have to: an earlier, less selective training set reduced F1 after fine-tuning. They went back and added a review stage to the curation pipeline. That is the honest version of "data selection matters," and it is more useful than the table.

Put this into practice

Lowest-friction path, in order, and you can stop at any step:

Step one, costs nothing. If you already trace to LangSmith, open a thread and switch to the trajectory view. Read one long session end to end. You are checking one thing: can a person who did not build this agent follow what it did? If yes, you have a review workflow you did not have last week. Trajectories are on all plans in the US at launch.

Step two, still costs nothing. Before installing anything, write the filter. smithtune's README makes the point that selecting an agent by name or filtering out errors alone does not establish training quality, and it shows you how to test a filter with the LangSmith CLI over your intended window for free. Filters match root runs, and each match pulls in its whole thread. Default pull window is the last 24 hours, so always set --start-time and --end-time or you will pull yesterday and think that is your data.

Step three, the diagnostic run. Install the CLI, run smithtune acknowledge-data-rights and smithtune doctor, then pull with --no-triage and read the summary. pull makes no model calls. What you are looking for is the exclusion reasons and, after prepare, the contents of rejected.json. If a large share of your sessions land there, you have learned something about your tracing setup that is worth more than the fine-tune would have been.

Step four, if the data survives. plan before train, both with the same flags, because plan is a preview and saves nothing for train. Every paid command requires --confirm, which is a good habit the tool enforces for you.

Do step three even if you never intend to train anything. The rejection list is a free audit of whether your traces record what you think they record.

Honest limitations

Start with the one the tool states about itself, because it changes how you should read the numbers above. From the smithtune README: replay predicts the next response or tool call from recorded context, generated tool calls are not executed, and scores measure agreement with recorded behavior, not end-to-end task completion. They also do not affect checkpoint selection, which is made on lowest validation loss.

So the evaluation is an imitation score. It answers "would the tuned model have done what the recorded run did," and it answers that with a judge model. It does not answer "did the task get done," and it cannot, because nothing runs. If your recorded trajectories contain a successful-looking path that was actually wrong, a higher replay score means you have successfully taught the model to reproduce the wrong path. The council review exists to catch that, and the README says plainly that council review helps assess quality and does not guarantee good training data.

Second: I have not run this. Everything above is a reading of LangChain's posts, the LangSmith documentation, and the smithtune README, checked against Anthropic's advanced tool use post for the deferred-loading mechanism. I have not measured a trajectory export, not run a council, and not trained a checkpoint.

Third: every performance figure in this piece is a vendor measuring itself. IssueBench is LangChain's internal benchmark. The code-review set is LangChain's internal set of their own pull requests. Nobody outside has reproduced any of it, and the post does not publish the benchmark, the sample size, or the judge configuration behind the Engine table.

Fourth: smithtune calls itself an early beta whose commands and saved-directory formats may change between releases, and LangChain's announcement post states it currently supports supervised fine-tuning. Public Beta on the LangSmith side, and self-hosted teams wait for a later release.

Fifth, and this is a gap rather than a flaw: I could find no published comparison of fine-tuning on trajectories with per-turn tool context against fine-tuning on a flat message export of the same sessions. That is the experiment that would turn the argument in this article from a well-reasoned claim into a measurement. Until someone runs it, the deferred-loading problem is a mechanism you can verify and a magnitude nobody has published.

What you actually do with this

The fine-tuning story is the headline, and it is probably not the part that matters to you this quarter. Most teams are not going to train a model on their agent sessions in the next six months.

The part that matters is smaller and arrives sooner. Your traces are either recording what the model could see at each turn, or they are not, and that question has a determinate answer you can go get today. If the answer is no, you have a debugging format, and that is fine, and you should know it is all you have. If the answer is yes, you have an asset, and everything downstream of that, review workflows, path-level evaluation, post-training, is available to you whenever you want it.

Go read one trajectory. Then go look at what your own exporter writes for a turn where a tool was discovered mid-session. Whatever you find, you will know something about your setup that a dashboard was never going to tell you.

Sources: LangSmith Fine-Tuning announcement, Trajectories announcement, LangSmith observability concepts, smithtune README, Anthropic on advanced tool use.

Medium metadata

Title: LangSmith Trajectories and the Tool Set Your Trace Export Forgot to Record Subtitle: Turning agent traces into fine-tuning data breaks on a detail almost no observability format stores Tags: AI Agents, Machine Learning, LangChain, Fine Tuning, Observability