LangSmith Adds AI Search: Find Agent Traces by Describing the Behavior

LangChain has added an AI Search mode to LangSmith, so you can search traces in plain language. A developer describes the behavior they are looking for, and the platform returns the runs where it happened. The announcement came out on 13 October 2026. It is short: it lays out the idea but names no model, no limits and no price. Still, the idea targets a long-standing pain for anyone running agents in production.

What exactly launched

LangSmith is an observability and evaluation platform from the creators of LangChain. Every agent run is recorded as a trace: a chain of model calls, tool calls, intermediate reasoning and responses. Once an agent has been running for a while, thousands of such traces pile up.

Until now, you searched them by structured fields: status, duration, tags, tool name, cost. That works if you know in advance which attribute to look for. AI Search changes the input. Instead of a set of conditions, you write a phrase like "the user is annoyed with the answer" or "the agent repeatedly calls the same tool." The system itself determines which traces match that description.

Key facts from the announcement:

• Query format: natural language, no special syntax.
• Search criterion: agent behavior, not field values.
• Result: a set of traces in which that behavior occurred.
• Announcement date: 13 October 2026, LangChain blog.
• Not disclosed: the model that interprets queries, limits, pricing, supported query languages.

Why agents in particular need behavior-based search

In a conventional service, a failure looks like a 500 error or a latency spike. Agents break more subtly. An agent can formally complete a task and still make a dozen extra calls, ignore an instruction from the system prompt, or confidently make up data. Such a trace will have the status "success," and a status filter will not surface it.

So teams end up reading samples of traces by hand or writing a separate evaluator for each type of problem. Search by description changes the workflow. First you state a hypothesis ("it seems the agent gets confused when the user changes the requirements mid-conversation"), then you immediately see examples. From the good examples, you can later build a test set or an evaluator.

For LangChain, this is a logical move. Competitors in LLM observability, from Langfuse to Arize and Braintrust, are also moving toward semantic analysis of traces. A product built on data about what agents did naturally adds a layer that lets you query that data.

What remains an open question

Several things are unclear from the announcement, and the feature's real value depends on them.

First, accuracy. Semantic search over long traces with nested calls may miss relevant results or return noise. The miss rate has not been published yet.

Second, scale. It is unknown whether the system searches the whole project or a selected time window, and how much it costs in tokens.

Third, privacy. Traces often contain user data. Teams with strict requirements will want to know which model reads their traces and where it runs.

How to use it in practice

If your agents already write traces to LangSmith, it makes sense to test the feature on known problems. Take three to five cases you previously found by hand, describe them in phrases, and compare the results with what you already know. That quickly shows you recall and noise.

Phrase queries around an observable action, not a judgment: "the agent called search more than three times in a row" is more reliable than "the agent performed badly." It is worth saving the traces you find into a dataset for regression tests, so a one-off finding becomes a permanent check.

One last point. Until LangChain publishes figures on quality and cost, treat AI Search as a quick way to scout, not as the single source of truth about agent behavior. Critical scenarios are still better covered by explicit evaluators and metrics.