LangSmith научился сравнивать дообученные open-source LLM с автоматической оценкой

LangChain has published a guide for testing and comparing fine-tuned open-source language models in LangSmith — a platform for evaluating, monitoring, and debugging LLM applications. The guide describes how to run multiple fine-tuned models in parallel, automate result evaluation, and select the best version for deployment based on the data.

Why test fine-tuned models

Fine-tuning is the adaptation of a base model on specialized data. In 2025, teams are actively fine-tuning Llama, Mistral, Falcon, Phi, and other open-source models for specific tasks: enterprise search, code generation, customer request classification, answers in narrow domains. The problem is that fine-tuning results are not guaranteed.

A model can overfit on training examples and lose general competence. It may not outperform the original model with a good system prompt. Finally, two training runs with the same data but different hyperparameters can produce fundamentally different quality — and understanding this without testing is impossible.

Add to this the fact that model training takes hours and costs tens or hundreds of dollars. Without an evaluation system, teams either guess the winner or spend as many resources on manual answer verification. LangSmith offers a third way.

What LangSmith can do

The platform organizes testing into a unified pipeline:

• Datasets — upload a set of test examples with queries and reference answers
• Parallel runs — multiple models are run on the same dataset simultaneously
• Automatic evaluation — LLM-judge or custom metric function: accuracy, F1, format compliance, custom metrics
• Comparison report — a single table with results across all metrics
• Tracing — for each call, you can see input data, model output, and the received evaluation

Beyond direct comparison "model A vs model B," LangSmith allows comparing different checkpoints of a single training, different versions of training datasets, and different fine-tuning strategies within a single experiment.

Example: three checkpoints on one dataset

A typical scenario: a team fine-tunes Llama 3.1 8B on corporate FAQs in three variants — with different numbers of epochs and learning rates. In LangSmith, a dataset is created from 200 real user questions.

All three versions are run in parallel, the LLM-judge evaluates each answer on a 1-5 scale. The resulting table immediately shows which checkpoint delivers the best quality with acceptable response speed. Without LangSmith, such a process requires custom scripts for each run and manual consolidation of results.

Built-in tracing adds another advantage: for each "bad" answer, you can open its trace and understand what went wrong — the model didn't understand the question, answered in the wrong format, or the evaluator made a mistake. This approach is especially important when iterating over the training dataset: seeing quality dynamics after each change in the data is the foundation of fast and meaningful improvements.

What it means

Fine-tuning of open-source models is turning from a niche practice of ML researchers into a standard tool for product teams. LangSmith closes a key gap: between training and deployment there is now a structured evaluation stage with numerical metrics. This reduces the risk of deploying a version worse than the original and makes the decision-making process transparent — not "it seems better," but "here are the numbers."