LangSmith can now compare fine-tuned open-source LLMs with automated scoring

LangChain has published a guide to testing and comparing fine-tuned open-source language models in LangSmith, a platform for evaluating, monitoring and debugging LLM applications. The guide describes how to run several fine-tuned models in parallel, automate the scoring of results and use the data to choose the best variant for deployment.

Why test fine-tuned models

Fine-tuning means adapting a base model on specialized data. In 2025, teams are actively fine-tuning Llama, Mistral, Falcon, Phi and other open-source models for specific tasks: enterprise search, code generation, classification of customer requests, and answers in a narrow subject area.

The problem is that the outcome of fine-tuning is not guaranteed. A model may overfit to its training examples and lose general competence. It may fail to beat the original model paired with a good system prompt. And two runs on the same data but with different hyperparameters can produce fundamentally different quality, which is impossible to know without testing.

On top of that, training a model takes hours and costs tens or hundreds of dollars. Without an evaluation system, teams either guess the winner or spend just as many resources checking answers manually. LangSmith offers a third path.

What LangSmith can do

The platform organizes testing into a single pipeline:

• Datasets: upload a set of test examples with queries and reference answers
• Parallel runs: several models run on the same dataset at the same time
• Automated scoring: an LLM judge or a custom metric function, such as accuracy, F1, format compliance or non-standard measures
• Comparison report: one table with results across all metrics
• Tracing: for each call you can see the input, the model output and the score it received

Beyond a direct "model A vs. model B" comparison, LangSmith lets you compare different checkpoints from the same training run, different versions of the training dataset and different fine-tuning strategies within a single experiment.

Example: three checkpoints on one dataset

A typical scenario: a team fine-tunes Llama 3.1 8B on corporate FAQs in three variants, with different numbers of epochs and learning rates. In LangSmith, they create a dataset of 200 real user questions. All three versions run in parallel, and an LLM judge rates each answer on a scale of 1-5. The summary table immediately shows which checkpoint delivers the best quality at an acceptable response speed.

Without LangSmith, this process requires custom scripts for each run and manual merging of results.

Built-in tracing adds another advantage: for every "bad" answer, you can open its trace and see what exactly went wrong. Perhaps the model misunderstood the question, answered in the wrong format, or the judge scored it incorrectly.

This approach matters most when iterating on the training dataset: seeing how quality changes after each data modification is the basis for fast, informed improvements.

What this means

Fine-tuning open-source models is shifting from a niche practice of ML researchers to a standard tool for product teams. LangSmith fills a key gap: between training and deployment there is now a structured evaluation stage with numerical metrics. This reduces the risk of shipping a version that is worse than the original to production and makes decision-making transparent: not "it seems better" but "here are the numbers."