How a Product Manager Can Evaluate the Quality of an AI Product: A Guide to Evals

Evals, the practice of assessing the quality of an LLM product, have suddenly become the most discussed skill among product managers at AI companies. Senior executives at Anthropic and OpenAI openly call the ability to build evaluation systems a core competency for any product manager working with language models. On Lenny Rachitsky's podcast, researchers Hamel Husain and Shreya Shankar explained how a PM should approach evaluating an AI product, and why intuition does not work here.

What an eval is and why you need one

An eval is a systematic check of how well a language model handles a specific task in the specific context of your product. Unlike classic software testing, where an answer is either right or wrong, in LLM products an answer almost always falls somewhere in between. A single query can produce dozens of different but equally acceptable answers, and the PM's job is to work out which one is best for a particular user in a particular situation.

Most teams start by judging models subjectively: they look at a few examples and draw a conclusion. That works for simple features but breaks down completely at scale. When a product handles a million requests a day, manual review is impossible. You need a system that runs automatically and reproducibly.

Three levels of AI product evaluation

The experts recommend building evals in three successive layers.

The first is defining success criteria. Before measuring anything, the PM must answer the question: what does a "good answer" mean for our product? It might be factual accuracy, consistency with the brand's tone, length, structure, absence of toxicity, or safety. Without this step, any metric is meaningless: you will be measuring something other than what matters to the user.

The second level is assembling a "golden set." This is a collection of example queries with ideal answers, written by hand or selected from real data. The model is tested against this set with every update. The quality of the golden set directly determines the quality of the entire evaluation system, which is both the main difficulty and the main responsibility of the PM.

The third level is automating evaluation. At this stage the team builds a pipeline: a new version of the model or prompt is run through the golden set, and results are compared against the references, either automatically or with a judge model, meaning another LLM that grades the answers. A regression shows up immediately in the numbers, rather than surfacing in user reviews a week after release.

Why a PM cannot delegate this to engineers

The temptation to hand evals to the technical team is strong, but it is a mistake. Evals are product decisions: what matters to the user, what the user considers a good answer, and what trade-offs we are willing to make for speed or cost. An engineer does not know why a user prefers a short answer to a detailed one, or why a "friendly but professional" tone matters more than an answer that is three percent more accurate.

It is the PM who connects eval metrics to real business outcomes. If the model became five percent more accurate but user satisfaction did not change, something is wrong with the evaluation criterion itself. Finding and fixing that mismatch is a product task, not an engineering one.

What this means for the market and careers

Two years ago the word "evals" appeared mostly in academic papers. Today it is a standard part of the roadmap of any serious AI product. Companies that have learned to measure the quality of their LLM solutions systematically gain a lasting competitive advantage: they detect regressions faster, compare models against each other more precisely, and make update decisions based on data rather than the team's subjective impressions.

For a product manager's career, the takeaway is blunt: if you work with AI products and cannot build evals, you are losing out to colleagues who can. The skill has become as essential as knowing how to work with a sales funnel or run A/B tests.