Sixtyfour: the Y Combinator startup that evaluates every AI agent output

Sixtyfour, a startup founded by Saarth Shah and Christopher Price, introduced a conceptually new approach to developing AI agents for research. The company, having gone through the Y Combinator incubator, developed a system that stands against the dominant industry practice: instead of running a language model on web data and blindly trusting the results, Sixtyfour built a rigorous evaluation stack. Every release of a research agent passes through a strict evaluation system: it is tested against a set of questions that specialists prepared manually, and only code that demonstrably improves final quality metrics goes to production.

Why Blind Trust in LLMs Is Insufficient

Most modern AI tools for information search and analysis work according to a simple scheme: point a language model at web sources, get an answer, and consider it true. The speed of implementing such systems is high, but reliability remains questionable. Language models tend to hallucinate — inventing facts that don't exist in the sources. They can misinterpret text, especially if the information is contradictory or requires calculation. Sometimes a model creates logical loops that look convincing but don't hold up under scrutiny.

Saarth Shah, Sixtyfour's founder, decided to take the opposite path: control every step of the agent. He maintains a precise scoreboard — literally a table where the quality of each agent version is recorded. If the score improves, the version is released to production. If the score drops, the team rolls back the changes, analyzes where the error was, and reworks the logic or parameters. This is radically different from the familiar approach of "ship and hope for the best."

How the Evaluation System Works

The core of Sixtyfour's Evaluation Stack is a set of control questions that were prepared manually by specialists with experience in research and data analysis. These questions reflect real-world scenarios: finding precise information in large volumes of text, analyzing an article for factual errors, synthesizing data from multiple sources, verifying claims through cross-reference searches.

Each new build (version) of the research agent is run against this set of control questions. The system records three critically important parameters: did the agent provide the correct final answer? How complete and detailed is the answer? Are there logical errors or hallucinations in the agent's reasoning? Based on these parameters, an overall quality score is calculated.

The methodology is similar to unit testing in classical software development, but applied to the quality of judgment in an AI agent rather than code correctness. If a developer makes an improvement to an agent's prompt or architecture, that improvement should not lower the overall score on the control set. Ideally, it should improve the score. This ensures that each change to production doesn't damage the system's reliability.

Human Control Over Machine Judgment

A critically important detail of Sixtyfour's philosophy: the evaluation system remains in human hands, not entirely entrusted to language models. The questions for the control set are prepared by people — specialists who understand the real risks and edge cases. This reduces the risk of cascade failures — situations where an error in generation of models N produces hidden errors in generation N+1, because the model was trained on data with errors.

What This Means for the Industry

Sixtyfour's approach points to a pivotal moment in AI agent development. The industry is gradually moving from marketing claims ("our agent is the smartest") to objective, reproducible quality metrics. This is a transition from an era of promises to an era of proof.

This is especially critical for critical applications of research and analysis: financial forecasting, medical recommendations, legal document analysis — areas where an agent error is costly or can harm humans.

Other startups and research labs will likely be inspired by Sixtyfour's methodology. But more broadly: the question of validation and evaluation of AI agents is not a narrow business problem for one startup, but a systematic problem across the entire industry of developing large language models and their agent applications.