Sixtyfour: The Y Combinator Startup That Scores Every AI Agent Output

Sixtyfour, a startup founded by Saarth Shah and Christopher Price, has introduced a conceptually new approach to building AI research agents. The company, which went through the Y Combinator accelerator, built a system that pushes back against the industry's dominant practice: instead of pointing a language model at web data and blindly trusting the results, Sixtyfour built a rigorous evaluation stack. Every release of the research agent goes through a strict scoring system: it is checked against a set of questions that specialists prepared by hand, and only code that demonstrably improves the final quality metrics reaches production.

Why blind trust in LLMs is not enough

Most modern AI tools for searching and analyzing information follow a simple pattern: point a language model at web sources, get an answer and treat it as correct. Such systems are fast to build, but their reliability remains in question. Language models tend to hallucinate, inventing facts that are not in the sources. They can misinterpret text, especially when the information is contradictory or requires calculation. Sometimes a model produces logical loops that look convincing but fail under scrutiny.

Saarth Shah, Sixtyfour's founder, chose the opposite path: control every step of the agent. He keeps a precise scoreboard, literally a table that records the quality of each agent version. If the score improves, the version ships to production. If the score drops, the team rolls back the changes, analyzes what went wrong, and reworks the logic or parameters. This differs radically from the familiar "ship it and hope for the best" approach.

How the evaluation system works

The core of Sixtyfour's evaluation stack is a set of control questions prepared by hand by specialists with experience in research and data analysis. The questions reflect real working scenarios: finding precise information in large volumes of text, checking an article for factual errors, synthesizing data from several sources, and verifying claims through cross-search.

Every new build (version) of the research agent is run against this question set. The system records three critical parameters: Did the agent give the correct final answer? How complete and detailed is the answer? Does the agent's reasoning contain logical errors or hallucinations? These parameters feed into the final quality score.

The methodology resembles unit testing in classical software development, but applied to the quality of an AI agent's judgment rather than the correctness of code. When a developer improves the prompt or the agent's architecture, that change must not lower the overall score on the control set. Ideally, it should raise it. This ensures that no change reaching production degrades the system's reliability.

Human oversight of machine judgment

A critical detail of Sixtyfour's philosophy: the evaluation system stays in human hands and is not handed entirely over to language models. People prepare the questions for the control set, specialists who understand real risks and edge cases. This reduces the risk of cascade failures, where an error in model generation N produces hidden errors in generation N+1 because the model was trained on flawed data.

What this means for the industry

Sixtyfour's approach points to a turning point in AI agent development. The industry is gradually moving from marketing claims ("our agent is the smartest") to objective, reproducible quality metrics. It is a shift from an era of promises to an era of proof.

This matters especially for high-stakes research and analysis uses: financial forecasts, medical recommendations, legal document analysis, areas where an agent's mistake is costly or can harm a person.

Other startups and research labs are likely to draw inspiration from Sixtyfour's methodology. More broadly, though, validating and evaluating AI agents is not a narrow business problem for one startup but a systemic problem for the entire industry building large language models and their agent applications.