Haiku Returned PASS and a Major Flag in a Review of Ten Articles: Breaking Down the Report

In a review of ten articles, Haiku returned a PASS verdict in one report and marked an issue in the same report as major, meaning significant. The authors of the Habr write-up did not pick which signal to believe. They went through the report with the sources in hand and sorted the flags into those that led to fixes and those they rejected. They also noted separately what context the model itself was missing.

An important caveat: the announcement does not include the full text of the analysis. What is known is the sample, the type of contradiction, and the general structure of the conclusions. The specific errors, the prompt wording, and the share of accepted fixes are not disclosed in the public part, so they are not guessed at below.

Where the report's logic breaks down

PASS and major describe different things. The former usually means the final decision: "the material can go through." The latter describes the severity of an individual finding. In a sound setup, a major flag should block a PASS. If it does not, two fields in the reviewer's single response disagree with each other.

The causes of this kind of mismatch in LLM reviews are known from general practice, though for this case they remain hypotheses:

• the verdict and the list of flags are generated sequentially, and the model does not always reconcile them;
• the overall assessment answers the question "does it read coherently and add up," while a flag records a local fact;
• a compact model sees the text of the article but not the repository, configs, and versions the article relies on.

The last point echoes the authors' own conclusion: some of the objections come down to missing context on the model's side, not to an error in the text.

What the check against the sources showed

Checking against the primary sources produced three categories of results. First: flags that held up, with articles that were corrected as a result. Second: flags that were rejected because the model made a mistake or misread a passage. Third: cases where the validity of a flag could not be assessed without data the reviewer never received.

The third category is arguably the most useful. It shows not the quality of the model in general but the limits of applicability of a specific "model plus input data" combination. If a reviewer systematically misses on facts from external sources, the fix is to widen its input, not to swap the model.

Key parameters of the experiment:

• reviewer: Haiku, a small model in the Claude lineup, going by its name;
• sample: 10 articles;
• recorded contradiction: a PASS verdict alongside a major flag;
• verification method: manual comparison of the report against the sources;
• outcome: fixes partly accepted and partly rejected, with context gaps identified separately.

Ten articles are too small a sample for statistics. As an observation of pipeline behavior it works; as a benchmark it does not.

How to build such a review in practice

The main takeaway for anyone putting an LLM on proofreading text or code: do not trust a standalone verdict field. If the report includes flag severity, compute the final outcome with a rule outside the model. Any major or critical flag sends the material to manual review, regardless of what the PASS says.

The second measure: give the reviewer the context the material under review refers to. That means source files, library versions, and links to documentation. If the model cannot see something, it should return an "insufficient data" status instead of a confident PASS. This status should be spelled out explicitly in the prompt.

The third measure: log mismatches such as PASS plus major as a separate metric. Their share shows how far the report format can be trusted and signals when it is time to make the scheme more elaborate, for example by adding a second pass or a stronger model for disputed cases only.

A cheap model remains a workable reviewer as long as its verdict is checked by rules and not taken on faith.