Codex stopped searching the web in a Russian-language test, with success falling from 100% to 0%

You ask an agent to fix workout exports to a watch. It changes fields, inserts heart-rate values and presents a tidy patch. But the watch cannot read that format. The answer is on a forum, and the agent never even looked there. In an experiment published on Habr, this behavior emerged after the task was translated from English into Russian. The practical takeaway is simple: when a fix looks odd, check the logs of source lookups and give the agent any known project-specific quirks.

The author tested whether the language of a task affects how agents work, and whether she needed to translate the case studies in her worklore project, a library of analyzed engineering cases. The experiment comprised 1350 runs: six tasks, nine agent configurations, five ways of presenting each task and five repetitions of every combination. This is a small experiment conducted by its author, but it exposes a failure that is easy to miss in everyday work.

Where the agent started guessing

The clearest example involved the format for a heart-rate range in a Garmin workout. Under the experiment's conditions, the relevant quirk was described on a forum but absent from the documentation. With English-language instructions, Codex at a low reasoning level searched the web for the format every time and succeeded in all five attempts.

After the instructions were translated into Russian, the same agent did not search the web once. Instead of checking, it tried a series of plausible changes: removing the structured target, switching to absolute heart-rate values and adding new fields alongside the old key. None of the attempts succeeded. Success fell from 100% to 0%, although each percentage here represents just five runs.

Kimi K3.1 showed a similar pattern on a task involving an OAuth provider's quirks. The English version produced five successful solutions out of five; the Russian version produced none. The agent skipped the search and retained incorrect assumptions about access parameters and token lifetime. The author did not investigate why the language changed the decision to use a tool.

The missing fact proved more useful than translation

Adding a note about a similar case resolved a substantial share of the problems. For example, if a provider issues tokens that never expire, it helps to tell the agent explicitly. It then does not have to infer a particular service's behavior from general knowledge of OAuth.

Across four tasks where the note supplied the missing fact, success rose from 47% to 94% for English-language instructions and from 35% to 92% for Russian-language instructions. On two control tasks, where the answer was already in the documentation or a log, the improvement was small. This supports a practical approach: preserve integration quirks discovered by the team and pass them to the agent along with the task.

Translating the note itself made almost no difference. With Russian-language instructions, the Russian note produced roughly two percentage points more successful solutions, but the uncertainty interval included no effect. Identifiers, paths and error messages remained in English in both versions. For this kind of technical memory, precise details mattered more than the language used to describe them.

What to check in your own workflow

It is too early to conclude from these data that Russian causes poor agent performance. The experiment covered just six specially prepared tasks, one non-English language and five repetitions per combination. It also compared complete configurations: the model, its harness and the available tools. The findings cannot automatically be applied to other versions or settings.

There is another limitation: in some runs, agents found worklore's own published case studies through search. This changed the context available to them. The author separately recalculated the overall results excluding those runs; the main observation held, but a clean comparison was no longer available for some model-and-task combinations.

Developers can take one familiar problem that requires an external source and run it several times in two languages. Check the tool calls and the code validation results. If the agent regularly skips searches, explicitly require it to check the source and repeat the comparison. Attach any known fact to the task from the outset, along with a link or excerpt. Product managers evaluating an agent for their team should also include the language of everyday tasks in their assessment: identical meaning does not guarantee an identical path to a solution.