Visible CoT Does Not Universally Improve SQL and pandas Code Generation, New Study Finds

Researchers tested one of the most widely repeated prompting tips, "let the model think out loud," on a task with objective verification: generating analytical code that can be run. The arXiv preprint 2610.10639 compares SQL and Python (pandas) on identical queries. The authors found no universal accuracy gain from a visible chain of reasoning (Chain-of-Thought), and no consistent benefit from reasoning "in the task's language." The effect depends on the persona in the prompt, the target language, the model configuration, and the internal reasoning mode.

How the experiment works

Analytical code is a convenient testbed. A natural-language request is ambiguous, the model has to anchor itself to the data schema, and the output must follow the strict rules of the target language. The same question about data can be answered with either a SQL query or a pandas script. So the authors built a benchmark in which queries are matched across the two languages, and tested a common assumption: visible reasoning works better when it is written in the same representation as the answer, that is, "think in SQL" or "think in Python."

The experiment crossed several factors:

• the wording of the persona in the prompt;
• the target language: SQL or Python (pandas);
• the visible CoT format, including reasoning in the target language's representation;
• control prefixes;
• direct generation without visible reasoning;
• the model's internal reasoning configurations.

Quality was judged by code execution, not by how closely the text matched a reference. The abstract gives no numerical results or model names; those will have to be checked in the full text.

What the results show

The main finding is negative, but practical. Visible CoT does not deliver a general accuracy gain. Aligning the reasoning representation with the target language does not work as a rule either: it helps in some cases and not in others. The sign and size of the effect depend on the combination of persona, language, model, and internal reasoning settings.

The control ablations are especially valuable. They separate the influence of the reasoning content from the influence of the prompt format. Some of the quality changes may be explained by the fact that an extra template or prefix simply shifts the model's behavior, regardless of whether it reasons usefully or not. The authors put it this way: visible reasoning sometimes improves executable generation, and sometimes mostly perturbs the model's behavior. In a number of configurations, the deciding factor is the internal reasoning, not what is shown in the answer.

What teams building text-to-SQL and analytics agents should do

Don't treat a "think step-by-step" prompt as the default. Test it on your own data, as you would any other parameter. Here is a minimal set of practices that follows from the paper:

• Keep direct generation without visible CoT as a mandatory baseline, and compare against it by query execution results.
• Test the prompt together with the model, persona, target language, and thinking mode. Carrying settings over between combinations is unreliable.
• Add control prefixes to tell the benefit of reasoning apart from the effect of a long template.
• For reasoning models, check separately whether visible CoT interferes with internal reasoning.
• Don't bake advice like "think in SQL" into the system prompt without measuring it on your schema.

The limitations are also obvious. This is a v1 preprint without peer review, and the abstract does not allow judging the scale of the effects or the range of models covered. Still, as a methodological framework for answering where CoT helps executable code, where it only adds noise, and where model settings matter more, the paper is useful to any team running an analytics assistant in production.