AI assistant raised students' scores by 48%, but without it they performed 17% worse

Picture a junior developer who accepts Copilot suggestions one after another. Tasks close quickly and metrics climb, but in code review he cannot explain why the solution is built the way it is. A recent analysis on Habr explains why this is no accident, and what those building AI products should do about it.

An experiment that spoils attractive metrics

A meta-analysis of 69 studies from 2022–2024 shows that ChatGPT improves academic performance and motivation and reduces mental effort. For a product team this looks like a perfect result. But in a field experiment with nearly a thousand high school students solving math problems, GPT-4 in a standard interface raised practice scores by 48%, and in a version with safeguard prompts by 127%. After access was removed, students from the first group performed 17% worse than those who never used AI. The assistant worked as a crutch: while it is nearby, results rise, but no learning has occurred.

Why AI is not the same as a good interface

Cognitive load theory distinguishes three kinds of effort: intrinsic (the difficulty of the material itself), extraneous (caused by poor presentation) and germane (the work of building knowledge). In UX, the main lever is reducing extraneous load, and that is almost enough: a CRM user wants the order to go out, not to understand the nomenclature directory.

With AI it is different. In an experiment with 91 students, those who worked with ChatGPT-3.5 reported lower load on all three scales, and their final justifications turned out worse than those of Google users. A series of seven experiments with 10,462 participants confirms that knowledge gained from an LLM summary is shallower than knowledge gained from search engine links. A search engine provides sources, and you have to connect them yourself. An LLM provides a ready-made synthesis, meaning the work on intrinsic complexity is done before the person sees the result. A good interface removes the noise around a task, while AI can remove the task itself.

There is also an unpleasant detail. A study of 589 students and young professionals distinguishes dependent offloading (thinking handed over to AI) from autonomous offloading (AI as support, with decisions left to the person). The immediate benefit of both modes is the same, so users cannot tell them apart by feel, and satisfaction metrics will show nothing. A caveat: these studies are correlational, and in some of them outcomes are based on self-assessment.

What product managers and developers should do

Skill-preservation mechanisms need to be built into the product, not left to users' conscientiousness:

• An IDE assistant first asks the junior to guess where the bug is, and only then shows the fix with an explanation.
• A research assistant provides not a summary but sources with their discrepancies, and asks the user to formulate the conclusion.
• A support assistant links to a knowledge base section and checks the operator's answer.
• An educational bot guides with questions rather than solving the problem for the student.

Support should not be permanent. The expertise reversal effect says that techniques that help novices hinder experts, so help should fade as skill grows: support, then verification, then independent work.

The practical minimum

Give AI search, formatting and routine. Leave linking, conclusions and evaluation to the human. Add explanations and verification before an answer is accepted. And above all, change your metrics: overall load and satisfaction cannot distinguish helpful offloading from harmful. Periodically check what the person can do without the tool. If they must stay in the loop and answer for the result, this is the only honest test.