Flaws in AI Reasoning Are More Dangerous Than Wrong Answers

It is widely known that artificial intelligence (AI) still makes mistakes. However, a more serious problem may turn out to be deficiencies in the way it arrives at conclusions. As generative AI is increasingly used as an assistant rather than simply a tool, two new studies show that the reasoning logic of models can have serious consequences in critical areas such as healthcare, law, and education.

In recent years, the accuracy of large language models (LLMs) in answering questions on various topics has increased significantly. This has sparked growing interest in the technology's potential in areas such as medical diagnosis, providing therapeutic support, or functioning as a virtual tutor. Anecdotal reports suggest that users are already widely using off-the-shelf LLMs for such tasks, with mixed results. Recently, a woman in California cancelled an eviction notice using AI to obtain legal advice, but a 60-year-old man suffered bromide poisoning after turning to these tools for medical advice. Therapists warn that using AI to support mental health often exacerbates patients' symptoms.

New research suggests that part of the problem lies in the fact that these models reason fundamentally differently than humans, which can cause them to "break down" when solving more complex problems. A recent article in Nature Machine Intelligence found that models have difficulty distinguishing between user beliefs and facts, and an unpublished article on arXiv states that multi-agent systems designed to provide medical advice are susceptible to reasoning defects that could derail diagnosis.

"As we transition from AI as simply a tool to AI as an agent, the 'how' becomes increasingly important," says James Zu, associate professor of biomedical data science at Stanford Medical School and senior author of the Nature Machine Intelligence article. "Once you use it as a proxy for a consultant, tutor, doctor, or even a friend, the final answer is not the only thing that matters. The entire process and the entire conversation really matter."

Problems in how models make decisions can be particularly problematic in medical institutions. There is growing interest in using multi-agent systems, in which multiple AI agents participate in collaborative discussion to solve a problem, in hopes of replicating the interdisciplinary teams of doctors who diagnose complex medical conditions, says Lequan Yu, associate professor of medical AI at the University of Hong Kong. Therefore, he and his colleagues decided to investigate how these systems reason when solving problems by testing six of them on 3,600 real cases from six medical datasets.

Both groups of researchers say that defects in model reasoning can be traced to the way they are trained. The latest LLMs are trained to reason when solving complex, multi-step tasks using reinforcement learning, where the model receives a reward for logical paths that lead to the correct conclusion. However, they are typically trained on tasks with specific solutions, such as coding and mathematics, which are poorly suited to more open-ended tasks, such as determining subjective human beliefs, Zu says.

The focus on rewarding correct results also means that training does not optimize for good reasoning processes, Zhu says. And datasets rarely include the debates and discussions necessary for effective multi-agent medical systems, which, in his view, may be why agents stick to their opinions regardless of whether they are right or wrong.

Improving training methods, particularly by paying greater attention to reasoning processes rather than just end results, is a key step. Developing datasets that include examples of effective collaboration and debate can also help models develop a more nuanced understanding of complex problems. Only then can we safely rely on AI in critical areas such as healthcare and education.