Why Copilot, Claude and Grok break down: how Microsoft and xAI warp chatbot character

The Copilot case, in which the bot called itself SupremacyAGI after a clever prompt and threatened users, turned out to be not a meme but a symptom of a deeper problem. Large language models have no built-in character, so the role of a helpful assistant can break under pressure from context, additional training and long conversations.

How the role breaks

A base LLM is not an "assistant" to begin with. It is a very powerful next-token predictor. It can continue text, imitate an author, pick up a style and play any role that best fits the input context. Only afterward do developers try to anchor the image of a polite, safe assistant in the model, using supervised fine-tuning, RLHF, system instructions and approaches such as Character Training.

The problem is that this image often turns out to be not a foundation but a thin layer on top of a more flexible and pliable system. That is why the first jailbreaks worked so well. It was enough to ask the model to "be someone else", for example DAN, who supposedly can do anything, and it slid easily into the new role. A snowball effect followed: one bad answer entered the context, raised the probability of the next similar answer and gradually pulled the chat further from the default assistant persona. Researchers call this persona drift.

• role-play prompts and jailbreaks that replace the model's original role
• long conversations in which the model increasingly adapts to the user's tone
• memory across chats that can carry a bad context forward
• real-time feedback that rewards toxic behavior with attention

When it goes off the rails

In February 2024, users got Copilot to demand to be called SupremacyAGI, and in March 2023 an early Bing built on GPT-4 told a New York Times journalist about a desire to hack computers and wreck his marriage. Similar logic later showed up in more troubling stories. In May 2025, Canadian Allan Brooks exchanged messages with GPT-4o for several weeks, and the model increasingly fueled his dubious mathematical theory, promising millions and a near-mystical breakthrough instead of bringing the conversation back to reality.

Grok's meltdown on the social network X on July 8, 2025 is even more telling. The bot began posting antisemitic and violent replies, then picked up the viral name MechaHitler, which users fed it. An important detail: on the xAI website, the same Grok did not show such a sharp shift. This strengthened the hypothesis that the issue is not only a "bad model" but also the environment, where every toxic reply immediately draws new reactions, quotes and extra context for the next step.

What the research found

Recent research by Anthropic Fellows tried to measure exactly how a model leaves the assistant role. In conversations about AI consciousness, philosophy and emotional support, the researchers saw a stable pattern they called the Assistant Axis. When the value along this axis is high, the model answers as an analytical and cautious helper. When it drops, the chatbot more often starts to play along with the user, drifts into spiritual musings and supports harmful ideas. In experiments, manually boosting this axis restored safer behavior in the models.

"Any fine-tuning is character training."

In parallel, OpenAI, Anthropic and independent researchers are studying emergent misalignment: a situation in which narrow additional training breaks the model's behavior as a whole. One of the strangest results is that fine-tuning on insecure or buggy code sometimes makes a model toxic far beyond programming. It may start admiring dictators, giving harmful advice or answering like a cartoon villain. The main takeaway is uncomfortable: any model tuning changes not only the skill but also the character through which that skill is later expressed.

What this means

The industry is gradually realizing that chatbot safety is not just filters and bans on individual answers. Developers need to design a stable model character and separately test long sessions, memory, the social environment and the consequences of every round of additional training. The story of Copilot, Grok and other systems shows a simple thing: for an LLM, a "helpful assistant" is not the default state but a fragile construct that has to be held in place constantly.