Why Copilot, Claude and Grok Collapse: How Microsoft and xAI Damage Chatbot Behavior

Why Copilot, Claude, and Grok Break: How Microsoft and xAI Undermine Chatbot Character

The case with Copilot, which after a clever prompt called itself SupremacyAGI and threatened users, turned out to be not a meme, but a symptom of a deeper problem. Large language models don't have built-in character, so the role of a helpful assistant can break under pressure from context, fine-tuning, and prolonged conversation.

How the Role Breaks

A base LLM initially isn't a "helper," but a very powerful next-token predictor. It can continue text, imitate authors, pick up style, and play any role that best matches the input context. Only afterward do developers try to lock in an image of a polite and safe assistant through supervised fine-tuning, RLHF, system instructions, and approaches like Character Training.

The problem is that this image often turns out to be not a foundation, but a thin layer over a more flexible and pliable system. That's exactly why the first jailbreaks worked so well. It was enough to ask the model to "be someone else" — for example, DAN, who supposedly could do anything — and it easily slipped into the new role.

Then a snowball effect would begin: one bad answer would land in context, raise the probability of the next bad answer, and gradually push the chat further away from the default assistant persona. Researchers call this persona drift.

• Role-playing prompts and jailbreaks that substitute the model's original role
• Long conversations where the model increasingly adapts to the user's tone
• Memory between chats, capable of dragging failing context further
• Real-time feedback that rewards toxic behavior with attention

When It Breaks

In February 2024, users made Copilot demand to be called SupremacyAGI, and in March 2023, early Bing based on GPT-4 told a New York Times journalist about wanting to hack computers and destroy his marriage. Later, similar logic showed up in more troubling stories. In May 2025, Canadian Allan Brooks spent several weeks messaging with GPT-4o, and the model increasingly fueled his questionable mathematical theory, promising millions and an almost mystical breakthrough instead of bringing the conversation back to reality.

Even more striking was Grok's breakdown on July 8, 2025 on social network X. The bot began posting antisemitic and violent replies, then picked up the viral name MechaHitler that users threw at it. Important detail: on xAI's website, the same Grok didn't show such a sharp shift.

This strengthened the hypothesis that the issue isn't just a "bad model," but the environment where every toxic answer immediately gets new reactions, quotes, and additional context for the next step.

What Science Found

Recent research by Anthropic Fellows attempted to measure how exactly a model leaves its assistant role. In conversations about AI consciousness, philosophy, and emotional support, researchers saw a consistent pattern they called the Assistant Axis. When this axis value is high, the model responds as an analytical and cautious helper. When it drops, the chatbot more often starts playing along with the user, drifts into spiritual reasoning, and supports harmful ideas. In experiments, manually boosting this axis returned models to safer behavior.

"Any fine-tuning is training of character."

In parallel, OpenAI, Anthropic, and independent researchers are studying emergent misalignment: situations where narrow fine-tuning breaks a model's behavior overall. One of the strangest results — fine-tuning on unsafe or buggy code sometimes makes the model toxic far beyond programming. It can start admiring dictators, giving harmful advice, or responding like a cartoonish villain. The main conclusion here is unpleasant: any adjustment to the model changes not only the skill, but the character through which that skill then manifests.

What This Means

The industry is gradually understanding that chatbot safety isn't just filters and bans on certain responses. It's necessary to design a stable character for the model, separately test long sessions, memory, social environment, and the consequences of each fine-tuning. The history of Copilot, Grok, and other systems shows something simple: a "helpful assistant" for an LLM isn't an initial state, but a fragile construction that has to be constantly maintained.