How PII Scrubbing Affects LLM Agents: The Hivetrace Dataclean Experiment

Scrubbing personal data before it reaches an LLM agent usually looks like an obvious security win, but the cost of that decision isn't always clear. In a Habr AI article, the authors tested how a banking agent's behavior changes when it receives masks or pseudonyms instead of real full names and other identifiers.

Where the conflict arises

LLM agents increasingly handle user workflows that can't work without personal data: banking requests, support, insurance, documents, transaction history. At this point, a team always has to choose between two risks. The first is to pass sensitive data to the model as is and face questions about privacy, compliance, and internal security. The second is to scrub the input but lose some of the context the agent relies on for its response logic, for linking related entities, and for accurate actions.

"How much does an agent degrade if it sees "PERSON_1" or "XXXXXXXX" instead of "Ivan

Ivanov"?"

In practice, this isn't an academic debate but an engineering problem. If the system can no longer tell that the same customer appears in several parts of a request, the damage goes beyond text quality to business logic: case routing, account detail verification, status interpretation, and checking the sequence of steps. That's why a blanket claim like "anonymization helps" or "anonymization gets in the way" isn't enough. What matters is a measurable answer for a specific class of agents and a specific scrubbing method.

How they tested the hypothesis

To test it, the authors spun up a minimalist banking agent and connected it to Hivetrace Dataclean. They then sent the agent 102 synthetic queries in each of three input variants: unchanged, masked, and pseudonymized. This design strips out the noise of real user histories and enables a controlled comparison. In every case, the agent solves the same task but sees a different form of identifiers, names, and other personal attributes.

The evaluation method matters too. The authors used DeepEval in LLM-as-a-judge mode: rather than comparing subjective impressions of a couple of responses, they tried to formalize quality through a single evaluation pipeline. For a quick applied study, that's a sensible move. It doesn't turn the conclusions into "absolute truth", but it shows where degradation is immediately visible and where the agent stays useful even after sensitive fields are scrubbed.

Three data modes

The point of the experiment isn't just to check whether the agent "works or doesn't" after anonymization, but to compare different levels of context loss. This matters especially for LLM systems that rely not only on the meaning of words but also on stable links between entities within a single conversation or document. Full masking and pseudonymization look similar to the model only at first glance; in a real task, they are different signals.

• Clean data — the baseline scenario with no distortion.
• A mask like XXXXXXXX — maximum privacy, minimum semantics.
• Pseudonyms like PERSON_1 — hide identity but preserve relationships.
• 102 synthetic queries per mode — enabling a fair comparison.

From a test like this, teams typically get a map of trade-offs rather than a universal ban or green light. If the agent only needs the user's general intent, a hard mask may prove acceptable. If the logic depends on the same entity recurring across several fragments, pseudonyms often look like the more practical option. That's why such checks are especially useful before deploying agents in banking, fintech, healthcare, and any process where mistakes with personal data quickly get expensive.

What it means

The article's main takeaway: anonymization for LLM agents can't be decided by intuition. Before going to production, teams need to measure separately how a specific scrubbing method affects a specific scenario: query understanding, preservation of links between entities, and final response quality. For teams building AI agents in sensitive domains, this is no longer optional but a core part of the architecture and the testing pipeline.