Authors Use 8 Examples to Check What Remains in Text After Masking Before an LLM

Eight artificial texts, one predefined list of "residues," and two pipeline components that can be switched off. That is the basis of a Habr write-up on how to verify personal-data masking before text is sent to an LLM. The starting thesis: removing a person's name is not enough. If a number that appears in a public registry stays in the text, the person can still be identified.

The publication is so far available only as an announcement. Below is what it claims and the logic of the approach. The specific measurement results from the full text are not reproduced here.

The Mask Covered the Name, but Not the Person

The typical setup looks like this. Before a request goes to an external model, the text passes through a masking tool. It replaces names, phone numbers and addresses with placeholders such as [PERSON] or [PHONE]. The team then notes that the personal data has been removed.

But a name is not the only thing that identifies a person. A sole-proprietor (IP) number, a taxpayer ID (INN), a court case number, a vehicle license plate or a contract number can lead to a specific individual through public registries and case files. If the masking tool is tuned for names and phone numbers, such values pass through untouched. The model, the request log and the third-party provider receive data that is formally "anonymized" but in fact is not.

The authors propose changing the question. Instead of asking what the masking tool found, they check what is left in the text after it has done its work.

How the Check Works

The approach rests on a small but reproducible test set. The experiment parameters, as stated in the announcement:

• 8 synthetic examples; the sample contains no real personal data;
• a predefined list of residues: values that must not appear in the output text;
• a comparison of the masking tool's output against that list, which is a check against the actual result rather than the detector's report;
• an experiment with normalization disabled;
• an experiment with name recognition disabled.

The last two modes were not chosen at random. Normalization brings text to a uniform form: it removes extra spaces and hyphens in numbers, standardizes case, and reduces different spellings of the same value to one. Without it, a rule or dictionary may miss a number written slightly differently from the template. Name recognition handles names that are not in dictionaries, as well as declensions and notation variants such as "Иванову А. П." If you switch off one of these components, you can see how much of the protection it provided. Exactly how much will be shown by the figures in the full write-up.

What Limits the Method

Eight examples work as a demonstration and a regression test, but not as a benchmark. Such a sample cannot tell you about the masking tool's recall and precision on a real data stream. Synthetic data is usually cleaner than live data. It has fewer typos, less OCR noise, fewer fragments of call transcripts and less language mixing. The predefined list also limits the check: it will find only what has been put into it. An unexpected identifier that the team did not think of will go unnoticed.

The approach also has a strength. It turns a vague "we mask the data" into a verifiable claim with a specific list and a specific result.

How to Apply It in Your Own Project

To carry the idea over to your own project, you need five steps:

• compile a list of identifiers that, in your domain, lead to a person through open sources, and do not limit it to names and phone numbers;
• build a synthetic corpus with different formats for writing the same number;
• run the pipeline in full mode and with each component switched off in turn, to understand each one's contribution;
• add the residue check to CI, so that every change to the rules or switch of the recognition model passes through it;
• periodically add to the corpus examples from real failures that have been anonymized in structure.

Masking remains an engineering measure, not a legal opinion. Whether it is sufficient for the requirements of personal-data law is a separate decision. But without a residue check, even a correctly configured masking tool remains an untested hypothesis.