Authors test 8 examples to see what survives in text after masking, before it reaches an LLM

Eight artificial texts, one predefined list of "residues" and two pipeline stages that can be switched off. That is the basis of a Habr write-up on verifying personal-data masking before text is sent to an LLM. The starting thesis: removing a full name is not enough. If a number that appears in a public registry is left nearby, the person can still be identified.

The publication is so far available only as an announcement. Below is what it claims and the logic of the approach. The specific measurement results from the full text are not included here.

The mask hid the name, but not the person

The typical setup looks like this. Before a request goes to an external model, the text passes through a masker. It replaces full names, phone numbers and addresses with placeholders such as [PERSON] or [PHONE]. The team then notes that personal data has been removed.

But a name is not the only thing that identifies a person. A sole proprietor registration number, a tax ID (INN), a court case number, a vehicle plate, a contract number can all lead to a specific individual through public registries and case files. If the masker is tuned for names and phone numbers, such values pass through untouched. The model, the request log and the third-party provider receive data that is formally "anonymized" but in fact is not.

The authors propose changing the question. Instead of asking what the masker found, they check what is left in the text after it has run.

How the check works

The approach rests on a small but reproducible set. The experiment parameters, as stated in the announcement:

• 8 synthetic examples; the sample contains no real personal data;
• a predefined list of residues: values that must not appear in the output text;
• comparison of the masker's output against that list, i.e. a check on the actual result rather than on the detector's report;
• an experiment with normalization disabled;
• an experiment with name recognition disabled.

The last two modes are not chosen at random. Normalization brings text to a uniform form: it removes extra spaces and hyphens in numbers, aligns case, and reduces different spellings of one value to a single one. Without it, a rule or dictionary may miss a number written slightly differently from the template. Name recognition handles full names that are absent from dictionaries, as well as declensions and spelling variants such as "Ivanovu A. P." With one of the stages switched off, you can see how much of the protection it provided. Exactly how much will be shown by the figures in the full write-up.

What limits the method

Eight examples work as a demonstration and a regression test, but not as a benchmark. Such a sample cannot be used to judge the recall and precision of a masker on a real data stream. Synthetic data is usually cleaner than live data. It has fewer typos, less OCR noise, fewer fragments of call transcripts and less language mixing. The predefined list also limits the check: it will find only what has been put into it. An unexpected identifier the team did not think of will go unnoticed.

The approach also has a strength. It turns the vague "we mask the data" into a verifiable claim with a specific list and a specific result.

How to apply it yourself

To carry the idea over to your own project, you need five steps:

• compile a list of identifiers that, in your domain, lead to a person through open sources, and do not limit it to names and phone numbers;
• build a synthetic corpus with different formats of the same number;
• run the pipeline in full mode and with each stage switched off in turn, to understand each one's contribution;
• add the residue check to CI, so that every rule change or change of recognition model passes through it;
• periodically add to the corpus examples from real failures, anonymized in structure.

Masking remains an engineering measure, not a legal opinion. Whether it is sufficient for the requirements of personal-data law is decided separately. But without a residue check, even a correctly configured masker remains an untested hypothesis.