Fine-Tuning an NER Model Raised the Share of Fully Masked Deal Secrets from 39.5% to 91.8%
In a CRM, a 12% discount can be either a secret or a public term, and only the first should be masked. The author fine-tuned GLiNER (287M parameters), an NER model that finds a span of text and determines its type. The share of fully masked secrets rose from 39.53% to 91.84%, and false masks on public mentions dropped to 8 out of 575 from 368. For context-dependent terms, it makes sense to fine-tune a compact model on your own data.
AI-processed from Habr AI; edited by Hamidun News
Imagine a CRM where managers have been writing deal notes for years. Before handing those notes to a contractor, an analytics team or an LLM prompt, the secret terms have to be hidden behind a mask. The problem is that the same value can be either a secret or public information.
In the note "agreed a 12% discount with the client, terms are non-public," the 12% must be hidden, but in the sentence "the 12% discount applies to all customers" it must be left alone. Regular expressions won't help here: context decides, not format. An author on Habr fine-tuned GLiNER for this task.
GLiNER is an NER (named entity recognition) model: it finds a span of text and determines its type. On a short synthetic test, the share of secrets masked in full rose from 39.53% to 91.
84%. Public mentions masked by mistake fell to 8 out of 575, from 368. The Exact F1 metric, which requires the span's type and boundaries to match the reference exactly, climbed from 22.
51% to 94.13%. If you clean data before sending it to an LLM, the experiment shows what fine-tuning on your own domain delivers.
What Was Trained, and on What
The author started from fastino/gliner2.5-multi-v1: about 287M parameters, with an mDeBERTa encoder. The model receives the names and descriptions of the target types along with the text, so the same schema of six entity types can be run both before and after training. The model returns the spans it finds, their types and confidence scores. Ordinary code builds the JSON and the final masked text. It replaces spans from right to left, so the length of an already inserted mask doesn't shift the coordinates of the rest.
The author defined the boundaries of a secret in advance. If a bonus agreement is confidential, the mask should cover "3% of annual turnover for purchases of 500 units or more," not just "3%." In a payment schedule, the mask covers both the percentages and the deadlines.
All texts are fictional. The corpus of 6,500 records is split into 5,000 for training, 500 for choosing the weights version and confidence threshold, and 1,000 for the final evaluation. A generator assembled the texts from 336 components. A quarter of the documents contain no secrets at all, and there are contrastive pairs in which the same value is non-public in one text and published in another. Training ran on a single 24 GB RTX 3090: three epochs, effective batch size 16.
Full Fine-Tuning, LoRA and the Confidence Threshold
The author compared full fine-tuning with LoRA, DoRA, rsLoRA and QLoRA. In the chosen LoRA recipe, adapters sit only in encoder.query and encoder.value. That trains 589,824 parameters, about 0.21% of the base model, and the remaining weights stay frozen.
The weights version and threshold were chosen on a separate dev split; the test set played no part in selecting settings. For full fine-tuning, the second epoch with a 95% confidence threshold won. For LoRA, it was the third epoch with a 90% threshold, and for rsLoRA, the first epoch with a 90% threshold. The selection criterion was strict micro-F1. When scores tied, the variant with fewer false masks on public values was preferred.
In additional checks, the author replaced the original labels with DeepSeek labels and trained a separate GLiNER. He also added Qwen and Gemma with NER heads and tested all the recognizers on long documents. The available part of the material contains no results of these checks, so they can't be used to compare the approaches.
Limitations and What to Do in Practice
The numbers should be read with caution. All data is synthetic, and there was no independent expert labeling. The text variants were built from related families rather than written by different people. On real correspondence with typos and jargon, the result may be lower.
The practical takeaway for teams that mask data before sending it to an LLM is as follows:
- First write down a policy: what counts as a secret and where the span boundaries lie.
- Leave customer names, company names and phone numbers to dictionaries and regular expressions.
- Hand the terms written in free text to a compact model fine-tuned on your own context. It fits on a single consumer GPU.
The author published the data, configurations and an example of running the model on your own JSONL in the materials repository.
- Nov 18, 2026After a Ban, the Author Launched an Agent Team in Kimi, Codex and OpenCode: Runs from $0.30
- Nov 18, 2026Qwen on two DGX Spark units runs 34–59% faster in tokens per second and outpaces GLM
- Nov 18, 2026Meituan open-sources 1.6-trillion-parameter agentic model trained without Nvidia
- Nov 12, 2026Railway Employee Ran a DeepSeek-Based AI Agent That Went After a Competitor's Infrastructure
Key Questions Answered
What Was Trained, and on What?
The author started from fastino/gliner2.5-multi-v1: about 287M parameters, with an mDeBERTa encoder. The model receives the names and descriptions of the target types along with the text, so the same schema of six entity types can be run both before and after training. The model returns the spans it finds, their types and confidence scores. Ordinary code builds the JSON and the final masked text.
What are the key details on Full Fine-Tuning, LoRA and the Confidence Threshold?
The author compared full fine-tuning with LoRA, DoRA, rsLoRA and QLoRA. In the chosen LoRA recipe, adapters sit only in encoder.query and encoder.value. That trains 589,824 parameters, about 0.21% of the base model, and the remaining weights stay frozen. The weights version and threshold were chosen on a separate dev split; the test set played no part in selecting settings. For full fine-tuning, the second epoch with a 95% confidence threshold won.
What are the key details on Limitations and What to Do in Practice?
The numbers should be read with caution. All data is synthetic, and there was no independent expert labeling. The text variants were built from related families rather than written by different people. On real correspondence with typos and jargon, the result may be lower. The practical takeaway for teams that mask data before sending it to an LLM is as follows: First write down a policy: what counts as a secret and where the span boundaries lie.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.