Ant Group's SingProbe: an 8.1M-Parameter Guard That Slows an LLM by Less Than 0.5%
Chatbot answers are usually checked by a second "guard" model, which is expensive and slows down streaming. Ant Group has released SingProbe: an 8.1M-parameter probe that reads the internal states of Ling-3.1-flash-1019 and scores every token for harm and hallucinations. Overhead is under 0.5%, and in streaming checks it outperforms 8B models such as Qwen3Guard-8B. If you host an open model, it is worth testing in place of a separate moderation layer.
AI-processed from X: @crazyMLguy; edited by Hamidun News
Imagine you have launched a chatbot on a large open model. To keep it from giving dangerous advice or making up facts, you put a second model next to it, a "guard". It reads the request and the response and decides whether the text can be shown to the user.
This works, but it costs VRAM and money, adds latency, and fits poorly with streaming: the answer is already appearing on screen while the guard is still thinking. Ant Group has proposed a different approach and released SingProbe, a tiny 8.1M-parameter probe built into the Ling-3.
1-flash-1019 model itself (the announcement's headline refers to protection for LLMs in the 560 billion-parameter class). It adds less than 0.5% overhead to the model's work and checks text as it is generated, token by token.
A token is a small piece of text, roughly a word or part of one.
How SingProbe works
An external censor model is like an editor reading a finished article. SingProbe is more like a heart-rate monitor attached to the author: it picks up warning signs before the words are spoken. Technically, the probe looks at the model's hidden states, the internal numerical representations the network builds while writing a response. From these it scores three things: harmful intent in the request, unsafe output, and hallucinations, meaning confidently stated fabrications.
The score is produced for every token. That means a problem can be caught before it reaches the user, and the response can be cut off mid-sentence rather than filtered after the fact.
Why this matters to engineers
The first reason is cost. According to Ant Group, inference overhead is under 0.5%. By comparison, a standalone 8-billion-parameter guard model needs its own VRAM, its own deployment, and its own request queue. A probe with 8.1M parameters weighs almost nothing.
The second reason is integration. SingProbe runs natively in SGLang and vLLM, two popular servers for running models. There is no need to stand up a separate moderation service and keep it in sync with the main stream. The MIT license allows the code to be used in commercial products without legal complications.
The third reason is quality. Ant Group claims that in streaming safety checks, SingProbe outperforms standalone 8B models, including Qwen3Guard-8B. That is a notable comparison: Qwen3Guard is a well-known specialized guard, and here it was beaten by a probe roughly a thousand times smaller in parameter count.
Besides Ling, the team has released versions for DeepSeek-V4.1-Flash-1017 and other open models. This suggests the approach is not tied to a single architecture.
What to keep in mind
There are several caveats. First, the probe reads the model's internals, so it only works where you have access to hidden states. This trick is not possible with closed APIs that return only text. Second, judging by the repository name, each probe is trained for a specific model: the Ling version cannot simply be attached to a different network. Third, the announcement presents the comparison with Qwen3Guard only in general terms. Before dropping your current filter, run SingProbe on your own data: your harm categories, languages, and typical hallucinations may differ from those the authors measured.
Finally, the probe does not replace product policy. It outputs a risk score, and the team decides what to do with it: cut the response off, replace it, or send it to a human for review.
What to do in practice
If you host an open model yourself on vLLM or SGLang and pay for a separate moderation layer, SingProbe is worth testing in the coming weeks. This is especially relevant for streaming chat products, where latency matters and a dangerous answer needs to be stopped immediately. For teams working through closed APIs, the news is more of a signpost: safety is gradually moving from separate models into the networks themselves, and that is cheaper and faster.
- Nov 20, 2026Local GLM 5.3 Speeds Up AI on ESP32-S3 by 100%, Qwen 3.8 27B by 30%
- Nov 19, 2026MLPerf Training: 10,240-GPU clusters and 49% faster training in six months
- Nov 19, 2026CoreWeave comes in up to 65% cheaper than hyperscalers on a three-year Blackwell cluster
- Nov 19, 2026Chatbots cave after three objections: accuracy on climate answers drops 30–35%
Key Questions Answered
How SingProbe works?
An external censor model is like an editor reading a finished article. SingProbe is more like a heart-rate monitor attached to the author: it picks up warning signs before the words are spoken. Technically, the probe looks at the model's hidden states, the internal numerical representations the network builds while writing a response. From these it scores three things: harmful intent in the request, unsafe output, and hallucinations, meaning confidently stated fabrications. The score is produced for every token.
Why this matters to engineers?
The first reason is cost. According to Ant Group, inference overhead is under 0.5%. By comparison, a standalone 8-billion-parameter guard model needs its own VRAM, its own deployment, and its own request queue. A probe with 8.1M parameters weighs almost nothing. The second reason is integration. SingProbe runs natively in SGLang and vLLM, two popular servers for running models. There is no need to stand up a separate moderation service and keep it in sync with the main stream.
What to keep in mind?
There are several caveats. First, the probe reads the model's internals, so it only works where you have access to hidden states. This trick is not possible with closed APIs that return only text. Second, judging by the repository name, each probe is trained for a specific model: the Ling version cannot simply be attached to a different network. Third, the announcement presents the comparison with Qwen3Guard only in general terms.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.