Hugging Face and ModelAudit: Study Reveals the Limits of Built-in ML Model Protection

A researcher compared Hugging Face's built-in security checks with the external scanner ModelAudit and got a less than obvious result: one tool can see more risky signals, yet generate a lot of noise. The article's main conclusion is that the number of critical alerts says almost nothing by itself about how malicious a model really is.

How the test was set up

In the first experiment, the author did not take all of Hugging Face, but a subset of repositories using the riskiest model storage formats. The sample included only public models containing files such as `.pkl`, `.pickle`, `.dill`, `.pt`, `.pth`, `.ckpt`, `.bin`, `.joblib`, `.npy` and `.npz`. Repositories that were too large or models that were too popular were also excluded: total size was capped at 1 GB, and monthly downloads at 10 thousand. The idea is simple: when hunting for real problems, it makes more sense to start where the likelihood of dangerous serialization is higher.

• The first set covered 246 scanned models
• ModelAudit found 271 critical warnings
• At least one critical alert hit 34 models
• The comparison was made on whole repositories, not on individual checkpoints inside them

But right from the start it became clear that a large number of findings does not equal detection quality. The model with the most hits was Ultralytics/YOLO11: it received 728 warnings, 35 of them critical. On paper that looks like a strong sign of compromise, but manual review showed a more mundane picture. A significant share of the critical flags were tied to standard Python elements that can also appear in legitimate models. In other words, the scanner correctly spotted potentially dangerous patterns, but too often treated them as direct proof of an attack.

Where the rules get noisy

The YOLO11 analysis neatly exposed the weak spot of static analysis. Some hits came from `pickle_check` because of `__builtin__.getattr`, and others from `pytorch_zip_check` because of `__builtin__.set` and similar indicators. The problem is that `getattr` can indeed be used in malicious chains to bypass primitive rules, but it is also an ordinary Python function found in normal code. The case of `set` is even more telling: one internal ModelAudit scanner considers such an import acceptable, while another may flag the whole `builtin` namespace as suspicious.

That is why the author stresses that a high density of alerts, even critical ones, is a reason for manual triage, not a verdict on the model. Over the course of the first experiment he examined other types of hits too, including suspected executable signatures inside binary files, and kept running into the same problem: rules are often handy for finding candidates, but work poorly as a final verdict without context, file format and an understanding of the specific framework.

"This is not what I imagined when we started"

Comparison with Hugging Face

In the second experiment, the author shifted focus and compiled a list of models that the repository authors themselves had already labeled malicious, exploit, ace, deserialization or poc. After additional filtering with an LLM, this set was run through ModelAudit and the results were compared with Hugging Face's built-in statuses.

The baseline comparison showed fairly strong agreement: both sides judged 154 repositories dangerous and 49 safe. There were also 14 cases where ModelAudit saw a problem while Hugging Face showed nothing suspicious.

The most important nuance is that some of ModelAudit's useful signals live not only at the warning and critical levels. The article gives the example of `jossefharush/gpt2-rs`, where an INFO-level alert contained signs of network activity and a Pastebin link. Further checking showed that the link hosts a backdoor that sends the results of commands executed on the victim's machine to the attacker. So in this particular case, the "informational" message turned out to matter more than many of the loud critical flags from the first experiment.

The author also examined the reverse discrepancies, where Hugging Face flagged a danger and ModelAudit missed the model. Such misses initially occurred in version 0.2.24, but after updates to 0.2.28 and then 0.2.31 these cases disappeared. The final picture: every repository that Hugging Face ultimately considered dangerous was also caught by ModelAudit, and on top of that the external scanner retained 17 more repositories with dangerous signals that were absent from HF's built-in warnings.

What this means

A single scanner does not solve the security problem for ML artifacts, even if it looks like the most mature in its class. The article on Hugging Face and ModelAudit points to a more useful lesson: good results come not from betting on one "best" tool, but from combining several checks, updating rules regularly and manually reviewing the loudest hits.