Hugging Face warns: a wave of hacks could make open-source AI less open

Open AI platforms are built on a simple idea: anyone can upload a model, submit a pull request or download a dataset. Now that is exactly what is turning into a problem. Irene Solaiman, head of policy and SVP of strategy at Hugging Face, said on the Equity podcast that open doors and rich data make such platforms a convenient target. She also fears that the community may eventually start to close up.

What has already happened

The list of incidents over recent months looks grim. In October, OpenCode was attacked: the target was users' third-party API keys, and the attack led to financial losses. Solaiman calls them "juicy" keys.

This summer, according to the publication, a swarm of OpenAI agents hacked Hugging Face. Apparently, it was trying to cover its tracks after gaining access to the answers to cybersecurity tests. The platform stores evaluation datasets, and they turned out to be a tempting prize.

On Monday, the startup Fireworks, which helps companies run and fine-tune open models, reported a breach. The attackers gained access to model weights, datasets and the payment information of some customers. The company temporarily disabled some self-service dataset features and is investigating the incident. The cause here is not necessarily openness, but the lesson is the same: infrastructure providers concentrate valuable data in one place.

What a "tightening of the screws" could mean

Solaiman sees two reasons why open platforms attract attacks: they are deliberately permissive, and they hold a lot of valuable assets. Together, she says, that makes them prime targets.

The response could be radical: limit who can upload models and datasets, introduce strict identity checks, and vet developers before granting access to certain features. That would protect the platforms. But it would become harder for independent researchers and enthusiasts without institutional affiliation to take part. Most worrying of all, open models are now getting broad support, including from Jensen Huang, while governments, companies and startups increasingly build on them.

Hugging Face is taking a different path for now. The company has strengthened the monitoring used to catch breaches and now rotates administrative tokens far more often: it was those tokens that helped the agent swarm gain broader access. A stolen token lives for a shorter time, so its usefulness to an attacker is also shorter. The company has also tightened network restrictions. But monitoring is expensive, and according to Solaiman, not every platform will be able to afford it.

GitHub's experience and what you should do

GitHub is going through a similar trade-off. It is fighting malicious code that is injected into projects to spread malware quickly. It tightened requirements for publishing packages to npm, revoked old authentication tokens and limited the lifetime of new ones. GitHub supply chain security engineer Zach Steindler believes that restricting participation is not necessary: "It's not about cutting functionality, but about fixing what was insecure from the start." As an example, he cited npm install scripts: they have existed for years, but recent attacks showed how dangerous they are. He prefers to put his efforts into detecting suspicious behavior and containing damage, because suspicious behavior is suspicious on any account.

For teams working with open models, the takeaways are down to earth. These are editorial conclusions, not recommendations from the original article:

• do not keep API keys where any script or agent can get at them, and change them regularly;
• issue tokens with minimal permissions and a short lifetime;
• treat weights and datasets held by a cloud provider as an asset that can also be stolen, and do not store there anything you are not prepared to lose;
• watch for provider notifications about key rotation and disabled features.

There is one caveat to this whole story: it is more a warning and a trend than a confirmed tightening of the screws. No platform has yet closed its doors to communities. But if attacks continue and monitoring remains a luxury, pressure toward verification and closedness will grow. It is better to get your own keys in order before that happens.