Researchers Find 7 "Dead" Models Among 305 in the Ollama Library: They Can't Write Code

When a developer runs a language model on their own computer, they almost always download a compressed file from a public catalog. The compression is called quantization: instead of 16 bits, the model's weights are stored in 2–8 bits so the file fits in memory. These files are published by catalog operators or volunteers, and neither the catalog pages nor the metadata record that anyone tested the file in practice. The researchers decided to check this at the scale of an entire catalog. They took all 305 eligible builds (tags) from Ollama's official library and ran programming tasks on each one. Seven builds, or 2.3%, turned out to be non-functional: they solved no more than 6 of 164 tasks.

How the dead files were found

The sample covered 15 model lines: eight tuned for code and seven general-purpose. In total there were 35 model variants; 291 builds are quantized and 14 are stored in standard 16-bit precision. The researchers added 22 popular community files from Hugging Face, drawn from 12 repositories.

The check ran in two stages:

• Quick filter. 15 tasks from HumanEval+, chosen from the 39 tasks that all 23 known-healthy Qwen2.5-Coder configurations solved. A build that solved fewer than 9 of the 15 came under suspicion. Generation takes about 30 seconds per file. • Full run. Suspects got all 164 HumanEval+ tasks. A build counted as non-functional if it solved less than 10% and also lagged far behind its own high-precision version.

All runs used an RTX 4090 GPU with greedy decoding and a fixed seed (seed 42), so a repeat run on a second machine produced the same 225 answers, byte for byte. Along the way, the researchers found a bug in the scoring itself: because of timeouts, task HumanEval/15 was marked as failed almost everywhere. After rechecking the suspicious builds, 39 remained instead of 50, and the seven main suspects did not change.

All seven are heavily compressed versions of models up to 4 billion parameters. Five of them are Qwen and phi3.5 builds. The other two are llama3.2 tags at 1 billion parameters. The researchers did not rebuild these because of the license on the original weights, so the cause remains unknown. Two community files also failed to work; the researchers tested them separately with different chat templates.

The files are intact; the recipe is the problem

The researchers rebuilt five of the suspects from the original weights using the recipe listed in the metadata, that is, without calibration. They then compared the tensors: type, shape and bytes. The result was unexpected: all five catalog files matched the rebuilds exactly. There was no corruption during conversion. These are exact quantizations of the original weights, and they simply don't work.

Next, the researchers tested calibration. This means an importance matrix (imatrix): activation statistics collected on a text sample. It tells the quantizer where rounding error does the most damage, so it can choose block scales more accurately. The matrix was built either from 100 WikiText-2 fragments of 512 tokens each, or from 1.5 MB of Python code from the llama.cpp repository.

The results:

• For Qwen2.5-Coder-3B, at three of the four broken compression levels, the score rose from zero to 76–113 of 164 tasks, or 46–69%. The 4-bit version of the same model solves about 80%. • The same uncalibrated recipe on Qwen2.5-Coder-1.5B produced a working model. So the recipe alone does not predict which builds will break. • Of 66 spot-checked builds that passed the filter, none are non-functional.

The first version of the study picked its reference by the conversion's author rather than by recipe, and called the files defective. Rebuilding with the same recipe corrected that conclusion.

What catalog operators and users should do

Problems with catalog files have happened before. In 2024, llama.cpp maintainers found that some conversions used incorrect text preprocessing before tokenization, and users reported failures. Two years later, 26 of 305 official tags still have no tokenizer type specified.

The authors recommend the following:

• Before publishing, run low-bit builds through a short code-execution test. They have released a tool called quantcheck and the audit data for this purpose. • If a build fails the test, it should be calibrated or not released. • Users should not assume a catalog tag has been verified, especially at 2–3 bits per weight and in models of up to 4 billion parameters.

The study has limitations. It covers only code models, one official library and a small sample from Hugging Face. The two non-functional llama3.2-1B tags remain unexplained. The filter relies on tasks selected using healthy Qwen builds, and among deepseek-coder-1.3B models, six tags score just 1–6 of 15 even at high precision, though that is a different story. But the main conclusion is clear: matching the recipe exactly does not mean a file works, and a cheap 30-second test caught what had been sitting in the catalog for months.