GPT-4o mini Holds 37–38 Word-Replacement Rules at 95% Accuracy

A paper published in Hugging Face Daily Papers proposes measuring a language model's limit in a countable number of rules, the way memory is measured in bytes. The unit of load here is a single rule of the form "replace word A with word B" in a text with no meaning. The result is called "instruction capacity": how many such rules the model can follow before its share of correct replacements falls below a set threshold. For GPT-4o mini, the 95% correct-replacement threshold is reached at 37–38 rules. If the words to be replaced take up the maximum share of the text, capacity drops to 18–19 rules.

Why older tests don't give a single number

It has long been known that quality drops as the number of conditions grows. But no one has yet managed to express a model's limit in units of load, with a margin of error. The reason lies in the tests themselves: their instructions differ in nature and difficulty.

IFEval, for example, asks a model to "write more than 400 words" or to "mention the word AI three times." FollowBench adds constraints of five types to a task one at a time. IFScale uses from 10 to 500 instructions of a single kind, and the best of 20 models follow only 68% of them at 500.

The check is often done by another model acting as judge, and judges inflate the result. In one of the cited papers, with ten instructions the judge gave a score of 0.657, while a programmatic check gave 0.

213.

As a result, the number of instructions gets mixed up with the composition of the set and with rising difficulty. The author needed an instruction that avoids these flaws:

• indivisible, meaning it cannot be half-followed; • independent of the other rules by the construction of the task; • uniform, so that the tenth rule is no harder than the first; • checkable by a program; • observable at every point of the response.

How the measurement works

Replacing a word in a meaningless text meets all five criteria. Each replacement has exactly one correct outcome, and a program computes the reference answer. The task is generated from three random seeds and can be reproduced byte for byte. The response is checked line by line, and errors fall into three types.

Each occurrence of a word to be replaced becomes a separate pass/fail trial, with a known position in the response and a known rule number in the list. This way a single model yields hundreds of thousands of observations rather than one score per task.

A statistical model then takes over. First, it measures, interval by interval, how the probability of a correct replacement depends on five factors:

• text length; • word position; • replacement density; • number of rules; • the rule's number in the list.

These dependencies are then described with smooth laws. A correction for approximation bias recovers a baseline level that does not depend on the choice of encoding. Capacity is computed by solving an inverse problem: at what load does the probability fall to the required level. The margin of error is calculated with the delta method and accounts for the spread in difficulty between individual requests.

What GPT-4o mini scored

The measurement took 2958 requests and 642 430 trials. For every load factor, the model showed a regular pattern of degradation. The paper separately describes patterns of response "collapse": how fast and at what point an answer falls apart. Besides capacity at 95% correct replacements, the author names two more measures, fatigue capacity and guaranteed capacity. The paper defines them in more detail in its main text than in the introduction.

The key figures:

• about 37–38 rules at a correct-replacement share of 0.95; • about 18–19 rules at full replacement density.

The twofold gap shows that the limit depends not only on the number of rules but also on how densely they trigger in the text. For anyone writing prompts, this matters: the same list of rules can be safe on sparse text and unreliable on dense text.

The tool's code, the measurement data and the model's "passport" are open. Every number in the passport can be recalculated from the published points, so the result is verifiable.

Why engineers should care

The author sees the model passport as the calibration for a future prompt "load meter." The idea is to estimate the complexity of an arbitrary prompt in the same units as the model's capacity, and to see in advance whether it fits within the model's limit. For now, this is a claim about the future, not a finished tool.

The limitations are also obvious. Only one model, GPT-4o mini, was measured, and the task is artificial: replacing single words in a text with no meaning. Real instructions about style, format and tool use are more complex. The AgentIF benchmark, built on prompts from real agents, showed that even the best models fully follow fewer than 30% of instructions. A word-replacement passport sets a clean lower scale, but it does not replace such tests.

The practical takeaway: if an agent relies on a long list of independent rules in its system prompt, compare the number of rules with the model's measured capacity and leave a margin. When comparing models, or generations within one model line, capacity with a margin of error is a more honest metric than an average score over a heterogeneous set.