A 17-million-parameter encoder picks the right tool in 14ms on CPU

The last step of many LLM pipelines looks modest: pick one option from a known set. Which tool to call, which label to route a query under, which path to send it down. This is usually done by prompting a large model, and that's slow, expensive, and sensitive to the order of options. Tardelli Ronan Coelho Stekel tested just how small an encoder can be for this job, and released the Crivo family in the open: six models ranging from 17 million to 1 billion parameters.

The key numbers. The 68M and 150M models beat the 435M-parameter DeBERTa-v3 zero-shot classifier by 4.9 and 5.9 accuracy points on 209 tasks from System1-Arena, which were audited for overlap with training data. On 145 tasks the models never saw during training, they hold at least on par with DeBERTa. The smallest model, 17M, answers a query in a median of 14ms on 16 CPU threads. DeBERTa's classifier, under the same conditions, has a median of 728ms.

Reading options together or one at a time?

The second question in the research matters more to system architects. An encoder can be fed all options at once (joint) or evaluate each option separately, paired with the query (per-option). The author ran a three-seed study on a 32M-parameter model using a held-out split from a set of 37 benchmarks.

Per-option wins by 5.8 points of randomness-adjusted skill at an equal number of optimizer steps. At roughly equal training compute, the gap narrows to 2.8 points. Two external benchmarks confirm the direction of the effect, but on unseen tasks the difference is not statistically significant.

The advantage doesn't always show up. It only appears when initializing from a reranker: its query-candidate pair input matches how the per-option scheme reads options. If you start from a raw encoder, the two approaches are indistinguishable.

The price of accuracy and joint's weak spot

Per-option requires 5–13 times more encoder passes. In median latency, that's no more than a twofold loss, but in mean latency it's already more than fourfold: long option lists hit the tail hard. That's why joint models made it into the release.

The joint approach has a measured flaw: shuffling the order of options changes 12–23% of answers. The per-option scheme has no such dependency — its answers don't change at all. For production use, that's a serious argument: the order of tools in a list shouldn't affect the choice, but here it does.

Up to 68M parameters, Crivo models lead their size class on the public leaderboard. Scoring runs on private tests that are mostly close to the training distribution, so this is a strong result for tasks similar to the training data, but not a guarantee of transfer.

What's in the release

• The Crivo family: six open joint encoders from 17M to 1B parameters, a collection on Hugging Face (tardellirs/crivo).
• 68M and 150M: +4.9 and +5.9 accuracy points over DeBERTa-v3 (435M, zero-shot) on 209 System1-Arena tasks.
• 145 unseen tasks: at least parity with DeBERTa-v3.
• 17M: median 14ms on 16 CPU threads versus 728ms for DeBERTa-v3.
• 32M study: per-option beats joint by 5.8 points at equal steps and 2.8 at equal compute, three seeds, 37 benchmarks.
• Cost of per-option: 5–13x more passes, up to 2x median latency and over 4x mean latency.
• Order sensitivity: 12–23% of joint answers change when options are shuffled.
• Code and results published in the crivo-decision-encoders repository on GitHub.

Where this will be useful

For routing queries between models, tool selection by an agent, and classification over a closed label set, these encoders offer a cheap alternative to calling an LLM. The 17M model fits in 14ms on an ordinary CPU with no GPU, and that changes the economics: a decision can be made at every step of an agentic loop instead of being batched into a separate, expensive call.

The practical takeaways are these. If stability matters, either randomize and average the order of options, or use the per-option scheme and accept the tail-latency hit. If you're training such a selector yourself, start from a reranker: a raw encoder won't gain anything from reading options one at a time. Before deployment, run the model on your own tasks: parity on unseen data means "no worse than DeBERTa," not superiority, and the biggest gains show up where the distribution is close to the training data.