SmolLM2-1.7B Beats 34B Models at MLIR Code Generation Without Fine-Tuning

In July 2026, researchers presented a method on arXiv that lets the language model SmolLM2-1.7B generate correct MLIR code on par with models 15-34 times larger. It uses no fine-tuning, relying instead on decoding constraints mechanically derived from dialect specifications.

Why MLIR is hard for language models

MLIR (Multi-Level Intermediate Representation) is the intermediate representation underpinning modern ML compiler infrastructure: TensorFlow, JAX/StableHLO, PyTorch Inductor and IREE. Yet MLIR appears only in trace amounts in language-model training corpora, so models generate it poorly.

The main difficulty is extensibility. By design, MLIR grows through new "dialects" for each application domain, and fine-tuning a separate model for every dialect does not scale. The authors asked whether inference-time priors, derived mechanically from a dialect's Operation Definition Specification (ODS), can replace adaptation through gradient-based fine-tuning.

How specification-derived constraints work

The method builds a three-layer constraint stack directly from the dialect schema, with no training. Each layer prunes invalid options during generation itself.

• C1 — a context-free grammar (CFG) over operation signatures
• C2 — type partitioning based on a type lattice extracted from the ODS
• C3 — an SSA scope validator with rejection sampling and five retries

According to the arXiv abstract, porting the stack from the arith+func+memref+linalg dialect set to StableHLO required not a single line of new constraint code, because the schema is derived from the specification automatically.

How far the small model got

On dialects where verifier semantics are defined by structural constraints, schema priors let SmolLM2-1.7B match or beat models with 15-34B parameters. On the linalg dialect, the model reached 80.0% valid programs (mean over three seeds, n=125).

• Advantage over CodeLlama-34B, Granite-Code-34B and StarCoder2-15B: 21 to 44 percentage points, with non-overlapping confidence intervals
• Generation speed: 8-25 times higher than the large baseline models
• Four NL-to-MLIR benchmarks released: MLIR-Spec-150, Linalg-Spec-30, StableHLO-Spec-30 and StableHLO-Held-Out-200, for a total of 410 text-to-MLIR pairs
• Datasets licensed under Apache-2.0, with datasheets following the Gebru methodology and Croissant 1.0 metadata
• The benchmarks, decoder, all generations and a Docker image for reproduction are publicly available

But the method does not win everywhere. On arith+func and on the parametric StableHLO-Held-Out-200 set, where correctness depends on attribute values rather than code structure, the larger models matched the small one or beat it. The authors openly label these cases "non-win cells."

"Schema priors let

SmolLM2-1.7B match or beat code models with 15-34B parameters at 8-25x faster generation," the study's arXiv abstract states.

What this means

For narrow, structurally strict languages like MLIR, specification-derived constraints can replace expensive fine-tuning: a small model with the right "rails" outperforms a large one and runs several times faster. That makes compiler-code generation cheaper and removes the problem of scaling to every new dialect.

Frequently asked questions

What is MLIR?

MLIR (Multi-Level Intermediate Representation) is an intermediate code representation used by the ML compilers behind TensorFlow, JAX/StableHLO, PyTorch Inductor and IREE. It is extended with "dialects" for specific tasks.

Why did a 1.7B model beat 34B models?

It was not fine-tuned. Instead, it received decoding constraints derived from the dialect specification. On linalg, this produced 80.0% valid programs, 21-44 points more than CodeLlama-34B, Granite-Code-34B and StarCoder2-15B.

Where did the method fall short?

On the arith+func dialects and the StableHLO-Held-Out-200 set, where correctness depends on attribute values rather than code structure. There, the larger models matched SmolLM2-1.7B or beat it.