Structured LLM pruning without accuracy loss: method tested on Llama-3-8B and Vicuna

In July 2026, a paper appeared on arXiv describing a new method for structured pruning of large language models. The researchers identify three systemic problems in porting the AFR technique to structured pruning and propose a unified approach that combines a power transform, sign-preserving aggregation, and percentile-based outlier clipping. Experiments on Llama-3-8B, Vicuna-v1.5-13B and LLaVA-v1.5-13B showed accuracy on par with unstructured pruning, with real inference speedups.

Why structured pruning is better for deployment

Pruning is one of the basic techniques for compressing neural networks: unimportant parameters are removed, and the model becomes smaller and faster. There are two fundamentally different approaches.

Unstructured pruning zeroes out individual weights, producing sparse matrices. Accuracy suffers less, but this yields almost no practical speedup on standard GPUs. It requires specialized kernels or hardware that most teams don't have access to.

Structured pruning removes whole neurons, attention heads or layers. The resulting model has standard density and runs faster on ordinary hardware with no extra tricks. The catch: its quality has traditionally lagged behind unstructured methods. The authors set out to close precisely that gap.

Three problems in adapting AFR

AFR (Adaptive Feature Retention) is an unstructured method that keeps "important" weights based on gradient information. The authors tried to carry it over to the structural level and ran into three obstacles:

• Distribution mismatch: importance scores from different layers have incomparable scales, so straightforward averaging gives a distorted result
• Loss of sign information: the sign of a score shows whether the direction of a weight update agrees with the optimization gradient; most aggregation methods discard this sign
• Influence of outliers: extreme values dominate aggregation and force the removal of structures for reasons unrelated to their real importance

Each of these problems lowers final quality on its own; together they make a direct port of AFR unworkable.

How the proposed solution works

The authors propose three techniques, applied in sequence within a single aggregation scheme.

The power transform equalizes score distributions non-linearly. Linear scaling copes poorly with the heavy tails typical of transformer activations. A power function with a tunable exponent makes scores from different sources comparable.

Sign-preserving aggregation combines the transformed scores without losing directional information. If a neuron receives a score with a particular sign, that is taken into account when selecting candidates for removal, instead of simply averaging absolute values.

Percentile-based outlier clipping removes extreme values before aggregation. The threshold is set in percentiles rather than absolute numbers, which keeps the method stable when the architecture or task changes.

Results on Llama, Vicuna and LLaVA

The method was tested on three open models of different classes:

• Llama-3-8B: accuracy after structured pruning is comparable to the unstructured version of AFR
• Vicuna-v1.5-13B: the conversational model keeps its answer quality with fewer parameters
• LLaVA-v1.5-13B: the multimodal model also showed no significant degradation

The authors stress a "practical inference speedup": compressed models run faster on standard GPUs without special libraries for sparse matrices. This is what makes structured pruning attractive for production deployments, where compute cost is counted directly.

What this means

The work closes a specific technical gap: adapting unstructured importance metrics to structured pruning previously required a trade-off in accuracy or costly recovery fine-tuning. Three targeted fixes (distribution alignment, sign preservation, percentile outlier clipping) make it possible to keep accuracy and get a real speedup at the same time, without post-pruning workarounds.