Copy-Paste Language Models Add a Copy Head to LLMs With 135M–1.7B Parameters

Copy-Paste Language Models (CPLMs) propose a simple idea: keep the transformer's regular LM head and add a lightweight Copy-Paste head next to it, which learns separately to copy tokens from the context. The authors pretrained models from 135M to 1.7B parameters and report improvements at all scales and on different backbones. Training becomes more efficient, and by their account the training and inference overhead is negligible.

Copying as a separate task

Transformers are generally considered good at copying. The attention mechanism finds the right token in the input and carries it into the output, so a model usually reproduces a name from a document or an identifier from code without trouble. The authors challenge the idea that this is enough. They show that an LLM's ability to copy from context can be strengthened by modeling it explicitly.

Copying mechanisms are not new: pointer-generator networks have long been used in tasks such as summarization. What is new here is that the Copy-Paste head is built into the pretraining of a general-purpose language model rather than a narrow task. The published abstract does not disclose implementation details. How exactly the head mixes its output with the LM head's distribution will become clear only from the full paper.

The LM head's gradient bottleneck

The most interesting part of the work is its explanation of why this helps. The standard LM head projects the model's hidden state into logits over the entire vocabulary. The hidden dimension is much smaller than the vocabulary size, so the whole training signal passes through this projection, and the projection limits how the gradient flows back into the network. This is what people call the gradient bottleneck.

According to the authors, the Copy-Paste head creates an alternative path for the gradient and partially eases this bottleneck. The word "partially" matters here: this is not a claim that the problem is fully eliminated, but an explanation of the observed gain. If the mechanism really works this way, the benefit comes not only from easier copying at inference but also from the model learning more efficiently during pretraining.

What the experiments claim

The abstract gives qualitative results without specific metric values:

• Model range: 135M to 1.7B parameters, pretrained from scratch.
• Improvements are claimed at all scales and on all tested backbones.
• Training efficiency and quality on downstream tasks increase.
• New operating points appear for long-context and retrieval scenarios.
• Training and inference overhead is described as negligible.

Practical takeaways

For teams that pretrain small models, the work offers a cheap candidate for ablations. One extra head does not require rebuilding the pipeline, and the claimed compute cost is close to zero. For RAG and long-context tasks, where the model must accurately carry fragments over from the input, explicit modeling of copying looks like a logical place to improve.

The limitations are also clear. The experiments top out at 1.7B parameters, so it is unknown whether the effect holds for models with tens or hundreds of billions of parameters. The results come from pretraining, not fine-tuning, so applying the method to an already finished model without additional training is unlikely to work. Finally, without concrete numbers from the full paper, it is hard to judge the size of the gain: "improvements at all scales" could mean a noticeable jump or a modest but consistent improvement.

Before adopting it, wait for the code and the results tables. If the claims hold up, this would be a rare case where an architectural tweak costs almost nothing and still fixes a specific weak spot in training.