COFFEQA Adds 8.77 EM on TriviaQA to Few-Shot Distillation Using Counterfactuals

Researchers have proposed COFFEQA, a way to distill a language model for context-based question answering when only a few dozen training examples are available. On TriviaQA with 32 pairs and T5-family models, the method outperforms distillation on factual examples alone by 8.77 points of average EM and 9.98 points of average F1. The experiments cover five in-domain and three out-of-domain QA benchmarks, with T5 and Qwen2.5 as the models. The work appeared on arXiv in the cs.CL section as a preprint and has not yet been peer-reviewed.

Example selection matters more than it seems

Knowledge distillation lets a small student model reproduce the behavior of a large teacher model under limited resources. The process is usually resource-hungry, requiring a lot of data and compute. This is especially visible in generative tasks, where the student imitates a full-length answer rather than picking a class. Few-shot distillation removes part of the problem by transferring the teacher's knowledge from a small sample.

The approach has a weak spot. The authors show that quality depends heavily on the specific samples chosen. The most surprising result concerns uncertain examples near the teacher's decision boundary. They seem like the most useful ones. In practice, such examples noticeably degrade the final distillation. With a small budget, every bad sample is expensive.

Counterfactuals instead of one more fact

COFFEQA adds an extra signal to hard factual examples. In open-book QA mode, where the model answers from a supplied context, a counterfactual is defined as a minimally modified context after which the teacher gives a different answer. The result is an "original context — edited context" pair with two different teacher answers. The student sees where the decision boundary runs, not just the points on one side of it. The authors provide a geometric justification: such pairs bound the divergence between the teacher's and the student's boundaries.

Not every context edit is suitable. That is why the paper introduces the Context Following Difficulty (CFD) metric. It checks counterfactuals before distillation begins against two criteria: whether the teacher's answer matches the intended alternative, and whether it depends on the modified context. Only pairs that pass the check go into training.

What the description of the work tells us:

• Method: few-shot distillation on "fact — counterfactual" pairs selected by CFD.
• Task: generative open-book QA.
• Models: T5 and Qwen2.5 families.
• Benchmarks: five in-domain and three out-of-domain.
• Best reported result: TriviaQA, T5, 32 pairs, +8.77 average EM and +9.98 average F1 over the baseline.
• Comparison conditions: the same few-shot budget; the baseline uses only factual examples.

What to do with this in practice

For a team compressing a model for RAG or internal document search, the work offers two concrete takeaways. First, don't pick the most contentious examples for a small sample just because they are "hard": according to the authors, this can backfire. Second, generating counterfactuals looks like a cheap way to get more out of the same number of samples. All it takes is changing the context and checking whether the teacher changes its answer. A filter like CFD can be reproduced as a separate quality-control step.

Caution is also warranted. The description does not make clear how the method behaves with closed teacher models or outside context-based QA. The +8.77 EM figure applies to one dataset and one configuration, with T5 and 32 pairs. The results for Qwen2.5 and for out-of-domain benchmarks are not disclosed in the abstract. Before building the approach into a production pipeline, it is worth waiting for the code, full tables, and independent reproductions.