Long Context
Long context refers to the capacity of a language model to process and reason coherently over very large sequences of tokens in a single input. Context window sizes have grown from hundreds of tokens in early models to one million or more in systems like Google Gemini 1.5 Pro.
The context window defines the maximum number of tokens a transformer model can attend to simultaneously during a single inference pass. Inputs and outputs together must fit within this limit. A token is roughly 0.75 words in English, so a 1 million-token context window can hold approximately 750,000 words—equivalent to several full-length novels, an entire mid-sized software repository, or many hours of meeting transcripts.
Standard transformer attention scales quadratically with sequence length in both memory and compute, making naively extending context windows prohibitively expensive. Researchers address this through efficient attention algorithms (Flash Attention, ring attention), modified positional encodings such as Rotary Position Embedding (RoPE) with dynamic scaling and YaRN extension, and architectural changes like sliding window attention used in Mistral models. Models are typically pretrained at shorter context lengths and then extended through continued training on longer sequences or positional interpolation techniques.
Long context unlocks qualitatively new use cases: analyzing an entire codebase in one pass, processing hundred-page legal or scientific documents without chunking, maintaining coherent conversational memory across extended interactions, and performing in-context few-shot learning with many demonstrations. It reduces reliance on retrieval-augmented pipelines that fragment documents into chunks and can miss cross-document relationships.
As of 2026, Google Gemini 1.5 Pro supports 1 million tokens in production and has demonstrated functionality at 2 million tokens in research settings. Anthropic's Claude supports 200,000 tokens, and OpenAI's GPT-4o supports 128,000 tokens. A known limitation called the 'lost in the middle' effect shows that models attend less reliably to information positioned in the center of very long contexts compared to information at the edges, and addressing this degradation remains an active research area.