Context engineering is the work of deciding what information a model has in front of it at the moment it acts. Not just the instruction, but the tool definitions, the retrieved documents, the conversation so far, the output of the last command and any notes carried from earlier. Anthropic describes it as curating and maintaining the best set of tokens during inference, which is a precise way of saying that what you leave out matters as much as what you put in.
Updated September 2026. This area is moving quickly, so check the linked sources for the current position.
Why a bigger context window did not solve this
The intuitive fix, a million-token window, does not work as advertised. In research published on 14 July 2025, Chroma tested 18 models including Claude, GPT, Gemini and Qwen families and found that performance on even simple tasks declined as input length grew, in every model tested. The effects were uneven and sometimes strange: a shuffled, incoherent set of documents outperformed a logically ordered one, and distracting passages hurt more as the input got longer.
Anthropic’s explanation is architectural. A transformer computes relationships between every pair of tokens, so attention is stretched thinner as the input grows, and models have seen fewer long sequences in training than short ones. The practical framing is an attention budget: every token you add spends a little of a finite resource, whether or not it earns its place.
Andrej Karpathy popularised the term in June 2025, calling it “the delicate art and science of filling the context window”. The label stuck because it named something practitioners were already doing badly.
5 context engineering techniques that work
These are the approaches described in published engineering guidance rather than in marketing. Each buys you something and costs you something.
- Just-in-time retrieval. Keep lightweight identifiers in context, such as file paths, queries or links, and load the content only when it is needed. Costs an extra round trip; saves you from carrying documents you never read.
- Compaction. When the window fills, summarise the history and start again from the summary, preserving decisions, unresolved problems and specifics. Costs a model call and some fidelity; buys you a long session that still works.
- Structured note-taking. Have the agent write durable notes to a file outside the window and read them back. Costs discipline in what gets written; buys memory that survives compaction and restarts.
- Tool consolidation. Fewer, non-overlapping, self-contained tools with unambiguous descriptions. Costs design time up front; removes the decision paralysis that comes from near-duplicate options.
- Sub-agent isolation. Give a noisy sub-task its own clean window and return only a short summary to the main thread. Costs tokens and coordination; keeps exploration out of the main context. See our guide to multi-agent systems for when this backfires.
Where context engineering gets hard
The techniques are simple to describe and fiddly to operate. Compaction silently discards things you later need, and you find out several steps afterwards. Notes accumulate until they are their own context problem. Retrieval that looks fine on curated test documents behaves differently on a messy real corpus, and the Chroma results suggest that similarity between the question and the target matters more as input grows, so weak matches degrade faster.
There is also a limit nobody has engineered around. A 166-page survey of the field posted to arXiv in July 2025, drawing on more than 1,400 papers, concluded that models handle complex context far better than they produce equally sophisticated long-form output. Feeding in more does not reliably get more out.
Security is a live concern too. Everything you place in the window is something the model will act on, so retrieved documents and tool descriptions are an attack route. This is the mechanism behind prompt injection, and it is why poisoned memory appears in OWASP’s agentic risk list.
The system prompt is part of the problem
Two failure patterns show up repeatedly in published guidance. One is hardcoding brittle if-then logic into the instructions until the prompt is a badly written program that breaks on the first case its author did not imagine. The other is retreating into vagueness so general that the model gets no usable signal. The advice is to aim between them: concrete guidance, organised into clear sections, with enough reasoning behind each rule that the model can generalise from it. As models improve, that middle band tends to need less scaffolding, not more.
Tool descriptions deserve the same treatment. They are read on every step, they compete with each other for the model’s judgement, and overlapping options produce the kind of hesitation that burns a whole loop. Consolidating five near-identical tools into two well-named ones is often a larger improvement than any rewording of the instructions above them.
How to start without a rewrite
- Measure first. Log how many tokens each run uses and where they come from. Most teams are surprised by which tool output dominates.
- Cut the system prompt to the smallest version that still works, then add back only what a failing test demands.
- Set a compaction threshold deliberately rather than waiting for the window to overflow mid-task.
- Write an evaluation set before tuning. Without one you cannot tell curation from superstition.
If you are coming at this from the writing side rather than the engineering side, our companion article on prompt engineering covers what still matters in the instruction itself, and what an AI agent is explains the loop all of this feeds.
Common questions
What is context engineering in simple terms? It is choosing what a model sees when it acts: the instructions, tool definitions, retrieved documents, history and notes. The aim is the smallest set of high-value information that still produces the right behaviour.
Why not just use a model with a huge context window? Because accuracy falls as input grows. Chroma tested 18 models in July 2025 and found performance declined with length on every one, even on simple retrieval tasks.
Is context engineering the same as RAG? No. Retrieval-augmented generation is one technique within it. Context engineering also covers system instructions, tool design, compaction, memory and how work is split between agents.
What is context rot? It is the observed degradation in a model’s reliability as the number of input tokens increases, including on tasks the same model handles easily at shorter lengths.
How do I know my context changes helped? Build an evaluation set with clear pass criteria and run it repeatedly, since these systems are non-deterministic and a single run proves very little.
Sources and further reading
Where the figures and rules above come from, so you can check them:
- Effective context engineering for AI agents: Anthropic
- Context Rot: how increasing input tokens impacts LLM performance (14 July 2025): Chroma
- A Survey of Context Engineering for Large Language Models: Mei et al., arXiv
- The origin of the term (June 2025): Andrej Karpathy
- Top 10 for Agentic Applications, including memory and context poisoning: OWASP GenAI Security Project
Photo credits: Oulu county archive's shelves 1933 (JOKAKAL3B-3508) by Wikimedia Commons, CC BY 4.0, via Wikimedia Commons. Boîtes archives avant traitement by DMontagne en résidence, CC BY-SA 4.0, via Wikimedia Commons.
Retrieval is a large part of this in practice, which is where a vector database comes in.
Join the discussion