Agentic Editor · Study Guide

8. Context Management & Prompt Construction

What This Subsystem Is and Why It Exists

Every request an agentic editor sends to an LLM is a single flat string (well, a structured list of messages that flattens into tokens). The model has no memory of your repo, no filesystem, and no awareness of what you did 10 minutes ago. Everything the model knows about your task in this turn must be packed into one finite window of tokens. Context management is the discipline of deciding what goes into that window, in what order, and how much — under a hard size ceiling and a real-money meter.

Think of the context window as a moving truck of fixed volume. You have a furniture warehouse (your whole codebase, docs, chat history, tool outputs) far larger than the truck. Context management is the loading crew: it decides which boxes ride along, how to stack them so the important ones are reachable, and what gets left in the warehouse for a later trip (just-in-time retrieval).

Why it's the make-or-break subsystem:

  • The window is small relative to a codebase. Modern windows run 128K (GPT-4o class), 200K (Claude Sonnet/Opus), up to 1M (Gemini 1.5/2.5, Claude with the 1M beta). A medium repo is tens of millions of tokens. You can fit roughly 0.1–1% of a real project at once.
  • Tokens cost money and time. Input tokens are billed per call, and latency (especially time-to-first-token) scales with prompt length because of the prefill/prompt-processing step. A bloated prompt is slower and pricier and, counterintuitively, often worse.
  • More context is not better context. Beyond a point, irrelevant tokens actively degrade output — the "lost in the middle" and "context rot" effects. The job is precision, not volume.

So this subsystem solves three coupled problems at once: fit (stay under the limit), economics (minimize cost/latency via caching and truncation), and quality (surface the right tokens in the right place so the model actually uses them).

How It Works: Prompt Assembly, Step by Step

When you hit Enter on a chat message or trigger an inline edit, the editor runs a prompt assembler that concatenates layers in a deliberate order. A representative pipeline:

  1. System prompt. Static instructions: who the model is ("you are a coding assistant in an IDE"), the tool/function schemas it can call, output format rules, safety constraints. This is large (often 1K–10K+ tokens with tool defs) and changes rarely — which matters enormously for caching (see below).

  2. Project rules / steering files. User- and repo-level instructions: CLAUDE.md (Claude Code), AGENTS.md (the emerging cross-tool convention, used by Codex/Cursor/others), .cursorrules or .cursor/rules/*.mdc (Cursor), .github/copilot-instructions.md (Copilot), .windsurfrules (Windsurf), Continue's rules. These encode "always use pnpm," "we use tabs," "no any in TypeScript." Loaded near the top so they govern everything after.

  3. Retrieved chunks. Output of the retrieval subsystem (embeddings/semantic search + lexical search, sometimes a reranker). Code snippets, doc fragments, symbol definitions judged relevant to the query.

  4. Open / attached / pinned files. What you're actively looking at — the active editor buffer, visible tabs, files dragged into the chat. High signal because it reflects current intent.

  5. @-mentions. Explicit user pointers: @file, @folder, @symbol, @docs, @web, @git. These are the strongest relevance signal in the whole pipeline because the human hand-picked them; assemblers treat them as near-mandatory inclusions.

  6. Conversation history. Prior user/assistant turns and, critically, tool results (file reads, terminal output, search hits). In long agent sessions this becomes the single largest and fastest-growing consumer of the budget.

  7. The current user message — placed last, closest to the model's "attention spotlight."

The assembler computes a token budget per layer, fills from highest-priority to lowest, and truncates or drops when the running total approaches the limit (minus a reserve for the model's output, typically a few thousand tokens). The final structure becomes the messages array sent to the API.

The Ordering Problem: Lost in the Middle

Transformers don't attend uniformly across the prompt. The well-documented "lost in the middle" effect (Liu et al., 2023) shows retrieval/QA accuracy is highest when key info sits at the beginning or end of the context and sags in the middle — a U-shaped curve. Models are also recency-biased: the last few hundred tokens carry outsized weight on the next token.

Practical ordering strategy that falls out of this:

  • Front: stable, authoritative material — system prompt, project rules. (Also a cache requirement; see below.)
  • Middle: bulk retrieved context and history — the stuff you can afford to have slightly under-attended.
  • End: the most decision-relevant, freshest material — the active file, the specific @-mentioned snippet, and the actual user question.

A common refinement is "sandwiching": restate the critical instruction or the exact question both before and after a big block of retrieved code, so a model that drifts in the middle re-anchors at the end.

Compression and History Summarization

Agent sessions don't end; they accumulate. After 30 tool calls you may have 150K tokens of file dumps and terminal logs. Strategies to keep sessions alive without blowing the window:

  • Summarization / compaction. When history crosses a threshold (e.g. ~70–80% of window), an LLM call condenses older turns into a compact summary ("we refactored auth.ts, tests pass, TODO: update callers"), and raw turns are dropped. Claude Code's /compact and its auto-compaction do exactly this; Cursor and others summarize older messages automatically.
  • Tool-result truncation. Cap each tool output (e.g. first/last N lines of a file, head of a log), or replace a giant blob with a pointer ("file X, 2,400 lines, summarized — re-read on demand").
  • Sliding window + pinned anchors. Keep the last K turns verbatim, keep pinned items (project rules, the task statement) always, summarize the rest.
  • Externalize memory. Write durable facts to a scratchpad file or memory store and re-retrieve on demand instead of carrying them in-context — turning expensive "RAM" into cheap "disk."

The analogy: history management is garbage collection for the context window. You can't keep every allocation; you compact the heap and keep live references.

Prompt Caching / KV-Cache Reuse — Engineering a Stable Prefix

This is the highest-leverage cost/latency trick, and a favorite interview topic.

When a transformer processes a prompt, it computes key/value (KV) tensors for every token in every attention layer — the "prefill." For two requests that share an identical leading prefix, the provider can reuse the cached KV state for that prefix instead of recomputing it. Anthropic, OpenAI, Google, and DeepSeek all expose this.

The economics are dramatic:

  • Cost: cached input tokens are billed at a steep discount — Anthropic charges ~10% of normal input price on a cache hit (with a one-time write surcharge of ~25%); OpenAI prices cached input at ~50% and does it automatically; DeepSeek similarly discounts cache hits.
  • Latency: skipping prefill on a long prefix can cut time-to-first-token by 50–80% on cache hits — the difference between an inline completion feeling instant and feeling laggy.

The catch: caching is prefix-based and exact. The cache matches from the start of the prompt up to the first differing byte. Change one token early and everything after it is a cache miss. This dictates the entire prompt architecture:

  • Put the most stable content first — system prompt, tool schemas, project rules — so it's cacheable across every request in a session and even across users sharing the same system prompt.
  • Put volatile content last — the user's new message, freshly retrieved chunks.
  • Never inject churn early. Don't put a live timestamp, a random request ID, or reordered retrieval results near the top; you'd invalidate the whole cache every turn.
  • Providers expose controls: Anthropic uses explicit cache_control breakpoints (mark up to 4 prefix segments as cacheable); OpenAI/Google cache automatically with TTLs (often a few minutes of idle before eviction).

The mental model: a cache hit is a bookmark. Keep the first chapters of the book identical and the provider opens straight to your bookmark instead of re-reading from page one. Rewrite chapter one and the bookmark is worthless.

Token Counting and Truncation Policy

The assembler must know the cost of every candidate before sending. In practice:

  • Tokenization. Count with the model's tokenizer — tiktoken (cl100k_base/o200k_base) for OpenAI-family, the provider's counter for Claude/Gemini. Rough rule of thumb: ~4 characters or ~0.75 words per token for English; code is denser and less predictable (punctuation, identifiers). Many clients keep a fast local estimator and reconcile against the API's returned usage.
  • Budgeting. Total window − reserved output tokens − safety margin = input budget. Allocate per layer, fill by priority, and when the budget is exhausted, drop lowest-priority items first (usually oldest history, then lowest-ranked retrieval chunks). @-mentions and project rules are typically protected from truncation.
  • Taming tool-result bloat. The biggest silent budget killer. Mitigations: cap output size per tool, paginate large reads, stream-then-summarize, strip ANSI/whitespace, dedupe repeated reads of the same file, and prefer "grep then read the matching range" over "cat the whole file." A single unbounded npm test log can eat 20K tokens.

The Strategic Tradeoff: Stuffing vs. Just-in-Time Retrieval

Two philosophies, and the industry has been migrating between them:

  • Context stuffing (front-load): retrieve aggressively up front, pack the window with everything plausibly relevant, send one big prompt. Fewer round-trips, lower latency per task, but risks context rot — irrelevant tokens distract the model, raise cost, and lose-in-the-middle hurts accuracy.
  • Agentic / just-in-time retrieval: send a lean prompt plus tools (read_file, grep, list_dir, semantic_search). The model decides what to pull and fetches it on demand across multiple turns. Keeps the window tight and relevant, but adds latency (more round-trips) and depends on the model's judgment to find the right files.

The modern consensus, visible in Claude Code, Cursor's agent mode, Aider's repo-map, and Windsurf's Cascade: lean prompt + strong tools + minimal pre-retrieval, then pull just-in-time. Frontier models with good tool use navigate a repo more precisely than a blind top-k embedding dump. Context rot — degraded reasoning as the window fills with marginally relevant tokens — is now treated as a first-class failure mode, so "less but right" beats "more just in case."

Concrete Technologies, Algorithms, and Numbers

  • Window sizes: GPT-4o ~128K; Claude 3.5/3.7/4 Sonnet & Opus 200K (1M in beta); Gemini 1.5/2.5 up to 1M (2M previewed); DeepSeek-V3 128K. Output is separately capped (often 4K–64K+).
  • Prefix-cache discounts: Anthropic cache read ~0.1× input, cache write ~1.25×, default TTL ~5 min (extendable to 1h); OpenAI automatic prompt caching ~0.5× on ≥1024-token prefixes; Gemini context caching billed per cached token-hour.
  • Tokenizers: tiktoken (o200k_base for GPT-4o), SentencePiece/BPE variants per vendor. ~4 chars/token heuristic.
  • Retrieval stack feeding context: embeddings (e.g. OpenAI text-embedding-3, Voyage Code, or local models) → ANN index (HNSW/IVF via FAISS, Qdrant, LanceDB) → optional cross-encoder reranker → chunking by AST/symbol boundaries (tree-sitter) rather than fixed lines.
  • Lost-in-the-middle: Liu et al. 2023 — accuracy U-curve across position.
  • Compaction trigger: commonly ~70–85% window fill before summarizing.
  • Tool-result caps: typical defaults like 250 lines / ~25K chars per read in agent harnesses.

How Leading Products Implement It

  • Cursor: Hybrid retrieval (server-side embeddings of the repo + lexical) feeds Chat/Composer; agent mode leans on tools and @-mentions (@Files, @Code, @Docs, @Web, @Git). Rules via .cursorrules / .cursor/rules/*.mdc (scoped, auto-attached by glob). Auto-summarizes long sessions. Heavy reliance on prompt caching to keep Composer affordable.
  • GitHub Copilot: Inline completions use a tight, latency-critical context window — current file, neighboring tabs, recently edited files, and "neighboring tabs" heuristics rather than full RAG, to hit sub-200ms-ish completion latency. Copilot Chat/agent and @workspace add repo-wide retrieval. Steering via .github/copilot-instructions.md.
  • VS Code (Copilot Chat backbone): Exposes context as #file, #selection, #codebase, @workspace participants; the workspace index (local embeddings, or remote for GitHub-hosted repos) supplies retrieved chunks.
  • Zed: Fast-native editor; assistant panel with explicit context via slash commands and /file, emphasizes user-curated context and tool use over opaque auto-retrieval. Supports prompt caching with Anthropic models.
  • JetBrains (AI Assistant / Junie): Uses IDE's deep PSI/symbol index — it already has a precise semantic model of the project, so context selection leans on the IDE's existing code graph rather than pure embeddings.
  • Sourcegraph Cody: Strongest on enterprise-scale retrieval — combines its code search engine (keyword + structural) with embeddings to fetch context across huge multi-repo monorepos; context fetching is its core differentiator.
  • Continue (open source): Transparent, configurable context providers (@codebase, @file, @docs, @terminal, custom MCP) and a rules system; you literally configure the assembly pipeline.
  • Aider: Famous for the repo map — a tree-sitter-derived, ranked summary of the repo's symbols/signatures that gives the model a "table of contents" within a small token budget, plus explicit /add of files into context. Lean and Git-native.
  • Claude Code: CLAUDE.md hierarchy (user/project/local) as steering; lean prompt + powerful tools (Read/Grep/Glob/Bash) for just-in-time retrieval; explicit /compact and auto-compaction; aggressive use of Anthropic prompt caching on the stable system+tools prefix.
  • Windsurf (Cascade): Agentic flow that builds and maintains its own context map of the codebase, pulling files just-in-time during multi-step tasks; .windsurfrules for steering.

Where they differ most: how much they pre-retrieve vs. fetch on demand (Sourcegraph/Cursor lean retrieval-heavy; Claude Code/Aider lean tool-heavy), and how they index (embeddings vs. IDE symbol graph vs. tree-sitter repo map).

Failure Modes and Mitigations

  • Lost in the middle: key context ignored because it sat mid-prompt. → Reorder (authoritative front, decisive last), sandwich the question, rerank so top hits go to the edges.
  • Context rot / distraction: too much marginal context degrades reasoning and invents irrelevant changes. → Retrieve less, prefer just-in-time tools, rerank hard, cap top-k.
  • Cache thrash: an early-injected timestamp/ID/reordered block invalidates the prefix cache every turn, silently 5–10×-ing cost and latency. → Freeze the prefix; push all volatility to the tail; verify with the API's cache_read_input_tokens usage field.
  • Truncation of the wrong thing: naive "drop oldest" deletes the task statement or a pinned constraint. → Priority tiers with protected items (@-mentions, rules, current task) never truncated.
  • Tool-result bloat / window exhaustion: one giant log or cat of a 5K-line file blows the budget mid-task. → Per-tool size caps, pagination, grep-then-read, summarize-on-overflow.
  • Stale context after edits: model reasons over a file version that changed three tool calls ago. → Re-read before write, invalidate cached file snapshots on edit, include current buffer last.
  • Conflicting instructions: project rules contradict the system prompt or each other. → Clear precedence order (system > project > local), and keep rules terse and non-overlapping.
  • Summarization information loss: compaction drops a detail the model later needs. → Summarize conservatively, keep recent turns verbatim, and externalize hard facts to a memory file that can be re-read.
  • Hidden token blowups from images/binaries: screenshots and large pastes cost far more than expected. → Count and cap multimodal inputs explicitly.

Interview Talking Points

  • The context window is a fixed, expensive budget; the core job is fit + economics + quality, not "stuff everything."
  • Assembly order: system prompt → project rules (CLAUDE.md/AGENTS.md/.cursorrules) → retrieved chunks → open/attached files → @-mentions → history/tool results → current message.
  • Lost in the middle is real: put authoritative content at the front, the decisive question/file at the end; sandwich critical instructions.
  • Prompt caching reuses the KV cache for an identical prefix — ~10% input cost on Anthropic hits, 50–80% TTFT reduction. Engineer a stable prefix (static first, volatile last); never inject churn early.
  • @-mentions are the strongest relevance signal because a human chose them; protect them and project rules from truncation.
  • Long sessions need compaction/summarization and tool-result caps — history is the fastest-growing consumer.
  • Industry trend: from context stuffing toward lean prompt + tools + just-in-time retrieval to fight context rot.
  • ~4 chars/token; windows 128K–1M; reserve output tokens; budget by priority tiers.

Likely Interview Q&A

Q1. Walk me through how you'd assemble the prompt for a single chat turn in a Cursor-style editor. Layered, priority-ordered concatenation: a stable system prompt and tool schemas first; then user/project rules (CLAUDE.md/.cursorrules); then retrieved code chunks from hybrid search; then open and attached files; then any @-mentions; then conversation history and tool results; and the current user message last. I compute a token budget (window minus reserved output minus safety margin), fill highest-priority first, and truncate lowest-priority items (oldest history, lowest-ranked chunks) when nearing the cap — while protecting @-mentions, project rules, and the task statement. Ordering also respects lost-in-the-middle (authoritative front, decisive last) and prefix caching (stable front, volatile last).

Q2. What is "lost in the middle" and how does it change your design? Transformers attend non-uniformly: accuracy is highest for content at the start and end of the prompt and dips in the middle (a U-curve, per Liu et al. 2023), with strong recency bias toward the last tokens. So I front-load authoritative, stable material and place the most decision-relevant content — the active file, the specific snippet, the actual question — at the very end. For big retrieved blocks I "sandwich": restate the question before and after them so the model re-anchors.

Q3. Explain prompt caching and how prompt structure interacts with it. Providers cache the per-layer key/value tensors computed during prefill. If a new request shares an identical leading prefix, they reuse that KV state instead of recomputing it — Anthropic bills cache reads at ~10% of input price and it can cut time-to-first-token by more than half. The catch is it's exact-prefix matching: the first differing byte invalidates everything after it. So I architect the prompt with all stable content (system prompt, tool defs, project rules) first and all volatile content (new user message, fresh retrieval) last, and I never inject a timestamp, random ID, or reordered block early — that would thrash the cache and silently multiply cost and latency.

Q4. How do you keep a long agent session from running out of context? Treat history like a managed heap. Cap each tool result (paginate reads, grep-then-read instead of cat, truncate logs). Keep the last K turns verbatim plus always-pinned items (rules, task). When fill crosses ~70–80%, run an LLM compaction pass that summarizes older turns and drops the raw versions — Claude Code's auto-compact and /compact do this. Externalize durable facts to a scratchpad/memory file and re-retrieve on demand rather than carrying them in-window.

Q5. Stuffing the context vs. just-in-time retrieval — which and why? Stuffing front-loads everything plausibly relevant: fewer round-trips but risks context rot and lost-in-the-middle, and costs more. Just-in-time gives the model tools and lets it pull files across turns: tighter, more relevant context but more latency and dependence on the model's navigation. With strong tool-using frontier models I lean just-in-time (Claude Code, Aider's repo map, Windsurf Cascade) for precision and to avoid context rot, but I keep a small high-signal pre-retrieval for @-mentions and the active file so the first turn isn't blind. It's a latency-vs-quality dial I'd tune per surface — inline completion needs stuffing for speed, agentic editing favors JIT.

Q6. (System design) Design the context engine for an editor over a 10M-LOC monorepo on a 200K-token model. You can hold ~0.1% of the repo, so the architecture is retrieval-first with JIT tools. Build an offline index: tree-sitter to chunk by symbol, embeddings into an ANN store (HNSW), plus a lexical/structural search engine (à la Sourcegraph) for exact symbol lookups; refresh incrementally on file save via a watcher. At query time: hybrid retrieve (semantic + lexical) → cross-encoder rerank → take a tight top-k → place at prompt edges. Provide read_file/grep/list tools so the model fetches more JIT. Engineer a frozen prefix (system+tools+rules) for cache hits across the session; push retrieval and the query to the tail. Budget tiers with @-mentions protected; cap tool outputs; compact history past ~75%. Track cost via cache-read ratio and quality via retrieval hit-rate evals. Differentiator at this scale is retrieval precision and incremental indexing, not window size.

Q7. (Tradeoff) You can cut cost 5× with caching but the team wants per-request personalized context injected at the top. Resolve it. Those goals collide: per-request volatile data at the top breaks the prefix and kills the cache. The fix is to relocate, not remove. Keep the genuinely static prefix (system, tools, stable team rules) first and cacheable; move personalized/volatile context to the tail, after the cached segment, where it doesn't invalidate the prefix. If personalization is semi-stable per user (e.g. their coding preferences), give it its own cache breakpoint as a second stable segment so it caches per-user. Only truly per-request data (the new message, fresh retrieval) stays uncached at the very end. You keep most of the 5× saving while still personalizing — you just stop putting the volatile bytes where they poison the cache.

Q8. How do you decide what to truncate when over budget, and what must never be cut? Priority tiers. Protected, never truncated: the current user message/task, @-mentions (human-chosen = highest signal), and project rules/constraints. Truncate first: oldest conversation turns (after summarizing them), then lowest-ranked retrieval chunks, then verbose tool outputs (cap or summarize). I also reserve output tokens up front so I never truncate the model's own room to answer. The principle is that explicit human signals and the active task outrank machine-guessed relevance.

Q9. Why can adding more relevant-looking context make answers worse, and how do you detect it? Context rot: as the window fills with marginally relevant tokens, attention spreads thin, mid-prompt content gets ignored, and the model picks up distractors — producing edits to the wrong file or hallucinated APIs. Detection: run retrieval/answer evals that vary top-k and measure accuracy vs. context size (you'll often see a peak then decline), monitor edit-acceptance rates, and A/B lean vs. fat prompts. Mitigation is precision: harder reranking, smaller top-k, JIT tools, and sandwiching the question so the model re-anchors on what actually matters.

Built from Markdown · 22 pages · Read it aloud and burn it in.