9. Model Inference Infrastructure
What This Subsystem Is and Why It Exists ¶
Every keystroke-fast completion, every "apply this diff," every multi-step agent loop in a Cursor-style editor eventually bottoms out at the same place: a GPU somewhere running a transformer forward pass. Model inference infrastructure is the layer that turns tokens into more tokens, fast enough and cheap enough that an interactive editor feels alive.
The problem it solves is brutal in three dimensions at once:
- Latency: a developer notices delay above ~200 ms for inline completions and gets impatient above ~1–2 seconds for chat. The model has to start producing text almost immediately.
- Cost: frontier models cost real money per token. An editor that calls GPT-4-class models on every keystroke would bankrupt the company. Margins live and die on this layer.
- Scale & reliability: millions of requests/day, spiky load, GPU scarcity, multi-region users, and a hard requirement that the editor never just "hang."
Think of it like a restaurant kitchen. The dining room (the editor UI) wants every dish instantly. The kitchen has a few very expensive ovens (GPUs), some dishes take 30 seconds and some take 10 minutes, and you have to feed hundreds of tables at once without anyone's food going cold. Inference infra is the kitchen's expediting system: batching orders, prepping ingredients in advance (caching), routing simple orders to the fast griddle and complex ones to the slow oven, and having a backup kitchen when the main one catches fire.
How It Works: The Two Phases That Govern Everything ¶
The single most important mental model for LLM serving is that a generation request has two phases with completely different performance characteristics.
Prefill (Prompt Processing) ¶
When you send a prompt, the model must process all input tokens to build up internal state before it can produce the first output token. This is prefill.
- It processes the entire prompt in parallel — all N input tokens go through the network at once.
- It is compute-bound: it saturates the GPU's matrix-multiply units (FLOPs). A long prompt = a lot of math.
- Its output is the KV cache (see below) plus the first token.
- The user-visible metric here is TTFT — Time To First Token. Prefill latency dominates TTFT.
Decode (Token Generation) ¶
After prefill, the model generates output tokens one at a time, each depending on the previous one (autoregression). This is decode.
- It is inherently sequential — you cannot generate token 5 before token 4.
- It is memory-bandwidth-bound, not compute-bound. For each new token the GPU must read the entire model's weights and the growing KV cache out of high-bandwidth memory (HBM). The actual arithmetic per token is tiny; the bottleneck is shoveling gigabytes of weights through memory for every single token.
- The user-visible metric is TPOT/ITL — Time Per Output Token / Inter-Token Latency, often reported as tokens/second.
The punchline analogy: Prefill is reading a whole book at once (you can speed-read in parallel). Decode is writing a book one word at a time, where to write each word you must re-skim the entire dictionary. Prefill burns FLOPs; decode burns memory bandwidth. They bottleneck on different hardware resources, which is why modern systems sometimes disaggregate them onto different GPU pools.
The Data Flow End-to-End ¶
- Editor sends request (prompt + sampling params) over HTTPS, usually opening a streaming connection.
- Load balancer / gateway routes to a region and a model-specific serving cluster.
- Scheduler admits the request into a running batch (continuous batching).
- Prefill: prompt tokens processed, KV cache built, first token emitted.
- Decode loop: one token per step, each streamed back immediately.
- Tokens flow to the client over SSE (Server-Sent Events) as they are produced.
- Request finishes on a stop token, max-length, or client cancellation (huge in editors — users abandon completions constantly).
The KV Cache: The Heart of It All ¶
In a transformer, each token attends to all previous tokens via keys (K) and values (V). Recomputing K and V for the whole sequence on every decode step would be quadratically wasteful. So you cache them: the KV cache stores K and V for every token already seen.
- The cache grows linearly with sequence length and must stay in GPU HBM for fast access.
- It is big: roughly
2 (K and V) × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_value. For a large model with a long context, KV cache for a single request can be several GB. With 100k-token contexts (common in coding agents) this is the dominant memory consumer — often larger than the model weights' share of per-request memory. - KV cache size is the real limit on how many requests you can batch together, and therefore on throughput and cost.
PagedAttention and vLLM ¶
Naively, you'd allocate one big contiguous KV buffer per request sized to the max possible length. That wastes enormous memory to fragmentation and over-provisioning (you reserve for 100k tokens but most requests use 2k).
PagedAttention (introduced by vLLM, 2023) borrows the OS virtual-memory trick: split the KV cache into fixed-size blocks/pages (e.g., 16 tokens each) and allocate them on demand, non-contiguously, via a block table. Benefits:
- Near-zero memory waste (under ~4% vs. 60–80% in naive systems).
- Prefix sharing: if two requests share a prompt prefix (a system prompt, a file header, a few-shot template), they can share the same physical KV blocks copy-on-write. This is the foundation of self-hosted prompt caching.
- 2–4× higher throughput in practice on the same hardware.
vLLM, along with TensorRT-LLM (NVIDIA), SGLang (especially strong at prefix caching via RadixAttention), TGI (Hugging Face), and LMDeploy, are the standard self-hosted serving engines. Cloud providers (Anthropic, OpenAI, Google) run proprietary internal equivalents.
Continuous Batching ¶
GPUs are massively parallel, so running one request at a time wastes them. You batch. But naive static batching waits for the whole batch to finish before starting the next — so one 2000-token generation holds up nine 10-token generations.
Continuous batching (a.k.a. in-flight or iteration-level batching) instead schedules at the token level: every decode step, finished requests are evicted and new ones are admitted into the running batch. This keeps the GPU saturated and is the single biggest throughput win after PagedAttention. It's why a busy serving cluster gets dramatically better $/token than a lightly loaded one — and why latency and cost are fundamentally in tension (bigger batches = higher throughput/lower cost but higher per-request latency).
Streaming Over SSE ¶
Editors stream output so the user sees text appear immediately rather than waiting for the full response.
- Server-Sent Events (SSE) over HTTP is the de facto standard (OpenAI, Anthropic, etc.): a long-lived
text/event-streamresponse withdata: {...}\n\nchunks, terminated bydata: [DONE]. It's one-directional (server→client), which is exactly what generation needs, and it survives proxies better than WebSockets. - Some self-hosted/local setups use gRPC streaming or WebSockets, but SSE dominates public APIs.
- Cancellation matters enormously in editors. A developer types another character and the in-flight completion is now stale. The client aborts the HTTP request; the server must promptly free that request's KV blocks. Good editors cancel aggressively to save money and GPU time.
Speculative Decoding: The Big Latency Win ¶
Decode is sequential and memory-bound, leaving GPU compute idle. Speculative decoding exploits that idle compute to generate multiple tokens per step.
How it works:
- A small, fast draft model (or a cheap heuristic) proposes the next k tokens (e.g., 4–8).
- The big target model verifies all k in a single parallel forward pass (verification is prefill-like — parallel and cheap relative to k separate decode steps).
- Tokens that match what the big model would have produced are accepted; the first mismatch is corrected and the rest discarded.
Because verification is mathematically guaranteed to produce the same distribution as the target model, output quality is unchanged — it's a pure latency optimization. Typical speedups are 2–3× on decode, higher when text is predictable.
Variants and why they matter for editors:
- Draft model: a small model (e.g., a 1–7B model drafting for a 70B+ target).
- Self-speculation / Medusa: extra prediction heads on the same model predict several future tokens.
- EAGLE: feature-level drafting, strong acceptance rates.
- Prompt lookup / n-gram speculation: when the output is likely to copy from the input — exactly the case for "apply this edit" and FIM, where most of the output is unchanged source code — you can "draft" by literally copying spans from the context. This is cheap and extremely effective. This is why apply and refactor operations can feel near-instant: the model is mostly confirming text it's copying, not inventing it.
Model Routing: The Right Model for the Job ¶
A single model can't be simultaneously cheap, fast, and brilliant. So editors run a portfolio and route per task:
- FIM / inline completions → a dedicated tiny, low-latency model (often 1–7B, sometimes custom-trained on fill-in-the-middle). Optimized for sub-200 ms TTFT. Cursor and Copilot both run specialized completion models distinct from their chat models.
- Apply / fast edits → a small specialized model that takes the original file + a terse model-proposed diff and produces the final file. Cursor's "Fast Apply" is the canonical example, commonly implemented as a small model with speculative-edits / prompt-lookup decoding to copy unchanged lines at high speed.
- Easy or latency-sensitive chat → a mid-tier model (e.g., Claude Haiku-class, GPT-4o-mini-class).
- Hard reasoning / agentic planning → a frontier model (Claude Opus/Sonnet-class, GPT-4-class, Gemini Pro-class, o-series reasoning models).
Routing signals: task type (completion vs. chat vs. agent), prompt length, presence of tools, explicit user model selection, and sometimes a learned router that predicts difficulty. The art is keeping expensive frontier calls rare while not visibly degrading quality.
Prompt Caching: Where the Margins Live ¶
Coding requests are extremely prefix-heavy: the same system prompt, the same file, the same retrieved context get re-sent across many turns of a conversation or agent loop. Prefill of those repeated tokens is pure waste — and prefill is compute-expensive.
Prompt caching stores the KV cache for a prefix so subsequent requests skip re-computing it.
- Provider-API caching:
- Anthropic: explicit
cache_controlbreakpoints; cache reads cost ~10% of input price, writes ~25% extra; default TTL ~5 min (extendable). You design the prompt so the stable stuff (system, tools, big files) comes first. - OpenAI: automatic prefix caching for prompts ≥1024 tokens, ~50% discount on cached input, no code changes.
- Google Gemini: both implicit and explicit context caching.
- DeepSeek and others offer similar context-cache pricing.
- Anthropic: explicit
- Self-hosted caching: vLLM/SGLang automatic prefix caching and RadixAttention reuse KV blocks across requests sharing a prefix — the same PagedAttention machinery.
For an agent doing 20 tool-calling turns over the same 50k-token codebase context, prompt caching can cut cost and TTFT by 5–10×. Cache-aware prompt construction (stable content first, volatile content last) is one of the highest-leverage things the editor's backend does.
Cost Economics: The Numbers That Matter ¶
Pricing is per million tokens (MTok), and input is far cheaper than output (output requires sequential decode). Indicative early-2025-era public list prices:
- Frontier tier (e.g., Claude Sonnet-class, GPT-4o-class): ~2.5–3 / MTok input, 10–15 / MTok output.
- Top reasoning/large tier (Opus/GPT-4-class): can run $15 / MTok input, $75 / MTok output.
- Small/fast tier (Haiku, 4o-mini class): 0.10–1 / MTok, sometimes far lower.
- Cached input: often ~10–50% of normal input price.
Why caching and small models dominate margins:
- Output tokens are 3–5× the price of input and far slower to produce → keep responses tight, stream, and cancel.
- Most editor traffic (completions, simple edits) is high-volume, low-difficulty — routing it to a 1–7B model instead of a frontier model is a 10–100× cost swing.
- A self-hosted small completion model at high batch utilization can cost a small fraction of a cent per completion; an unbatched frontier API call for the same thing is orders of magnitude more.
The business reality: a flat-fee editor subscription only works if the median request is cheap. That forces small models + caching + aggressive cancellation, with frontier calls reserved for the moments that actually need them.
Scale, Multi-Region, GPU Scarcity, and Reliability ¶
- GPU scarcity is the binding constraint. H100/H200/A100 capacity is finite and contended. Editors mix self-hosted clusters (for high-volume small models — better unit economics and data control) with frontier-model APIs (for the hard tail).
- Multi-region capacity reduces network RTT (which is pure additive TTFT) and provides failover. Route users to the nearest healthy region.
- Fallback / failover: if the primary provider is rate-limited or down, fail over to a secondary (e.g., the same model on Azure OpenAI vs. OpenAI direct, or Bedrock/Vertex vs. first-party). Degrade gracefully: a smaller model beats no response.
- Timeouts & retries: short connect timeouts, streaming idle timeouts, capped retries with jittered backoff. Never retry a non-idempotent expensive call blindly — and for completions, just drop it (a stale completion is worthless).
- Load shedding & queueing: under overload, shed low-priority background work (e.g., speculative prefetch completions) before user-initiated chat.
Inference Observability ¶
You cannot manage what you don't measure. Key signals:
- TTFT (p50/p95/p99) — the headline latency metric.
- TPOT / tokens-per-second — decode speed.
- End-to-end latency and time-to-cancel.
- Queue/scheduling delay vs. compute time.
- Cache hit rate (huge cost lever).
- GPU utilization & KV-cache memory pressure / preemption rate (when KV cache fills, requests get preempted/recomputed — a quiet killer).
- Tokens in/out, $ per request, per-model traffic split.
- Error/timeout/failover rates per provider and region.
How Leading Products Differ ¶
- Cursor: heavy investment in custom small models for autocomplete ("Cursor Tab," predicting multi-line and next-cursor-location edits) and Fast Apply (small model + speculative-edit decoding to apply LLM-proposed changes at high token/sec). Routes chat/agent to frontier APIs (Anthropic, OpenAI, Google). Aggressive prompt-prefix design and caching.
- GitHub Copilot: pioneered low-latency FIM completions (originally Codex-derived). Now multi-model — lets users pick GPT-4-class, Claude, or Gemini for chat/agents, while completions stay on a fast specialized model. Backed by Azure's GPU fleet.
- VS Code: the editor/extension host; the inference itself comes from Copilot or other extensions/providers. Provides the streaming UI and the Language Model API abstraction.
- Zed: native, performance-obsessed editor; integrates multiple provider APIs and supports local models; emphasizes low-latency streaming and a fast edit-prediction model (Zeta).
- JetBrains: ships a local, on-device small completion model (runs on the developer's CPU/GPU for instant single-line completions and privacy) plus cloud AI Assistant for chat/larger tasks — a clear small-local / big-cloud split.
- Sourcegraph (Cody): routing across multiple providers, strong on retrieval/context; historically used StarCoder-class models for fast completions and frontier models for chat.
- Continue: open-source, bring-your-own-model; routes to any provider or local (Ollama, vLLM, LM Studio). The user owns the inference layer.
- Aider: terminal-based, provider-agnostic; leans on frontier models and pioneered efficient diff/edit formats (e.g., search-replace, unified diff) to minimize output tokens — an application-layer cost optimization that complements speculative apply.
- Claude Code: Anthropic's agentic CLI; runs on Claude models, leans hard on prompt caching and tool-use loops; routes between Claude tiers (e.g., a faster model for lightweight steps, a stronger model for reasoning).
- Windsurf: agentic IDE (Cascade); mixes its own fast models for completions/apply with frontier APIs for reasoning, similar in spirit to Cursor.
The common pattern across all of them: specialized small/local models for the hot path (completions, apply), frontier APIs for the hard tail, and caching everywhere. Where they differ is how much is self-hosted vs. outsourced, and how much custom model training they do.
Failure Modes and Mitigations ¶
- Slow TTFT (sluggish feel): caused by long prompts (prefill-bound), cold caches, queueing, or distant region. Mitigate with prompt caching, prefix sharing, smaller/closer models, prefill optimization, and shedding background load.
- Slow decode / low tokens/sec: memory-bandwidth-bound large model or KV-cache pressure. Mitigate with speculative decoding, quantization (FP8/INT8/INT4 weights), and better batching.
- KV-cache exhaustion → preemption thrash: too many long-context requests fill HBM; the scheduler preempts and recomputes, tanking latency. Mitigate with PagedAttention, admission control, context-length caps, and capacity headroom.
- Stale completions wasting GPU: user keeps typing. Mitigate with debouncing, aggressive client-side cancellation, and short timeouts.
- Provider outage / rate limits (429s): mitigate with multi-provider failover, regional redundancy, backoff with jitter, and graceful degradation to a smaller model.
- Cost blowup: an agent loop re-sending huge context every turn. Mitigate with prompt caching, context trimming/compaction, tight output budgets, and model routing.
- Quality regression from over-aggressive optimization: too-low quantization or a too-weak draft/router model degrades output. Speculative decoding itself is safe (same distribution); routing and quantization are not — guard with evals and A/B monitoring.
- Streaming stalls / dropped connections: proxies buffering SSE, idle timeouts. Mitigate with proper
Cache-Control: no-cache, disabled proxy buffering, heartbeat events, and idle-timeout tuning.
Interview Talking Points ¶
- LLM serving has two phases: prefill (parallel, compute/FLOP-bound, sets TTFT) and decode (sequential, memory-bandwidth-bound, sets tokens/sec). They bottleneck on different resources — this is the core mental model.
- The KV cache stores per-token keys/values; it grows with context length, lives in GPU HBM, and is the real limit on batch size, throughput, and cost.
- PagedAttention (vLLM) = OS-style paging for the KV cache → near-zero fragmentation + prefix sharing → 2–4× throughput. Continuous batching schedules at the token level to keep GPUs saturated.
- Speculative decoding uses a cheap draft to propose tokens the big model verifies in one pass — 2–3× faster decode, identical output distribution. Prompt-lookup speculation makes apply/FIM near-instant because output mostly copies the input.
- Model routing: tiny dedicated models for completions/apply, mid-tier for easy chat, frontier for hard reasoning. Most traffic must be cheap or the economics collapse.
- Prompt caching (Anthropic explicit ~10% cached reads; OpenAI automatic ~50% off ≥1024 tokens; vLLM/SGLang automatic prefix caching) cuts cost/TTFT 5–10× for prefix-heavy agent loops. Stable content first.
- Output tokens cost 3–5× input; caching + small models dominate margins; aggressive cancellation saves real GPU time.
- Reliability = multi-region, multi-provider failover, timeouts with jittered backoff, load shedding, and observability on TTFT/TPOT/cache-hit/KV-pressure.
Likely Interview Q&A ¶
Q1. Why do prefill and decode have different bottlenecks, and why does it matter? Prefill processes all prompt tokens in parallel, so it saturates the GPU's matrix-multiply units — it's compute/FLOP-bound and dominates Time-To-First-Token. Decode generates one token at a time; each step must stream the entire model's weights (and the growing KV cache) out of HBM while doing tiny arithmetic, so it's memory-bandwidth-bound and sets tokens/sec. It matters because the optimizations differ: prefill benefits from caching and prefix sharing; decode benefits from speculative decoding, quantization, and bandwidth. Some systems even disaggregate the two onto separate GPU pools so a long prefill doesn't stall others' decode.
Q2. What is the KV cache and why does it limit how much you can serve? It caches the attention keys and values for every token already processed so you don't recompute them each decode step. It lives in fast GPU memory and grows linearly with sequence length — for long coding contexts it can exceed several GB per request. Since GPU HBM is fixed, total KV-cache capacity caps how many requests you can batch concurrently, which directly caps throughput and drives up cost per token. PagedAttention exists precisely to use that memory efficiently.
Q3. Explain speculative decoding and whether it hurts quality. A small draft model (or a heuristic like copying from the prompt) proposes the next k tokens; the large target model verifies all k in a single parallel forward pass; matching tokens are accepted and the first mismatch is corrected. It exploits the idle compute during memory-bound decode to get 2–3× speedups. Crucially, the verification step is constructed so the accepted tokens follow the exact same probability distribution as the target model alone — so quality is unchanged; it's a pure latency win. For "apply edits" and fill-in-the-middle, prompt-lookup speculation copies unchanged source spans, giving huge speedups because the model is mostly confirming, not generating.
Q4. How does prompt caching work and why is it so impactful for a coding agent? Coding requests re-send the same prefix constantly — system prompt, tool definitions, the open file, retrieved context — across many conversation/agent turns. Caching stores the KV cache for that prefix so repeated prefill is skipped. Anthropic uses explicit cache_control breakpoints (cached reads ~10% of input price); OpenAI caches automatically for prompts ≥1024 tokens (~50% off); self-hosted vLLM/SGLang do automatic prefix caching via PagedAttention/RadixAttention. For a 20-turn agent over a 50k-token context, this can cut cost and TTFT 5–10×. The design rule: put stable content first and volatile content last so the cacheable prefix is as long as possible.
Q5. Why not just use the best frontier model for everything? Cost and latency. Frontier models can be 10–100× more expensive than a small model and far slower (lower tokens/sec, higher TTFT). Most editor traffic — inline completions, simple applies — is high-volume and low-difficulty, perfectly served by a 1–7B specialized model with sub-200 ms TTFT. A flat subscription only stays profitable if the median request is cheap, so you route: tiny models for the hot path, mid-tier for easy chat, frontier for the hard reasoning tail. Using frontier for everything would be slow and unaffordable.
Q6. A user types a character and a completion is mid-flight. What happens, and why does it matter? The pending completion is now stale, so the client aborts the HTTP/SSE request immediately. On the server, that frees the request's KV-cache blocks and removes it from the running batch. It matters because completion traffic is enormous and developers type fast — without aggressive debouncing and cancellation you'd waste massive GPU time and money generating tokens nobody will ever see. Good editors treat cancellation as a first-class cost lever.
Q7 (System design / hard). Design the inference backend for a Cursor-style editor serving millions of developers. Walk through the architecture and tradeoffs. I'd split traffic by task. Completions and apply: self-hosted small specialized models (1–7B) on vLLM/SGLang clusters with PagedAttention, continuous batching, speculative/prompt-lookup decoding, FP8 quantization, and sub-200 ms TTFT targets — self-hosting wins on unit economics at this volume and keeps code on infra we control. Chat and agents: route to frontier APIs (Anthropic/OpenAI/Google) with a difficulty-aware router, defaulting to mid-tier and escalating only when needed. Across both, design prompts prefix-first to maximize prompt-cache hits. A regional gateway routes users to the nearest healthy cluster to minimize RTT/TTFT and provides failover; each frontier model has a secondary provider (e.g., direct + Bedrock/Azure/Vertex) for outages and rate limits. Add admission control and context caps to prevent KV-cache exhaustion, load shedding that drops speculative background completions first, and aggressive cancellation. The central tradeoff is latency vs. throughput/cost: larger batches lower $/token but raise per-request latency, so I'd run latency-class SLOs per task type (tight for completions, looser for agents) and size batches accordingly. Observability on TTFT/TPOT/cache-hit/KV-pressure/failover drives capacity planning.
Q8 (System design / hard). You're told inference spend is growing faster than revenue. Where do you cut without hurting UX? First, instrument: break spend down by task type, model, input vs. output tokens, and cache-hit rate. Likely wins, in order of leverage: (1) Prompt caching — restructure prompts prefix-first and adopt provider caching; agent loops re-sending context are usually the biggest leak, and caching is near-free UX-wise. (2) Output discipline — output tokens cost 3–5× input and are the slow path; use compact diff/edit formats (like Aider's), tighter system prompts, and bounded max-tokens. (3) Routing — move easy chat and all completions off frontier models onto small/self-hosted ones; add a learned router. (4) Cancellation & debouncing — kill stale completions. (5) Self-host the hot path with quantization and high batch utilization. (6) Speculative decoding to raise tokens/sec so each GPU serves more. The constraint is quality: routing and quantization can degrade output, so gate every change behind offline evals and online A/B monitoring; speculative decoding and caching are safe because they don't change the output distribution. I'd protect TTFT/quality SLOs as hard guardrails and optimize cost underneath them.
Q9. What goes wrong under load, and how do you keep the editor from hanging? Under load the KV cache fills, the scheduler starts preempting and recomputing requests (thrash), TTFT spikes, and you start seeing 429s from providers. Mitigations: PagedAttention plus admission control and context-length caps to bound KV usage; capacity headroom and autoscaling; load shedding that drops low-priority background prefetch before user chat; multi-region and multi-provider failover with jittered backoff; and short connect/idle timeouts so a stuck call fails fast. Critically, the client should always degrade gracefully — fall back to a smaller model or show a clear error rather than spin forever. The guiding principle: a fast wrong-sized answer beats an infinite hang.