Capstone: "Design Cursor From Scratch" — A 45-Minute System Design Walkthrough
This is the capstone. The preceding 16 sections are your knowledge base; this is the performance. The interviewer says "Design an AI coding editor like Cursor." You have 45 minutes. Your job is not to dump everything you know — it's to drive a structured conversation, make the hard tradeoffs visible, and go deep where it counts. This guide is the script for doing that well.
The single most important framing to internalize and say out loud early: this is not "wrap a chatbot in an IDE." It is a latency-tiered, multi-model distributed system strapped onto a real-time interactive editor, where the seams between an unpredictable LLM and a 16ms typing loop have to be invisible. If you say that in the first two minutes, you've signaled seniority.
Minute 0–5: Clarifying Questions (Do Not Skip This) ¶
Weak candidates start drawing boxes immediately. Strong candidates spend five minutes scoping, because "Cursor" means very different systems depending on the answers. Ask:
- Which surfaces are in scope? Inline autocomplete (Tab), chat/inline-edit, and the autonomous agent are three different products with three different latency budgets. "Should I cover all three or focus on one?" If they let you choose, cover all three at the data-flow level and deep-dive one.
- Scale? "Are we talking 10K developers or 10M?" This determines whether you self-host completion models or call APIs, and whether indexing is per-laptop or a multi-tenant cloud service.
- Repo size? A 50K-line startup repo and a 20M-line monorepo demand completely different retrieval architectures. Always ask.
- Build vs. buy the models? "Do we train our own models or only call frontier APIs?" Cursor does both; the answer changes the inference section entirely.
- Privacy posture? "Consumer product, or enterprises with zero-retention and data-residency requirements?" This is load-bearing — it dictates where embeddings live, where code execution runs, and whether you can use the data flywheel.
- Fork or extension? Worth flagging: "Are we building the editor surface, or shipping into VS Code?" Cursor forks VS Code because the editing interaction (inline apply, Tab-jump) is the product; Copilot ships an extension for reach. State the rule: fork when differentiation lives in the editing surface; extend when it lives in the model/backend and reach matters more.
Lock assumptions and say them back: "I'll assume millions of developers, repos up to a multi-million-line monorepo, we train small models and rent frontier ones, and we need an enterprise privacy tier. Let me know if you want to flip any of these."
Minute 5–10: Requirements ¶
Functional ¶
- Inline completion (Tab): ghost text as you type, accept on Tab; must handle mid-file edits (insert between existing code), not just end-of-file continuation.
- Chat / inline edit (Cmd-K): ask questions and request edits grounded in the repo.
- Agent mode: give a goal ("fix this failing test"), the system reads, edits, runs tests, and iterates autonomously with human review of diffs.
- Codebase awareness: answers and edits reference this repo, not generic code.
- Safe application of edits: multi-file changes land reversibly, reviewably, atomically.
Non-Functional (this is where you score points — be quantitative) ¶
- Latency, tiered by surface. This is the spine of the whole design, so make it explicit:
- Tab completion: sub-300ms end-to-end, ideally <150ms, or the user out-types the suggestion. Time-to-first-token dominates.
- Chat: first token in ~1–2s; total response streams.
- Agent: latency-tolerant (seconds to minutes) but cost- and safety-sensitive.
- Scale: millions of concurrent developers, completions firing on nearly every keystroke pause → potentially millions of requests/minute, most of which are never accepted.
- Cost: a flat-fee subscription only works if the median request is cheap. Output tokens cost 3–5× input; the economics force small models + caching + aggressive cancellation, with frontier calls reserved for the hard tail.
- Privacy: enterprise zero-retention and no-training guarantees; the local↔︎cloud boundary is an explicit design artifact.
- Reliability: the editor must never hang — degrade gracefully (smaller model, fewer context chunks) rather than fail.
The line to deliver: "Every architectural decision in this system is downstream of the latency tier it serves. Tab and the agent are almost opposite systems."
Minute 10–18: High-Level Components & APIs ¶
Draw a hub-and-spoke. The client talks to a backend proxy/orchestrator, which fans out to specialized services. Then immediately answer the question a good interviewer will ask: why a proxy at all, even in privacy mode?
Why the proxy exists (name these — it's a senior signal): API-key custody (the client never holds the OpenAI/Anthropic key — it gets a session token), server-side model routing (so "Auto" mode logic changes without a client release), centralized prompt caching to maximize hit rates, rate limiting and abuse prevention, telemetry/A-B, and hosting vendor-owned models (fast-apply, Tab). "Privacy mode means zero-retention — code transits in memory and is never persisted or trained on — it does not mean code never leaves the machine." Stating this distinction crisply is a known interview differentiator.
Core components:
- Client (forked editor): rope/piece-tree buffer, virtualized rendering, debounce, request cancellation, local context assembly, diff review UI. Keeps the keystroke→paint loop off any I/O or model call.
- Backend proxy / orchestrator: routing, caching policy, auth, the agent loop driver.
- Inference layer: a portfolio of models — tiny FIM model for Tab, fast-apply model, frontier APIs for chat/agent, plus embedding + reranker models.
- Indexing & retrieval service: turns the repo into a searchable index (deep dive below).
- Apply service: materializes intent into byte-precise file edits (deep dive below).
- Tool/execution service: runs commands in a sandbox; closes the agent's feedback loop.
- State stores: Redis (sessions, rate limits), Postgres (metadata), object storage (checkpoints), vector DB (index).
The key API surfaces stay small: complete(prefix, suffix, context), chat(messages, attachments), agent_run(goal) returning a stream of tool calls/diffs, and internal retrieve(query), apply(file, edit_spec), execute(command). Mention that LLM inference is stateless — you replay context every call — which is exactly what lets you load-balance any request to any worker and push session state to external stores.
Minute 18–24: The Three Data Flows (Show You Understand Why They Differ) ¶
(a) Tab completion — the reflex path ¶
Keystroke → debounce ~25–75ms → client assembles a small prompt (cursor prefix/suffix + "neighboring tabs" snippets, scored by cheap token overlap) → single request over a kept-alive HTTP/2 connection → proxy routes to a small (1–7B) FIM-trained model → stream first-token-first → render ghost text → log accept/reject.
Hit the non-obvious points: this path uses Fill-in-the-Middle, not chat completion — the cursor is the boundary between prefix and suffix sentinel tokens, because code is written bidirectionally. It's stateless and cache-aggressive. A separate lightweight contextual filter model decides whether to show the suggestion at all — a bad suggestion is worse than none because it erodes trust and tanks acceptance rate, the north-star metric. And every new keystroke cancels the in-flight request — stale completions waste GPU and money.
(b) Chat — retrieval-grounded ¶
Message + attached context → proxy calls retrieval (hybrid search over the repo index) → assembles a layered, priority-ordered prompt → routes to a mid/frontier chat model → streams over SSE. Emphasize prompt assembly order: stable content (system prompt, tool defs, project rules) first for cache hits and authority; the decisive file and question last (to beat "lost in the middle" and recency bias). @-mentions are the strongest relevance signal because a human chose them — protect them from truncation.
(c) Agent — the autonomous loop ¶
Goal → orchestrator starts a run with a trace ID → ReAct loop: model emits a tool_use block → harness (not the model) executes it (read, grep, edit, run tests) → appends the tool_result as an observation → calls the model again → repeat until done or a budget trips. Say the mental model out loud: "the model is a function from history → next action, and the harness is a while-loop that executes the action and feeds the result back." The compiler/test output is the reward signal that lets it self-correct. Guards: turn caps, token/time budgets, loop detection, approval gates on dangerous tools. Cost grows ~quadratically because the whole transcript is resent each turn — prompt caching of the stable prefix is the #1 mitigation (5–10×).
Minute 24–38: Deep Dives (Pick 2–3 — These Are Where the Interview Is Won) ¶
Signal which you'll go deep on: "The three hardest components are indexing, apply, and inference serving. Let me go deep on those."
Deep Dive 1: Codebase Indexing & Retrieval ¶
Frame the problem: a 1M-line repo is ~10–15M tokens; it doesn't fit, costs too much to re-send, and "lost in the middle" means stuffing it actively hurts. So: "out of millions of lines, which 50–500 does the model need right now?"
Pipeline: walk files (honor .gitignore/ignore files — a privacy control too) → chunk on AST/Tree-sitter boundaries (functions/classes, not fixed windows, so chunks are coherent) → embed with a code-tuned model → store in an HNSW ANN index. Sync incrementally with a Merkle tree so a one-file edit re-embeds only that file (the Git/Dropbox trick). At query time: hybrid retrieval — dense embeddings (semantic) plus sparse BM25/ripgrep (exact identifiers, error strings) — fused with Reciprocal Rank Fusion, then a cross-encoder reranker narrows top-100 → top-10. "Retrieve then rerank" resolves the accuracy/throughput tradeoff: cheap ANN to get candidates, expensive cross-encoder only on those.
The privacy architecture is worth volunteering: embed server-side but store only vectors + obfuscated paths, never plaintext source; the client reads actual file contents locally to assemble the final prompt. Offer fully-local embedding for the strictest enterprises.
Then the forward-looking nuance that earns points: the frontier is shifting toward agentic search — give the model grep/read_file tools and let it explore iteratively (always fresh, no stale index, no uploaded-vector privacy headache) — and most mature systems now hybridize: index for a fast first guess, then agentic grep/read to verify.
Deep Dive 2: The Apply Model ¶
This is the component candidates underrate, so going deep here is a differentiator. The core insight: "LLMs reason about code semantically, but files are mutated positionally." The model knows the import goes "at the top"; it doesn't reliably know that's line 14 with 3 spaces of indent. Apply closes that gap.
Walk the edit-representation spectrum and its tradeoffs: full-file rewrite (simple, reliable placement, but token-heavy/slow and risks lazy truncation); unified diff (cheap tokens but brittle — models can't do the hunk-header line arithmetic); search/replace blocks (content-matched, no arithmetic, but requires a unique byte-exact match — the dominant failure mode is whitespace/quote mismatch); and the lazy-edit + fast-apply pattern that defines Cursor's feel. In the last, the big model emits a cheap sketch with // ... existing code ... placeholders, and a small fine-tuned apply model materializes the full file. Economic logic: "pay frontier rates for ~50 tokens of intent, cheap-model rates for the 600-token materialization — don't make your architect measure studs."
The trick that makes it feel instant: speculative decoding / prompt-lookup — because the output is mostly identical to the input file, you speculate the next tokens equal the original and verify in parallel, accepting long runs for free. Cursor reported ~1000 tok/s this way. Wrap everything in a validation/retry harness: unique-match check → whitespace-tolerant fallback → Tree-sitter parse check post-apply → bounded re-prompts with the specific error. The Myers diff rendered as accept/reject hunks is the trust surface — apply can be imperfect because a human (or a check) gates the write.
Deep Dive 3: Inference Serving ¶
Lead with the one mental model that governs everything: a request has two phases with opposite bottlenecks. Prefill processes all prompt tokens in parallel — compute/FLOP-bound, sets time-to-first-token. Decode generates one token at a time — memory-bandwidth-bound (it streams the whole model + growing KV cache out of HBM per token), sets tokens/sec. Different bottlenecks → different optimizations.
The KV cache is the heart: it grows linearly with context, lives in GPU HBM, and is the real limit on batch size, throughput, and cost. PagedAttention (vLLM) pages it like OS virtual memory → near-zero fragmentation + prefix sharing (the foundation of self-hosted prompt caching) → 2–4× throughput. Continuous batching schedules at the token level so finished requests are evicted and new ones admitted every step, keeping GPUs saturated. Speculative decoding (a cheap draft proposes k tokens, the target verifies in one pass) gives 2–3× faster decode with identical output distribution — and prompt-lookup speculation makes apply/FIM near-instant.
Then model routing as the cost story: tiny dedicated models for completions/apply, mid-tier for easy chat, frontier for hard reasoning. Most traffic must be cheap or the economics collapse. Prompt caching (Anthropic ~10% on cache reads, OpenAI ~50% automatic, vLLM/SGLang prefix caching) cuts cost/TTFT 5–10× on prefix-heavy agent loops — so engineer a stable prefix: static content first, volatile last, never inject a timestamp early.
Minute 38–43: Bottlenecks & How to Scale Each ¶
Be systematic — name the bottleneck, then the fix:
- Tab latency tail (p95). Edge PoPs near users, kept-alive connections (skip TLS handshake), small quantized models, KV/prefix-cache reuse, speculative decoding, hard-timeout-and-abandon past ~400ms rather than show stale text. Cancellation is a first-class cost lever.
- KV-cache exhaustion under load → preemption thrash, TTFT spikes. Fix: PagedAttention + admission control + context-length caps + capacity headroom.
- Cost blowup on agent runs (quadratic token growth). Fix: prompt caching of the stable prefix, history compaction/summarization past ~75% fill, tool-result truncation (head+tail), sub-agents for context isolation, cheaper models for grunt subtasks.
- Indexing surge starving inference. Fix: separate the services — inference is GPU-bound/spiky/latency-critical, indexing is CPU-IO-bound/bursty/throughput-oriented. Scale each on its own signal (GPU utilization vs. queue depth).
- Stale index after edits. Merkle-tree incremental re-embedding + always layer live ripgrep over the working tree so exact matches are never stale.
- Provider 429s / outages. Multi-provider and multi-region failover, retries with jittered backoff, load shedding (drop background prefetch before user chat), graceful degradation to a smaller model.
- Stateful agent runs vs. fungible workers. Keep compute stateless; persist checkpoints to durable storage so a run survives a worker restart and is resumable.
Minute 43–45: MVP → v2 → v3 Evolution ¶
Closing with a phased roadmap shows product judgment, not just systems knowledge.
- MVP: Fork VS Code. Inline completion via a hosted FIM model. Chat with simple retrieval (open files + ripgrep, skip the vector index initially). Edits via search/replace blocks with a unique-match contract and Tree-sitter validation — cheapest reliable apply, no second model. Single frontier provider. Local execution with approval gates. Ship it; instrument acceptance rate and apply-success rate.
- v2: Add the server-side vector index (AST chunking, HNSW, Merkle sync, hybrid + rerank). Add model routing and prompt caching for cost. Build the agent loop (ReAct, tool use, budgets, streaming + interruptibility, checkpoint/undo). Add the fast-apply model with speculative decoding once telemetry shows large-file apply latency is the bottleneck. Add MCP for extensibility.
- v3: Train custom models — a next-edit-prediction Tab model, distilled from a frontier teacher; fine-tune apply on git-history + accepted-edit pairs; RL-from-execution-feedback for agent skills. Cloud background agents in Firecracker microVMs (open PRs asynchronously). Enterprise tier: zero-retention, permission-aware retrieval, self-hosted/VPC, on-prem/local model options. Close the data flywheel (accept/reject telemetry → DPO) from consenting tiers only.
State the evolution principle: start with the thinnest reliable harness and scaffold surgically where telemetry shows pain — the "bitter lesson" says general methods that ride model improvement beat hand-engineered structure over time.
What a Strong vs. Weak Answer Sounds Like ¶
Weak answer:
- Jumps straight to boxes without clarifying scope or stating requirements.
- Treats all three surfaces as one "call the LLM" flow — doesn't grasp that Tab, chat, and agent are different systems with different latency budgets.
- Says "just put the whole repo in the 1M-token context window" — ignores cost, latency, and lost-in-the-middle.
- Hand-waves apply as "the model writes the file" — misses that positional mutation is one of the two or three hardest problems.
- No numbers. Can't say what latency budget Tab needs or why output tokens cost more.
- Treats the model as the product; never mentions that the harness, tools, retrieval, and apply are where products actually differ (same model, different harness = 15+ SWE-bench points).
- Forgets safety entirely, or bolts it on as "we'll add auth."
Strong answer:
- Spends five minutes scoping and states assumptions back.
- Anchors the whole design on latency tiers and explains why each surface forces different choices.
- Is fluent with the two-phase prefill/decode model, KV cache, PagedAttention, continuous batching, speculative decoding, prompt caching — and ties each to a concrete cost or latency win.
- Goes genuinely deep on indexing (hybrid retrieve→rerank, Merkle sync, the vectors-not-source privacy trick) and apply (the edit-representation spectrum, lazy-edit + fast-apply, speculative decoding, validation harness).
- Discusses tradeoffs out loud as tradeoffs: stuffing vs. just-in-time retrieval; thin harness vs. heavy scaffolding; fork vs. extension; local vs. cloud inference; build vs. buy models. Picks a side and justifies it, but acknowledges the other.
- Knows the metric that matters per surface — acceptance rate + retention for Tab, SWE-bench Verified for agents, cost-per-resolved-task for economics — and that benchmarks under-measure real feel.
- Treats safety as architecture, not wording: the lethal trifecta (private data + untrusted content + exfiltration), egress restriction as the highest-leverage control, sandboxing for cloud agents, prompt injection as unsolved at the model level so you bound the blast radius.
- Ends with a credible MVP→v3 roadmap and a stated principle for when to add complexity.
The meta-signal an interviewer is grading: do you understand that an agentic editor is a systems and product problem disguised as an AI problem? The model is rented or small-and-distilled; the moat and the difficulty live in the harness, the retrieval, the apply pipeline, the latency engineering, the safety boundary, and the data flywheel. Say that, and back it with the depth above, and you've given a senior answer.