Agentic Editor · Study Guide

16. End-to-End System Design & Backend

What This Subsystem IS and Why It Exists

This is the glue layer — the distributed backend that sits between the editor on your laptop and the GPUs running large language models in a data center. Every other subsystem in this guide (Tab completion, chat, indexing, fast-apply, agents, tools) is a feature. This section is the plumbing that lets all of them coexist, scale, stay cheap enough to run, and not fall over.

The naive mental model is "editor calls OpenAI/Anthropic directly." Almost no serious product does this. Instead there is a server-side proxy owned by the vendor (Cursor's backend, GitHub's Copilot proxy, Sourcegraph's gateway) that brokers every request.

The core architecture is a hub-and-spoke:

                         ┌─────────────────┐
                         │  Model Providers │ (OpenAI, Anthropic,
                         │  + vendor models │  Google, Fireworks, vLLM)
                         └────────▲────────┘
                                  │
  ┌──────────┐         ┌──────────┴──────────┐         ┌───────────────┐
  │  Client  │◄───────►│   Backend Proxy /   │◄───────►│ Indexing Svc  │
  │ (editor/ │  HTTPS  │   Orchestrator      │         │ (embeddings,  │
  │  ext)    │  gRPC   │                     │         │  vector DB)   │
  └──────────┘         └──────┬───────┬──────┘         └───────────────┘
                              │       │
                   ┌──────────▼─┐   ┌─▼────────────────┐
                   │ Apply Svc  │   │ Tool/Exec Svc    │
                   │ (fast-apply│   │ (terminal, file  │
                   │  model)    │   │  ops, sandbox)   │
                   └────────────┘   └──────────────────┘

Why a Proxy Exists Even in "Privacy Mode"

This is the question interviewers love. Even when a product promises "we don't store your code," requests still flow through the vendor's server. Reasons:

  • API key management. The vendor's contract keys to OpenAI/Anthropic are worth real money and must never ship to the client. The proxy holds them; the client authenticates with a session token instead.
  • Model routing. "Auto" mode picks a model per request (cheap model for simple edits, frontier model for hard reasoning). That decision logic lives server-side so it can change without shipping a client update.
  • Prompt caching. Anthropic and OpenAI bill cached prompt prefixes at ~10% of normal input cost. Routing all users through one proxy maximizes cache hits across requests and lets the vendor manage cache breakpoints centrally.
  • Rate limiting & abuse prevention. Stops a single user (or a stolen token) from burning the vendor's provider quota; enforces per-plan limits (free vs Pro vs Business).
  • Telemetry & experimentation. A/B tests, latency metrics, acceptance rates — all measured at the proxy.
  • Hosting the fast-apply model. The custom speculative-edit model (see the Apply chapter) is a vendor asset, served from vendor GPUs, not the client.

"Privacy mode" typically means zero-retention: code transits the proxy in memory, is forwarded to providers under zero-retention agreements, and is never written to disk or used for training. It does not mean the code never leaves your machine. (Exceptions: fully local setups like Ollama-backed Continue, or on-prem Sourcegraph.)

Analogy: the proxy is like a hotel concierge. You don't get the restaurant's direct line or pay them yourself — you ask the concierge, who knows which restaurant fits, has the relationships and credit line, batches requests, and remembers your preferences. You stay anonymous to the kitchen.

How It Works: Three Concrete Data Flows

(a) Tab Completion — the latency-critical path

Goal: show a ghost-text suggestion within ~100–300 ms of the keystroke pause, or the user out-types it.

  1. Debounce on client. Editor waits ~20–50 ms after typing stops (avoids one request per keystroke).
  2. Context assembly (client-side). Grab the prefix/suffix around the cursor, recently edited files, and a few "nearby" snippets. This is small and assembled locally to save round trips.
  3. Single request to proxy over a kept-alive HTTP/2 or gRPC connection (connection reuse avoids TLS handshake cost).
  4. Proxy routes to a small, fast autocomplete model — a fine-tuned ~1–7B model served on low-latency infra (often vendor-hosted on Fireworks/own GPUs), not a frontier chat model.
  5. Streaming token output, often with speculative decoding to cut time-to-first-token.
  6. Client renders ghost text. Acceptance (Tab) is logged for telemetry.

Key property: this path is stateless and cache-aggressive. No conversation memory; each completion is self-contained. Time budget is so tight that even a 50 ms cross-region hop matters, so vendors run PoPs/edge regions near users.

(b) Chat — multi-turn with retrieval

Goal: a useful answer that references your codebase, first token in ~1–2 s.

  1. User sends a message + attached context (open files, @-mentioned symbols, selection).
  2. Proxy builds the prompt. It may call the indexing service to retrieve relevant chunks (embedding search over the repo's vector index), rerank them, and stuff the top-k into context.
  3. Model routing picks the chat model (e.g. Claude Sonnet, GPT-class) based on plan and request type.
  4. Prompt caching: the stable prefix (system prompt + retrieved repo context that hasn't changed) is cached; only the new turn is uncached.
  5. Stream tokens back to the client over SSE/streaming HTTP.
  6. Session state: conversation history is held (client-side and/or proxy-side keyed by a conversation ID) so follow-ups don't re-send everything — but the inference itself is stateless; full history is replayed into the model each turn.

(c) Autonomous Agent Task — the long-running loop

Goal: "implement this feature / fix this bug" — minutes of autonomous work, many tool calls.

  1. User issues a high-level goal. Proxy starts an agent run with a unique trace/run ID.
  2. Agentic loop (ReAct-style): model produces a thought + a tool call (read file, grep, edit, run terminal, run tests). Each tool call is dispatched to the tool/execution service.
  3. Execution service runs the action — file ops against the workspace, commands in a sandbox (container/VM), edits applied via the apply service / fast-apply model.
  4. Observation returned (file contents, test output, diff) → fed back into the model → next step. Loop until done or step/time budget hit.
  5. Distributed tracing spans the whole run: each LLM call, each tool call, each apply is a span with latency/token/cost attributes, tied to the run ID.
  6. Checkpointing: state (conversation, files touched, todo list) is persisted so a run can survive a worker restart or be resumed.

The agent loop is the opposite of Tab completion: latency-tolerant (users expect minutes), but expensive (dozens of frontier-model calls), stateful, and needs strong isolation because it executes code.

Engineering Challenges and Tradeoffs

Stateless Requests vs Session State

  • Inference is stateless by nature: the model has no memory between calls; you replay context each time. This is what lets you load-balance any request to any inference worker — huge for scaling.
  • Sessions need state: conversation history, agent run progress, the repo index. The trick is to keep state in external stores (Redis, Postgres, object storage, vector DB) and keep the compute workers stateless. Workers become fungible; you can autoscale and recycle them freely.
  • Agent runs are the awkward middle: long-lived and stateful. Solution: persist checkpoints to a durable store and treat the worker as resumable, not as the source of truth.

Scaling Indexing and Inference Independently

These have completely different resource profiles, so they're separate services:

  • Inference: GPU-bound, spiky, latency-sensitive. Scale on GPU utilization / queue depth. Uses continuous batching (vLLM/TGI) to pack multiple requests per GPU.
  • Indexing: CPU/IO-bound, throughput-oriented, bursty on repo open. Embedding generation + vector upserts. Scale on a work queue.

Coupling them would mean an indexing surge starves inference of capacity. Separation lets each autoscale on its own signal.

Multi-Tenant Isolation

  • Data isolation: every embedding, conversation, and cache entry is namespaced by tenant (org/user). A query must never retrieve another tenant's chunks. Usually enforced with tenant-scoped indexes/partitions and row-level checks — defense in depth, not just app logic.
  • Execution isolation: agent code execution is the scariest surface. Each run gets an ephemeral sandbox (container, gVisor, or microVM like Firecracker) with no network or scoped network, CPU/mem limits, and teardown after the run. Many products (Cursor included) keep code execution on the client by default precisely to dodge server-side sandbox risk; cloud agents (Copilot Workspace, Cursor background agents, Devin) run in vendor sandboxes.
  • Noisy-neighbor: per-tenant rate limits and quotas so one org can't starve others.

Cost & Reliability Engineering

  • Fallbacks: if the primary provider 429s or errors, fail over to a secondary (different provider or region). Sourcegraph's "Cody Gateway" and Cursor's routing both do provider failover.
  • Timeouts & budgets: every LLM call has a deadline; agent runs have a max-steps / max-token / wall-clock budget to cap cost and prevent infinite loops.
  • Graceful degradation: under load, downgrade to a cheaper/faster model, shrink retrieved context, or disable speculative decoding rather than fail outright.
  • Hedging: for tail latency, fire a duplicate request after a delay and take whichever returns first (used sparingly — it doubles cost).
  • Cost is dominated by tokens. Prompt caching (10x cheaper cached input), routing cheap models for easy work, and trimming context are the three biggest levers.

Rollout & Experimentation

  • Feature flags gate model versions and prompt templates so changes ship without client releases.
  • Server-side prompt/model swaps: because logic lives in the proxy, the vendor can change the autocomplete model overnight.
  • A/B tests measure acceptance rate (Tab), thumbs-up, task completion, and latency. Shadow traffic tests new models against live ones without affecting users.

Concrete Technologies and Numbers That Matter

  • Transport: HTTP/2 + SSE for streaming; some use gRPC/Connect (Sourcegraph) or WebSockets. Keep-alive connections to avoid TLS setup.
  • Inference serving: vLLM, TGI, TensorRT-LLM with continuous batching and PagedAttention; speculative decoding for low TTFT.
  • Vector DB: Turbopuffer (Cursor reportedly uses it), Pinecone, pgvector, Qdrant; HNSW indexes.
  • State: Redis (sessions/rate limits), Postgres (metadata), S3/GCS (checkpoints, logs).
  • Tracing: OpenTelemetry, Datadog, LangSmith/Langfuse for LLM-specific traces.
  • Sandboxing: Firecracker microVMs, gVisor, Docker, or E2B-style sandboxes.
  • Latency budgets: Tab ~100–300 ms; chat first token ~1–2 s; apply for a file ~ <1–2 s via fast-apply at hundreds–thousands of tok/s; agent steps seconds each, runs minutes.
  • Cost anchors: frontier input/output ~$3–15 / M tokens; cached input ~10% of that; a single agent run can be tens of thousands to millions of tokens.

How Leading Products Differ

  • Cursor: heaviest custom backend. Vendor proxy does model routing ("Auto"), hosts a custom fast-apply model and a custom Tab model, uses a remote vector index (Turbopuffer) with embeddings computed from obfuscated/chunked code in privacy mode. Background agents run in vendor cloud sandboxes.
  • GitHub Copilot: mature Copilot proxy in front of OpenAI/Anthropic/Google; strong enterprise controls (content exclusion, proxy support, audit). Copilot Workspace/coding agent runs in GitHub Actions–backed sandboxes.
  • VS Code (Copilot Chat / extensions): the editor is the client; the Language Model API lets extensions route through whatever model backend is configured. Increasingly open-sourced client side; backend is GitHub's.
  • Zed: Rust editor with Zed-hosted collab + AI; emphasizes low-latency local editing with a thin cloud layer for model access (Zed AI / bring-your-own-key).
  • JetBrains: AI Assistant + local full-line completion model running on-device, plus cloud chat. Hybrid: small model local, big model remote.
  • Sourcegraph (Cody): Cody Gateway is an explicit, documented proxy with provider abstraction, rate limiting, and failover; strong on enterprise/on-prem where the whole stack can be self-hosted.
  • Continue: open-source, client-owns-everything — you point it at any provider or a local Ollama model; effectively no mandatory vendor proxy.
  • Aider: CLI tool, runs locally, talks to providers directly with your own key; state is your git repo (commits per change). Minimal backend.
  • Claude Code: terminal agent; talks to Anthropic's API; runs tools/commands locally on your machine. The "backend" is Anthropic's API + local execution; agentic loop and tracing logic live in the client.
  • Windsurf: Cursor-like vendor proxy with "Cascade" agent flows and indexing; vendor-hosted models and routing similar in spirit to Cursor.

The fault line: vendor-proxy products (Cursor, Copilot, Windsurf, Sourcegraph cloud) optimize routing/caching/custom models centrally; local-first tools (Continue, Aider, Claude Code, Ollama setups) trade that for privacy and control.

Failure Modes and Mitigations

  • Provider outage / 429s: multi-provider failover, retries with jittered backoff, queue + load shedding, degrade to cheaper model.
  • Tail latency on Tab: edge PoPs, connection reuse, request cancellation when the user keeps typing, hedged requests.
  • Runaway agent (infinite loop / cost blowout): hard step/token/time budgets, loop detection (repeated identical tool calls), human-in-the-loop confirmation for risky actions.
  • Stale index → wrong context: incremental re-embedding on file save, Merkle-tree/hash diffing to re-index only changed chunks, TTLs.
  • Cache poisoning / wrong cache breakpoints: version cache keys by prompt template and model; invalidate on repo change.
  • Cross-tenant data leak: namespaced indexes, tenant-scoped tokens, query-time tenant assertions, audits.
  • Sandbox escape (code execution): microVM isolation, no/limited egress, ephemeral teardown, least privilege; default to client-side execution where possible.
  • Partial failure mid-agent-run: checkpoint persistence + resumable runs; idempotent tool operations; transactional apply (diff is reviewable/rejectable).
  • Thundering herd on deploy / repo open: queueing, backpressure, autoscaling on queue depth, rate limits.

Interview Talking Points

  • The system is a hub-and-spoke: client ↔︎ backend proxy ↔︎ {model providers, indexing service, apply service, tool/exec service}.
  • A server-side proxy exists even in privacy mode for API-key safety, model routing, prompt caching, rate limiting, abuse prevention, telemetry, and hosting custom models (fast-apply, Tab). Privacy mode = zero-retention, not "code never leaves the machine."
  • Three flows, three profiles: Tab = stateless + ultra-low-latency + small model; chat = retrieval + caching + streaming; agent = stateful loop + tool execution + sandboxing + budgets.
  • Keep compute stateless, push state to external stores (Redis/Postgres/S3/vector DB) so inference workers are fungible and autoscalable.
  • Scale indexing (CPU/IO, queue-driven) and inference (GPU, batch-driven) independently — different resource profiles.
  • Reliability levers: provider failover, timeouts, budgets, graceful degradation, hedging. Cost levers: prompt caching (~10x cheaper), cheap-model routing, context trimming.
  • Distributed tracing per agent run (OpenTelemetry-style spans for every LLM/tool/apply call) is essential for debugging and cost attribution.
  • Multi-tenant isolation on both data (namespaced indexes) and execution (ephemeral sandboxes/microVMs).

Likely Interview Q&A

Q1. Why route through a server-side proxy instead of calling the model provider directly from the editor? To protect vendor API keys (clients get session tokens instead), to centralize model routing and prompt caching, to enforce per-plan rate limits and abuse prevention, to collect telemetry/run experiments, and to host vendor-owned models like the fast-apply and Tab models. Doing it client-side would leak keys, fragment caching, and freeze logic behind client release cycles. Even "privacy mode" keeps the proxy — it just guarantees zero retention rather than keeping code on-device.

Q2. Walk me through the latency path of a Tab completion and where the milliseconds go. Client debounces ~20–50 ms after typing stops, assembles local context (prefix/suffix + recent edits), and sends one request over a kept-alive HTTP/2 connection. The proxy routes to a small fine-tuned model on low-latency infra, which streams tokens using speculative decoding to minimize time-to-first-token. Budget is ~100–300 ms total, so vendors run edge PoPs near users, reuse connections to skip TLS handshakes, and cancel in-flight requests when the user keeps typing. It's stateless — no conversation memory — which is what allows aggressive caching and any-worker routing.

Q3. How do you keep inference workers stateless when chat and agents are inherently stateful? LLM inference is stateless — you replay full context each call, so any worker can serve any request. The statefulness (conversation history, agent run progress, repo index) lives in external stores: Redis for sessions/rate limits, Postgres for metadata, object storage for checkpoints, a vector DB for the index. Workers stay fungible and autoscalable; agent runs are made resumable via persisted checkpoints rather than in-memory worker state.

Q4. Why separate the indexing service from the inference service? They have opposite resource profiles. Inference is GPU-bound, spiky, and latency-critical, scaled on GPU utilization with continuous batching. Indexing is CPU/IO-bound, throughput-oriented, and bursty (a whole repo on open), scaled on queue depth. Coupling them means an indexing surge starves latency-sensitive inference of capacity. Separate services let each autoscale on its own signal and fail independently.

Q5. (System design) Design the backend for an autonomous agent that can take minutes and run code. What are the must-haves? Components: a stateful orchestrator that drives the ReAct loop with a unique run ID; an LLM-call layer with model routing, retries, and provider failover; a tool/execution service; and durable state. Must-haves: (1) budgets — max steps, max tokens, wall-clock — to cap cost and stop infinite loops; (2) sandboxed execution — ephemeral microVMs/containers (Firecracker/gVisor) with scoped network and teardown, since the agent runs arbitrary code; (3) checkpointing to durable storage so a worker crash doesn't lose the run and it can resume; (4) distributed tracing — every LLM call, tool call, and apply is a span tied to the run ID for debugging and cost attribution; (5) idempotent, reviewable tool ops — edits land as diffs that can be rejected; (6) prompt caching of the stable prefix to control cost across dozens of turns. I'd keep the compute worker stateless-with-checkpoints and store the run state externally, and apply per-tenant quotas to prevent one run from starving others.

Q6. (System design / tradeoff) Your provider starts returning 429s and p99 latency spikes during peak. How does the system stay up? Layered defense. First, retries with jittered exponential backoff for transient errors. Second, provider/region failover — route to a secondary model or region. Third, graceful degradation — downgrade to a cheaper/faster model, shrink retrieved context, or disable speculative decoding. Fourth, load shedding and queueing with backpressure so we protect the healthy path rather than collapse globally; non-critical work (background re-indexing) is paused first. Fifth, budgets and timeouts so slow calls don't pile up. For chronic tail latency I'd consider hedged requests but use them sparingly since they double cost. All of this is feature-flagged so we can dial aggressiveness live, and tracing tells us which dependency is the bottleneck.

Q7. What does "privacy mode" actually guarantee, and what does it not? It guarantees zero retention: code transits the proxy in memory, is forwarded to providers under zero-retention/no-train agreements, and isn't persisted or used for training. It does not mean code never leaves your machine — in most cloud products it still flows through the vendor proxy and provider GPUs. For true on-device guarantees you need a local-first setup (Continue + Ollama, on-prem Sourcegraph) or on-device models like JetBrains' local full-line completion. In privacy mode, indexing often sends only embeddings or obfuscated chunks rather than raw source.

Q8. How do prompt caching and model routing affect cost, with rough numbers? Frontier models run roughly $3–15 per million input/output tokens; cached input prefixes bill at ~10% of normal, so caching the system prompt and unchanged repo context cuts cost dramatically across multi-turn chats and long agent runs. Model routing sends easy work (simple completions, small edits) to cheap/small models and reserves frontier models for hard reasoning. Combined with context trimming (retrieve top-k relevant chunks instead of dumping the whole repo), these three are the dominant cost levers — far more impactful than infra optimization, since tokens dominate the bill.

Q9. How do you achieve multi-tenant isolation for both data and code execution? For data: namespace every embedding, conversation, and cache entry by tenant; use tenant-scoped indexes/partitions plus query-time tenant assertions so a retrieval can never surface another org's chunks — defense in depth, not just app logic. For execution: give each agent run an ephemeral sandbox (container or microVM) with CPU/mem limits, scoped or no network egress, and teardown after the run; default to client-side execution where feasible to avoid server sandbox risk entirely. Layer per-tenant rate limits and quotas to prevent noisy-neighbor starvation, and keep audit logs per tenant.

Built from Markdown · 22 pages · Read it aloud and burn it in.