Agentic Editor · Study Guide

15. Model Training & Customization for Code

What This Subsystem Is and Why It Exists

A Cursor-style editor is not "a chat box wired to GPT." Under the hood it is a fleet of models, each tuned for a different job: a big reasoning model for agentic edits, a small blazing-fast model for inline Tab completion, a dedicated "apply" model that merges a diff into a file, and sometimes an embedding model for retrieval and a reranker for context selection. Model training and customization is the discipline of producing and tuning those models so the product is fast, cheap, and good enough to trust.

The core problem: frontier API models alone are not enough.

  • Latency. GPT-class frontier models stream at maybe 50–100 tokens/sec with 300–800 ms time-to-first-token. That is fine for a 20-second agent run but unusable for Tab, where the user expects a suggestion in under ~100–300 ms as they type. You physically cannot hit that with a 400B-parameter model over the network.
  • Cost. Tab fires on nearly every keystroke pause. If each prediction cost a frontier API call (dollars per million tokens), the unit economics collapse. Inline completion has to run on a small, cheap, often self-hosted model.
  • Task shape mismatch. "Apply this edit to this 800-line file" is a narrow, mechanical task. A frontier model can do it, but it's slow, expensive, and occasionally rewrites things it shouldn't. A purpose-built apply model does it faster and more reliably.
  • Behavioral control. You want consistent diff formatting, consistent "predict my next edit" behavior, and domain adaptation to a customer's codebase. Prompting only gets you part way.

So the mental model is: frontier models are the brains for hard reasoning; custom-trained small models are the reflexes for the latency-critical paths. The art is deciding which is which.

How It Works, Step by Step

The model fleet in a typical product

  1. Inline completion / "Tab" model — small (commonly 1B–15B params), self-hosted, trained with the Fill-in-the-Middle (FIM) objective, optimized for sub-300 ms latency.
  2. Apply / merge model — takes the original file plus a terse edit spec ("change function X to do Y") and emits the full rewritten file or a clean diff. Often a fine-tuned mid-size model running on fast inference.
  3. Agent / chat reasoning model — usually a frontier API model (Claude, GPT, Gemini) for multi-step planning and tool use.
  4. Embedding model — turns code chunks into vectors for semantic retrieval.
  5. Reranker (optional) — scores retrieved chunks for relevance before they enter the prompt.

Fill-in-the-Middle: how completion models are trained

A normal language model is trained left-to-right: predict the next token given everything before it. But code completion is rarely "finish at the end of the file." You're usually typing in the middle, with code both above (prefix) and below (suffix) the cursor.

FIM solves this by reformatting training documents. You split a file into three parts — prefix, middle, suffix — and reorder them with sentinel tokens so the model learns to predict the middle given prefix and suffix. The classic "PSM" layout:

<PRE> prefix tokens <SUF> suffix tokens <MID> middle tokens

At inference, you feed <PRE> code-before <SUF> code-after <MID> and let it generate the middle. This is the trick that lets a left-to-right transformer fill a hole.

  • The seminal reference is OpenAI's "Efficient Training of Language Models to Fill in the Middle" (2022), which showed FIM can be learned essentially for free — mix ~50–90% FIM documents into pretraining with no loss in left-to-right ability.
  • Open models bake this in with documented sentinels: StarCoder / StarCoder2 (<fim_prefix>, <fim_suffix>, <fim_middle>), CodeLlama, DeepSeek-Coder, CodeGemma, Qwen2.5-Coder.

Analogy: left-to-right training teaches you to finish someone's sentence. FIM teaches you to fill in a redacted word in the middle of a paragraph using both sides — which is what coding-in-context actually is.

Repository-level FIM

Modern code models go further: they concatenate multiple files from the same repo into one training document, with file-path markers, so the model learns cross-file dependencies (import this, call that). DeepSeek-Coder and StarCoder2 popularized this. It's why a good Tab model can complete a call to a function defined in another file it was given as context.

Training an apply / merge model

When the agent says "rewrite handleAuth to use the new token format," you don't want it to re-emit the whole 800-line file token by token in the main expensive model — that's slow and risks collateral edits. Instead products commonly use a dedicated apply model:

  • Input: the full current file + a compact edit description or a "lazy" diff (with // ... existing code ... placeholders).
  • Output: the full merged file, or a precise unified diff.
  • Trained via supervised fine-tuning (SFT) on pairs of (original file, edit spec) → (final file), mined from real git history and from the product's own accepted edits.

This is the model that makes "fast apply" feel instant. Cursor has publicly discussed running fast-apply at ~1000+ tokens/sec using speculative techniques, treating apply as a near-deterministic transform rather than open-ended generation.

Reinforcement learning from execution feedback

For code, you have a reward signal that text generation generally lacks: you can run the code.

  • Generate a candidate patch → apply it → run the test suite or compiler → reward = tests pass / compiles / lints clean.
  • This is Reinforcement Learning from Execution Feedback (RLEF) / "code RL," and it's how models get genuinely better at producing working code rather than plausible-looking code.
  • Algorithms: PPO historically, increasingly GRPO (Group Relative Policy Optimization, popularized by DeepSeek), which skips the separate value network and normalizes rewards within a group of sampled completions — cheaper and well-suited to verifiable rewards.
  • This is a flavor of RLVR (RL with Verifiable Rewards): the reward is objective (test passed: yes/no), not a learned preference. That makes it far less hackable than fuzzy human ratings.

Analogy: RLHF is like grading an essay on style. RLEF is like grading a physics problem — there's an answer key (the test suite), so the model can't bluff its way to a high score.

RLHF on real edit-acceptance signals

The other reward source is the user. Every time you press Tab to accept a suggestion, dismiss one, or hand-edit the model's output, you generate a label:

  • Accepted → positive signal.
  • Rejected / immediately overwritten → negative signal.
  • Accepted then edited → partial signal; the edit reveals the "correct" answer.

These become preference pairs for RLHF / DPO (Direct Preference Optimization) or simply weighted SFT. DPO is popular because it skips training a separate reward model — you optimize directly on (preferred, rejected) pairs with a simple classification-style loss.

Distillation: shrinking the giant into the sprinter

The latency-critical paths can't run a frontier model, but you still want frontier-quality behavior. Knowledge distillation is the bridge:

  • Run a big "teacher" model to produce high-quality outputs (or full token-probability distributions) on a large set of coding tasks.
  • Train a small "student" model to mimic the teacher — matching its outputs (sequence-level / "hard" distillation) or its probability distribution (logit / "soft" distillation, KL divergence loss).
  • The student is 10–50× smaller, runs in milliseconds, and captures most of the teacher's competence on the narrow task it was distilled for.

This is exactly how you get a Tab model that "feels smart" but answers in 80 ms. You're not asking it to reason about architecture — you're asking it to reproduce the teacher's local-edit instincts.

The Data Flywheel (and Its Privacy Tension)

The strategic moat of these products is the data flywheel:

  1. Users edit code with the product.
  2. Acceptance/rejection/edit telemetry flows back.
  3. That telemetry becomes training data for the next model.
  4. The next model is better → more usage → more data.

More users → more edit data → better Tab → more users. This is why incumbents with millions of users (Copilot) and fast-growing startups (Cursor) have a structural advantage that's hard to replicate from a cold start.

The privacy tension is real and unavoidable:

  • Edit data is source code — frequently proprietary, sometimes containing secrets, credentials, or regulated data.
  • Enterprises demand contractual guarantees: "zero data retention," "we do not train on your code," "code is not persisted."
  • So products bifurcate:
    • Telemetry that can train: opt-in, often only from consumer/free tiers, or fully anonymized/aggregated signals (accept rate by language, latency) rather than raw code.
    • Privacy mode / enterprise: prompts and code are not stored beyond the request and never enter training sets. Cursor's "Privacy Mode," GitHub Copilot's business/enterprise no-retention promises, and zero-data-retention API endpoints (offered by Anthropic/OpenAI for enterprise) all exist precisely to resolve this.
  • The hard part: you want the flywheel, but your best, highest-value users (enterprises) are exactly the ones who forbid it. So real systems lean heavily on execution-feedback RL (synthetic, repo-agnostic, runs in sandboxes you own) and public-code corpora to improve without touching customer code.

Key Engineering Challenges and Tradeoffs

Latency

  • Tab budget is brutal: end-to-end ~100–300 ms including network. That dictates small models, self-hosting near users, aggressive caching, and speculative decoding.
  • Apply needs to feel instant on big files → speculative edits / draft models pushing 1000+ tok/s.

Cost

  • Self-hosting a small model on your own GPUs amortizes far better than per-token API calls for the high-frequency Tab path.
  • Quantization (INT8/INT4, FP8) and batched inference (continuous batching via vLLM/TensorRT-LLM) cut serving cost dramatically.

Quality

  • Verifiable rewards (tests, compilation) raise real-world correctness more reliably than human preference alone.
  • But you must guard against reward hacking: a model that deletes the failing test to "pass," or writes assert True. Mitigation: hidden held-out tests, reward shaping, multiple independent test sources.

Privacy

  • The flywheel vs. enterprise no-train guarantees, as above. Synthetic and public-data training is the pressure valve.

Scale

  • Serving millions of Tab requests/sec at the edge; routing the right request to the right model; keeping a multi-model fleet versioned, monitored, and rollback-able.

Speculative decoding (the latency multiplier worth knowing cold)

A small draft model proposes the next k tokens cheaply; the large target model verifies them all in a single forward pass. Tokens the target agrees with are accepted for free; on the first disagreement you fall back. Result: 2–3× faster generation with mathematically identical output distribution to the target model alone. Variants: Medusa (extra decoding heads instead of a separate draft model), EAGLE, and self-speculation. For apply/merge, the draft is essentially "the original file," because most output tokens equal the input — this is why fast-apply hits four-digit token rates.

Tokenizer and long-context considerations for code

  • Tokenizer matters more for code than prose. Indentation, braces, and identifiers tokenize inefficiently with a generic tokenizer. Code-tuned tokenizers add tokens for runs of whitespace (e.g., a single token for 4/8/16 spaces) and common code patterns, cutting token counts ~10–30% — directly lowering cost and latency.
  • Long context lets you stuff whole files/repos in. Frontier models now offer 128k–1M token windows. But attention cost grows with sequence length, and quality degrades in the middle ("lost in the middle"). So retrieval (RAG) still beats "dump everything," even with huge windows — you want the right 8k tokens, not a noisy 200k.
  • Techniques enabling long context: RoPE scaling / position interpolation, sparse/sliding-window attention, and FIM-at-repo-scale training so the model is trained on long multi-file documents, not just stretched at inference.

When to fine-tune vs. prompt vs. retrieve (the decision matrix)

A question you will get asked. The hierarchy:

  • Prompt / in-context first. Cheapest, instant to iterate, no training infra. Use for behavior you can describe in instructions and a few examples. Default starting point.
  • Retrieve (RAG) for knowledge. When the model lacks facts — this repo's APIs, the user's codebase, current docs — inject them at inference. Don't fine-tune to teach facts; facts go stale and fine-tuning bakes them in badly.
  • Fine-tune for behavior/format/latency. When you need a consistent skill or style (diff format, next-edit prediction), or you need to distill a capability into a small fast model. Fine-tuning teaches the model how to act, not what to know.

Rule of thumb: "Fine-tune for form, retrieve for facts, prompt for everything you can get away with." And only fine-tune once you've exhausted prompting + RAG, because it's the most expensive to build and maintain.

How Leading Products Actually Implement It

Vendor internals are largely proprietary; the following is "commonly understood / publicly stated" — flagged where uncertain.

  • Cursor — The most aggressive on custom models among editors. Publicly describes a custom Tab model trained to predict the user's next edit (not just complete text), and a fast-apply model using speculative edits at very high throughput. Uses frontier APIs (Claude, GPT, Gemini) for agent reasoning. Heavily invests in the edit-data flywheel; offers Privacy Mode for no-retention.
  • GitHub Copilot — Started on OpenAI Codex (a GPT model fine-tuned on public GitHub), pioneered FIM-style completion at scale and the accept-rate flywheel. Now multi-model: routes to OpenAI, Anthropic, and Google models for chat/agent; completion uses fast tuned models. Enterprise tier promises no training on customer code.
  • VS Code — The open platform underneath Copilot; ships the editor and extension API. Microsoft open-sourced the Copilot Chat extension. Model training is upstream (OpenAI/GitHub); VS Code itself is the harness, not a model trainer.
  • Zed — Performance-first Rust editor. Pushed Zeta, an open-source edit-prediction model (fine-tuned from a Qwen2.5-Coder base) with an open dataset — a notable counter-move to proprietary Tab models. Also integrates external frontier models for chat/agent.
  • JetBrains — Ships a local, on-device full-line completion model (a small ~100M-class model running on the user's CPU/GPU, per-language) for privacy and zero-latency, plus cloud AI Assistant / Junie agent using frontier models. A clean example of the "small local reflex + big cloud brain" split.
  • Sourcegraph (Cody) — Retrieval-first. Their bet is that superior context (code search / graph) beats model tuning; they're model-agnostic, routing to multiple frontier providers and using their code-intelligence graph for context. Less about training custom models, more about feeding good ones well.
  • Continue — Open-source, bring-your-own-model. Lets you point at local (Ollama, vLLM) or hosted models and configure separate models per role (autocomplete vs. chat vs. embeddings). Their value is the harness + config, not training.
  • Aider — Open-source CLI agent, model-agnostic, designed around clean diff/edit formats and git integration. Famous for its own code-editing leaderboard measuring how well models apply edits in its diff format — effectively benchmarking the "apply" capability across vendors. No custom model; relies on prompting frontier models well.
  • Claude Code — Anthropic's agentic CLI; runs on Claude models (Anthropic both trains the model and builds the harness, so model and product co-evolve). Emphasis on agentic tool-use and long-context reasoning rather than a separate tiny Tab model. Offers zero-retention enterprise options.
  • Windsurf (formerly Codeium) — Built its own completion models and inference stack from the start (Codeium's heritage was self-hosted fast completion). Ships custom-tuned models plus frontier APIs for its "Cascade" agent; strong enterprise/self-hosted privacy story.

Where they diverge most: (a) Do they train their own small models? — Cursor, Windsurf, JetBrains, Zed yes; Aider, Continue, Sourcegraph no. (b) Retrieval vs. tuning emphasis — Sourcegraph bets on context, Cursor bets on both. (c) Local vs. cloud — JetBrains/Zed ship on-device options; most others are cloud-first.

Failure Modes and Mitigations

  • Reward hacking under execution-feedback RL. Model games the metric (deletes tests, writes trivial asserts, hard-codes outputs). → Hidden held-out tests, diverse reward signals, penalize test edits, manual audits of reward spikes.
  • Distillation capability collapse. The small student is fine on tasks it was distilled for but brittle off-distribution. → Distill on a broad, representative task mix; keep a frontier fallback for hard requests; monitor accept-rate by language/scenario.
  • Catastrophic forgetting in fine-tuning. SFT on narrow edit data erodes general ability. → LoRA/adapter tuning (touch few params), replay general data in the mix, low learning rates, eval on a broad held-out suite before shipping.
  • Stale knowledge baked into weights. Fine-tuning taught the model an API that since changed. → Don't fine-tune for facts; use RAG so knowledge is swappable at inference.
  • Privacy/data leakage. A model trained on customer code regurgitates a secret or proprietary snippet to another user. → No-train guarantees for enterprise, scrub secrets from any training corpus, dedup/memorization tests, train mainly on public + synthetic + execution data.
  • Speculative decoding correctness bugs. A broken implementation changes the output distribution. → Speculative decoding is provably output-preserving when done right (acceptance test guarantees it); verify against non-speculative baseline in CI.
  • Tokenizer mismatch. Serving a model with a tokenizer that doesn't match training corrupts everything. → Pin tokenizer to the model; test FIM sentinels round-trip exactly.
  • Lost-in-the-middle on long context. Crucial code buried mid-prompt gets ignored. → Retrieve and place the most relevant context near the prompt edges; rerank; don't over-stuff.
  • Flywheel feedback bias. Training only on accepted suggestions teaches the model to produce "acceptable-looking" code, not correct code (users accept plausible wrong answers). → Combine acceptance signals with execution verification; track post-accept edit/revert rates as a truer quality signal.

Interview Talking Points

  • A coding editor is a fleet of models: frontier model = brains (agent/chat), small custom models = reflexes (Tab, apply). Latency and cost force this split.
  • FIM (Fill-in-the-Middle) is the training objective for completion: reorder prefix/suffix/middle with sentinel tokens so a left-to-right transformer can fill a hole at the cursor. Costs ~nothing to add to pretraining; repo-level FIM teaches cross-file completion.
  • Dedicated apply/merge model, SFT'd on (file, edit-spec) → final file, makes "fast apply" feel instant — often using speculative edits at 1000+ tok/s.
  • RL from execution feedback (RLEF/RLVR): run the tests, reward passing. Objective and hard to game (vs. fuzzy RLHF). GRPO is the modern algorithm of choice.
  • Edit-acceptance telemetry (accept/reject/overwrite) → DPO/RLHF preference data → the data flywheel. Privacy mode and enterprise no-train guarantees are the counterweight.
  • Distillation shrinks frontier quality into a millisecond-latency student for Tab/apply.
  • Speculative decoding: cheap draft proposes k tokens, big model verifies in one pass → 2–3× faster, identical output.
  • Decision rule: prompt for what you can, retrieve for facts, fine-tune for form/latency — fine-tune last because it's costliest to maintain.
  • Product split: Cursor/Windsurf/JetBrains/Zed train custom small models; Aider/Continue/Sourcegraph stay model-agnostic and bet on harness/context.

Likely Interview Q&A

Q1. Why can't you just use GPT-4-class models for everything, including autocomplete? Latency and cost. Inline completion has a budget of roughly 100–300 ms end-to-end and fires constantly as you type; a 100B+ model over the network is both too slow (hundreds of ms just for time-to-first-token) and far too expensive per call. You self-host a small (1–15B) FIM-trained model for Tab, reserve frontier models for the rare hard reasoning, and often distill frontier behavior into the small model so it still feels smart.

Q2. Explain the FIM training objective and why it's needed. Standard LMs predict left-to-right, but coding happens in the middle of a file with context on both sides. FIM reformats training documents into prefix/suffix/middle, reordered with sentinel tokens (e.g., StarCoder's <fim_prefix> <fim_suffix> <fim_middle>), so the model learns to predict the middle given both surrounding sides. OpenAI's 2022 FIM paper showed you can mix it into pretraining essentially for free without hurting left-to-right ability. Repo-level FIM concatenates multiple files so the model learns cross-file completions.

Q3. How would you make code models produce code that actually works, not just code that looks right? Reinforcement learning from execution feedback. Sample candidate patches, apply them, run the compiler and test suite, and reward outcomes that pass — RL with verifiable rewards. Unlike RLHF on human preference, the reward is objective and hard to fake. Use GRPO or PPO. Critically, guard against reward hacking (model deleting tests or writing trivial asserts) with hidden held-out tests and reward shaping.

Q4. What is the data flywheel, and what's the tension it creates? Every Tab accept/reject/overwrite is a free label. That telemetry trains the next model, which drives more usage, which yields more data — a compounding moat (why Copilot and Cursor are hard to catch). The tension: edit data is proprietary source code, and enterprises contractually forbid training on it. So products bifurcate — opt-in/anonymized telemetry for training, strict no-retention "privacy mode" for enterprise — and lean on synthetic data, public code, and execution-feedback RL to improve without touching customer code.

Q5. When do you fine-tune vs. prompt vs. use RAG? Prompt first — cheapest and instant to iterate; use it for anything describable in instructions plus a few examples. Use RAG when the model lacks facts (this repo's APIs, current docs) — never fine-tune to teach facts, because they go stale and bake in badly. Fine-tune for behavior, format, or latency: consistent diff formatting, next-edit prediction, or distilling a capability into a small fast model. Fine-tuning is last because it's the most expensive to build and maintain. Slogan: fine-tune for form, retrieve for facts, prompt for everything else.

Q6. What is speculative decoding and why does it matter here? A small draft model proposes the next k tokens cheaply; the big target model verifies all k in one forward pass. Accepted tokens are free; on first disagreement you fall back — output distribution is provably identical to the target alone, with 2–3× speedup. For the apply/merge path it's even better: since most output tokens equal the input file, the "draft" is essentially the original file, enabling 1000+ tok/s fast-apply. Variants include Medusa (extra heads) and EAGLE.

Q7 (System design — hard). Design the model architecture for a new Cursor-style editor. What models, where do they run, how do they improve over time? I'd run a four-model fleet. (1) Tab model: 3–7B, FIM + repo-level trained, distilled from a frontier teacher, quantized (FP8/INT8), self-hosted on GPUs near users with continuous batching (vLLM/TensorRT-LLM) and speculative decoding to hit sub-200 ms. (2) Apply model: mid-size SFT model trained on git-history and accepted-edit pairs, served with speculative edits using the original file as draft for 1000+ tok/s. (3) Agent/chat: route to frontier APIs (Claude/GPT/Gemini) — don't train these, the cost isn't worth it. (4) Embedding + reranker for retrieval. Improvement loop: collect accept/reject/overwrite telemetry (opt-in tiers only), generate DPO pairs for the Tab model, and run execution-feedback RL in sandboxes on public + synthetic repos for the agent-adjacent skills. Privacy: enterprise gets a no-retention path; training data comes from consenting tiers, public code, and synthetic execution. Versioning: shadow-deploy new models, compare accept rate and post-accept revert rate, canary then roll out, keep instant rollback.

Q8 (System design / tradeoff — hard). Your enterprise customers forbid training on their code, but you need data to improve. How do you still get a flywheel? Decouple "signals you can learn from" from "raw code you can't keep." First, harvest aggregate, code-free metrics even from no-train accounts — accept rate by language, latency, revert rate — to drive evaluation and routing, not weight updates. Second, build the real training signal from sources you do own: execution-feedback RL on public and synthetic repositories in your own sandboxes (verifiable, infinitely generatable, no customer data), and public-code corpora for FIM/SFT. Third, for the minority of consenting consumer/free users, collect opt-in edit data with secret-scrubbing and memorization tests. Fourth, use frontier teachers to generate distillation data so your small models inherit capability without customer code. The result: the flywheel runs on synthetic + execution + public + consented data, while enterprise code stays request-scoped and never enters training.

Q9. What are the main failure modes of fine-tuning a small code model, and how do you mitigate them? Three big ones. Catastrophic forgetting — narrow SFT erodes general ability; mitigate with LoRA/adapters, low LR, replaying general data, and broad held-out evals before shipping. Stale baked-in knowledge — fine-tuning an API into weights breaks when the API changes; mitigate by keeping facts in RAG, not weights. Distillation collapse — the student is brittle off its training distribution; mitigate with a representative task mix, a frontier fallback for hard cases, and per-scenario accept-rate monitoring. Across all of them: never trust accept-rate alone (users accept plausible-but-wrong code), so pair it with execution verification and post-accept revert tracking.

Built from Markdown · 22 pages · Read it aloud and burn it in.