4. Autocomplete, Tab & Fill-in-the-Middle
What This Subsystem Is and Why It Exists ¶
Inline completion is the gray "ghost text" that appears ahead of your cursor as you type, which you accept by pressing Tab. It is the single most-used AI feature in any editor: it fires on nearly every keystroke pause, must answer in well under a human's perception threshold, and is judged by one brutal question every few seconds — did the developer hit Tab or not?
This subsystem exists because chat is too slow and too heavy for the inner loop of coding. When you are mid-line writing for (let i = 0; i <, you do not want to open a chat panel, type a prompt, and wait three seconds. You want the editor to read your intent from context and finish the thought in 150 milliseconds. Autocomplete is the reflex layer of the agentic editor: no conversation, no tool calls, just continuous low-latency prediction woven into typing.
The defining technical fact is this: autocomplete is not chat. A chat model predicts text that comes after a prompt. But code is rarely written strictly left-to-right at the end of a file. You add an argument in the middle of a function signature, insert a line between two existing lines, fill a hole inside a JSX block. The model must condition on both the text before the cursor (the prefix) and the text after it (the suffix). That capability is called Fill-in-the-Middle (FIM), and it is the foundational training trick that makes editor autocomplete work.
Think of chat completion as finishing a sentence and FIM as solving a fill-in-the-blank quiz where you can see the words on both sides of the blank.
Fill-in-the-Middle: The Training Objective and Prompt Format ¶
The core idea ¶
A normal autoregressive language model is trained left-to-right: predict token n+1 given tokens 1…n. FIM, introduced in OpenAI's 2022 paper "Efficient Training of Language Models to Fill in the Middle" (Bavarian et al.), showed you can teach the same model to fill gaps without hurting its left-to-right ability — they called this the "FIM-for-free" property.
The trick is a data transformation applied at training time. Take a document, split it randomly into three parts — prefix, middle, suffix — then reorder them and let the model learn to generate the middle.
Two layouts: PSM and SPM ¶
Documents are rearranged into one of two orderings using special sentinel tokens:
- PSM (Prefix-Suffix-Middle):
<PRE> prefix <SUF> suffix <MID> middleThe model reads prefix, then suffix, then generates the middle until it emits an end-of-middle / EOT token. - SPM (Suffix-Prefix-Middle): suffix first, then prefix, then middle. SPM can give better key/value cache reuse because the prefix (which changes most as you type) sits closest to the generation point.
Real sentinel tokens from shipping models make this concrete:
- CodeLlama / Llama:
<PRE>,<SUF>,<MID>, plus<EOT>. - StarCoder / StarCoder2:
<fim_prefix>,<fim_suffix>,<fim_middle>. - DeepSeek-Coder:
<|fim▁begin|>,<|fim▁hole|>,<|fim▁end|>. - CodeGemma:
<|fim_prefix|>,<|fim_suffix|>,<|fim_middle|>.
So at inference time, the editor takes everything left of your cursor as the prefix, everything right as the suffix, formats them with the model's exact sentinel tokens, and asks the model to generate the middle. The cursor is literally the boundary between <fim_prefix> and <fim_suffix>.
Why FIM differs from chat completion ¶
| Dimension | Chat completion | FIM autocomplete |
|---|---|---|
| Conditioning | Prefix only (left context) | Prefix and suffix |
| Prompt shape | Role messages, system prompt | Raw code + sentinel tokens |
| Trigger | User hits Enter | Every keystroke / debounce |
| Output | Long, conversational | Short code span, no prose |
| Stop condition | EOS / turn end | EOT sentinel, often a single line |
| Latency budget | Seconds OK | ~100–300 ms |
Mixing these up is a classic interview trap. If you feed a FIM hole to a chat model it will happily explain what it would write instead of writing it. The model must be trained with the FIM objective (typically 50–90% FIM rate during pretraining) to handle the sentinels natively.
How the Copilot-Style Pipeline Actually Works ¶
GitHub Copilot pioneered the production pattern that nearly everyone copied. The flow, on each completion opportunity:
1. Trigger and debounce ¶
A keystroke, cursor move, or pause fires the engine. The editor debounces — waits a short interval (commonly tens of milliseconds, e.g. ~30–75 ms of typing quiet) — so it does not fire a request on every single character of fast typing. If you keep typing, the pending request is cancelled.
2. Prompt assembly (the secret sauce) ¶
This is where most of the quality lives — not in the model, in the context selection. Copilot's prompt builder assembles:
- The current file, split at the cursor into prefix and suffix.
- Neighboring tabs: snippets from other files you have open in the editor. Copilot's "neighboring tabs" technique scans open tabs for the most relevant fragments.
- Recently viewed / edited files and recently accessed symbols.
- Language and path metadata, sometimes a comment header like
// Path: src/util/math.ts. - Sometimes symbol context (imports, type definitions) pulled from the language server.
Candidate snippets are scored for relevance (a cheap similarity measure such as Jaccard overlap of token sets between the snippet and the code around the cursor) and packed into the context window until the token budget is full — typically a few thousand tokens, since a bigger prompt means higher latency. This is a knapsack problem solved greedily: highest-relevance snippets first.
Analogy: the prompt builder is a travel packer. The suitcase (context window) is fixed-size; it stuffs in the most useful items (relevant snippets) by value-per-space and leaves the rest behind.
3. Single-line vs multi-line decision ¶
The engine decides how much to ask for. If you are mid-line, it usually wants one line or just the rest of the line. If you just typed function foo() { and hit Enter, it may request a multi-line block. This decision uses cursor position, the syntactic context (open brace, empty body), and indentation. Multi-line requests cost more tokens and more latency, so they are issued selectively.
4. Generation and candidate ranking ¶
The model generates one or more candidates (sampling with low temperature, sometimes a small beam or n>1). Candidates are ranked by the model's mean log-probability (average per-token confidence). Copilot historically surfaced the top candidate as ghost text and let you cycle alternates with Alt+] / Alt+[.
5. The contextual filter model — whether to show anything ¶
This is the most underappreciated piece. Copilot runs a separate, lightweight classifier (commonly described as a logistic-regression-style "contextual filter" over features like the language, prior acceptance rate, the leading characters, and the candidate's confidence) that decides whether to display the suggestion at all. A low-confidence completion is suppressed rather than shown, because a bad suggestion is worse than none — it costs the user attention and erodes trust. The north-star is acceptance rate, and showing junk tanks it.
6. Post-processing ¶
Before display: de-duplicate against text already present in the suffix (avoid suggesting a closing brace that is already there), trim to a sensible stop (end of line/block), fix indentation, and drop empty or whitespace-only completions.
7. Render and resolve ¶
Show ghost text. On Tab, insert and log an acceptance event. On continued typing or Esc, log a rejection. Telemetry flows back to training and tuning.
Cursor Tab: Next-Edit Prediction, Not Just Caret Completion ¶
Cursor reframed the problem. Classic autocomplete answers: "what text goes at my cursor?" Cursor's Tab model answers a bigger question: "what is the developer's next edit, anywhere nearby?"
Key differences:
- It predicts edits, not just insertions. It can suggest modifying existing code — changing a variable name on the next line, updating a call site after you changed a signature, deleting a now-dead argument. It is diff-aware.
- It predicts a cursor jump. After you accept one edit, Tab can propose where to go next — you press Tab again and it leaps you to the next location that needs a change, then offers the edit there. This turns Tab into a "ride the edits" experience across a refactor.
- It conditions on your recent edit history, not just static file state. If you just renamed
userIdtoaccountIdin three places, the model infers you'll want the fourth and offers it proactively.
Mechanically this is commonly implemented as a model trained on (recent-edits, current-state) → (next-diff, next-cursor-position) tuples, served by a fast custom model with aggressive speculative prefetching. Windsurf shipped a comparable feature ("Supercomplete" / Tab-to-jump). The mental model: classic autocomplete is autopilot holding the lane; next-edit prediction is the GPS routing you through the whole multi-step maneuver.
The Latency Budget ¶
Autocomplete lives and dies on latency. Targets are aggressive:
- Debounce: ~25–75 ms of typing-idle before firing.
- End-to-end suggestion target: sub-300 ms is the goal; the very best feel is under ~150 ms. Beyond ~500 ms the suggestion arrives after you have already typed past it and feels broken.
- Time-to-first-token matters more than total time because ghost text can render the first line as it streams.
Techniques to hit the budget:
- Request cancellation: every new keystroke aborts the in-flight request (HTTP cancel / abort signal). Without this you waste GPU and risk showing stale ghost text.
- Speculative / predictive requests: fire a completion before the user finishes the pause, or pre-warm at likely trigger points, then discard if context changed.
- Caching: cache by a hash of the (prefix, suffix, context) prompt so re-typing the same character or undo/redo returns instantly. Prefix-cache reuse on the server (KV cache) is the bigger win — because each keystroke only appends one token, the model can reuse most of the previous forward pass. SPM ordering exists partly to maximize this reuse.
- A small, fast, specialized model. This is the central serving decision. Autocomplete does not use GPT-4-class frontier models. It uses small code models (~1B–7B parameters) so a forward pass is milliseconds, often quantized and served on tight, dedicated low-latency infra. Quality per token is lower than a frontier model, but the FIM specialization plus context engineering plus speed wins the UX.
- Speculative decoding on the server (a tiny draft model proposes tokens, the bigger model verifies in parallel) to raise tokens/sec.
The tradeoff triangle: latency vs. context size vs. model quality. Bigger prompt and bigger model = better suggestions but slower. The product tuning is finding the smallest model and prompt that still earns Tab presses.
Acceptance Rate: The North-Star Metric and the Telemetry Loop ¶
The defining KPI is acceptance rate: of suggestions shown, what fraction did the user accept (press Tab)? Copilot has publicly cited acceptance rates around the high-20s percent in some languages, and characters-retained metrics on top of that.
But raw acceptance is gameable and shallow, so teams layer richer signals:
- Characters / lines retained after N seconds — did the accepted code survive, or did they immediately delete it? This catches "accepted then regretted" completions.
- Per-language, per-context breakdowns — Python in a test file behaves differently from TSX in a component.
- Suppression effectiveness — did the contextual filter correctly hide low-value suggestions?
The loop closes like this: the editor emits anonymized telemetry events (shown, accepted, rejected, retained, with features but ideally not raw proprietary code unless the user opts in). This data tunes the contextual filter threshold, informs prompt-assembly weights, and feeds offline evaluation and fine-tuning of the model. Acceptance rate is both the scoreboard and the training signal — a classic data flywheel.
Concrete Technologies and Numbers ¶
- Models built FIM-native: StarCoder/StarCoder2 (BigCode), CodeLlama (Meta, 7B/13B/34B with FIM), DeepSeek-Coder (1.3B/6.7B), CodeGemma, Codestral (Mistral, FIM endpoint), Qwen2.5-Coder. Local-first tools favor 1B–3B for speed.
- FIM training rate: ~50–90% of documents transformed during pretraining.
- Context budget: a few thousand tokens of assembled prompt is typical for the inner loop, even when the model supports much larger windows — because latency, not capacity, is the constraint.
- Stop tokens: model-specific EOT/
<fim_middle>-terminating sentinel, plus client-side single-line/single-block truncation. - Similarity scoring: cheap Jaccard / token-overlap for snippet ranking; no embeddings in the hot path (too slow).
How Leading Products Differ ¶
- GitHub Copilot: The reference pipeline — neighboring tabs, contextual filter, OpenAI Codex-lineage models. Server-side, cloud-only. Defined the genre.
- Cursor: Next-edit prediction (Tab) with cursor-jump, edit-history conditioning, custom fast models, heavy speculative prefetch. The most ambitious reimagining of inline completion.
- VS Code (built-in / Copilot host): Provides the
InlineCompletionItemProviderAPI and ghost-text rendering surface that most extensions plug into. The editor owns debounce, rendering, and accept/reject UX; the provider supplies completions. - Zed: Built its own Zeta open model for edit prediction, emphasizing speed in a Rust-native editor; also integrates Copilot/Supermaven. Acquired Supermaven, known for extreme low-latency completion via a large-context proprietary model.
- JetBrains: Ships full-line completion running a local, on-device small model (per language) so it works offline with strong privacy, alongside cloud AI Assistant. The local model is the differentiator.
- Sourcegraph Cody: Leans on its code-graph and embeddings retrieval for context; autocomplete uses fast models (e.g. StarCoder/Claude-class) with retrieval-augmented context from the whole repo, plus aggressive caching and a context-quality focus.
- Continue (open source): Pluggable — you choose the FIM model (Codestral, StarCoder, DeepSeek, Qwen) and can run fully local via Ollama. Implements its own context gathering and debounce; transparent and configurable.
- Aider: Primarily a chat/edit CLI, not an inline ghost-text tool — it operates at the request/diff level, not the keystroke level, so it largely sidesteps the FIM autocomplete problem.
- Claude Code: A terminal-based agent. Like Aider, it works at the instruction-and-edit granularity, not per-keystroke ghost text; there is no Tab-completion inner loop.
- Windsurf: Cascade for agentic edits plus "Supercomplete," a next-edit/jump feature comparable to Cursor Tab.
The big split: inline ghost-text products (Copilot, Cursor, Zed, JetBrains, Cody, Continue, Windsurf) invest heavily in FIM models and the latency pipeline; agentic CLIs (Aider, Claude Code) skip it entirely because their interaction granularity is a whole task, not a keystroke.
Failure Modes and Mitigations ¶
- Suggesting code that already exists (duplicate braces / repeated lines). Cause: ignoring the suffix. Mitigation: true FIM conditioning on the suffix + post-process de-dup against text after the cursor.
- Stale ghost text. A slow response arrives after context changed. Mitigation: request cancellation, attach a context-version token, discard responses whose context hash no longer matches.
- Latency spikes killing the feel. Mitigation: small model, prompt caching, KV/prefix reuse, speculative decoding, hard timeouts that abandon the request.
- Hallucinated APIs / nonexistent functions. Small fast models hallucinate more. Mitigation: inject real symbol/import/type context from the LSP into the prompt; rank by confidence; let the contextual filter suppress low-confidence output.
- Over-suggesting (noise). Constant low-quality popups train users to ignore Tab. Mitigation: the contextual filter model; conservative thresholds; suppress in low-value contexts (e.g. inside comments or strings unless asked).
- Wrong indentation / broken syntax on insert. Mitigation: reindent to the cursor's context, balance brackets against the suffix, validate the edit.
- Privacy leakage. Prompts contain proprietary source; telemetry could capture it. Mitigations: zero-retention / opt-out modes, on-device models (JetBrains full-line), enterprise no-train guarantees, code-referencing filters that block verbatim regurgitation of public-repo snippets (Copilot's duplication detection filter).
- Cost at scale. Millions of requests per minute, most never accepted. Mitigations: cheap models, caching, debounce + filter to cut request volume, batching on the server.
- Cursor-jump disorientation (next-edit prediction). Jumping the user somewhere unexpected is jarring. Mitigation: only jump on high confidence, show a clear visual cue, keep jumps local and reversible.
Interview Talking Points ¶
- Autocomplete is the reflex layer: per-keystroke, sub-300 ms, judged purely by acceptance rate.
- FIM (Fill-in-the-Middle) is the key training objective — condition on prefix AND suffix via sentinel tokens (
<fim_prefix>/<fim_suffix>/<fim_middle>), unlike left-to-right chat completion. "FIM-for-free" (OpenAI 2022). - PSM vs SPM ordering; SPM helps KV-cache reuse since the cursor prefix changes most.
- Copilot pipeline: debounce → prompt assembly (current file split + neighboring tabs + recent files, packed by relevance into a few-thousand-token budget) → single vs multi-line decision → generate → rank by mean log-prob → a separate contextual-filter model decides whether to show anything → post-process → render → Tab.
- Cursor Tab = next-edit prediction + cursor jump, conditioned on recent edit history; predicts diffs anywhere nearby, not just text at the caret.
- Latency tricks: request cancellation, speculative prefetch, prompt/KV caching, a small fast specialized FIM model (1B–7B), speculative decoding.
- The metric and the data flywheel are the same thing: acceptance rate (plus characters-retained) tunes the filter, prompt weights, and fine-tuning.
- Tradeoff triangle: latency vs context size vs model quality.
- Agentic CLIs (Aider, Claude Code) skip inline FIM autocomplete because their granularity is a task, not a keystroke.
Likely Interview Q&A ¶
Q1. Why can't you just use a chat model for inline autocomplete? A chat model is trained to continue text left-to-right after a prompt. Code editing is bidirectional — you insert in the middle of functions, between lines, inside brackets — so the model must condition on the suffix as well as the prefix. That requires Fill-in-the-Middle training with sentinel tokens; a chat model fed a FIM hole produces prose or ignores the right-hand context, and duplicates code already present after the cursor. Also, chat models are too large/slow for a 150–300 ms per-keystroke budget. Autocomplete uses small, FIM-native models.
Q2. Explain the FIM training objective and prompt format. Take a document, split it into prefix, middle, suffix. Reorder into PSM (<PRE>prefix<SUF>suffix<MID>middle) or SPM and train the model to generate the middle ending in an EOT sentinel. Applied to ~50–90% of training docs, it teaches infilling without hurting left-to-right ability ("FIM-for-free"). At inference, the cursor is the prefix/suffix boundary; the editor formats both sides with the model's exact sentinel tokens and asks for the middle. SPM (suffix first) maximizes KV-cache reuse because the prefix changes most as you type.
Q3. What does the contextual filter model do and why is it separate? It's a lightweight classifier that decides whether to show a candidate at all, using features like language, the candidate's mean log-prob, leading characters, and the user's recent acceptance rate. It's separate from the generation model because suppression is a cheap binary decision that protects the north-star metric: a wrong suggestion is worse than none — it costs attention and erodes trust, dragging acceptance rate down. Keeping it separate lets you tune the show/hide threshold from telemetry without retraining the generator.
Q4. How is Cursor Tab different from classic Copilot ghost text? Classic ghost text answers "what text goes at the cursor?" Cursor Tab predicts the developer's next edit anywhere nearby — including modifying or deleting existing code — and predicts a cursor jump to the next location that needs changing, so repeated Tab presses ride you through a whole refactor. It conditions on recent edit history, not just static file state, so after you rename a symbol three times it offers the fourth proactively. It's diff prediction, not caret insertion.
Q5. Walk me through the latency budget and how you'd hit sub-200 ms. Debounce ~25–75 ms to avoid firing on every keystroke; cancel in-flight requests on new input; cache by prompt hash for undo/retype; reuse server-side KV/prefix cache since each keystroke appends one token; use a small (1B–7B) quantized FIM model on dedicated low-latency infra; stream first-token-first so the first line renders immediately; optionally speculative-decode and speculatively prefetch before the pause completes. The governing tradeoff is latency vs context size vs model quality — pick the smallest model and prompt that still earns Tab presses.
Q6. How do you measure success, and how does telemetry close the loop? North-star is acceptance rate — fraction of shown suggestions accepted via Tab — supplemented by characters/lines retained after N seconds (catches accepted-then-deleted) and per-language/per-context breakdowns. The editor emits shown/accepted/rejected/retained events with features (ideally not raw proprietary code unless opted in). That data tunes the contextual-filter threshold, reweights prompt assembly, and feeds offline eval and fine-tuning. The metric is simultaneously the scoreboard and the training signal — a data flywheel.
Q7 (system design). Design the autocomplete backend for 10M developers, optimizing for p50 latency and cost. Client: debounce, request cancellation, prompt-hash cache, context-version tokens to drop stale responses. Edge: a thin gateway that does prompt assembly server-side (or accepts a pre-assembled prompt), with a shared cache (Redis) keyed on prompt hash for instant hits. Serving: a fleet of small FIM models (e.g. 3B, quantized) on GPUs with continuous batching, prefix/KV-cache reuse keyed by session so each keystroke reuses the prior forward pass, and speculative decoding to raise tokens/sec. Use SPM ordering to maximize cache reuse. Route by language to specialized model shards. Cut cost by: aggressive debounce + contextual-filter suppression to drop request volume (most requests never accepted), caching, and never invoking frontier models in this path. Track p50/p95 time-to-first-token and acceptance rate per shard; autoscale on queue depth; hard-timeout and abandon requests past ~400 ms rather than show stale text. Privacy: zero-retention mode, regional routing, and an opt-out from telemetry training.
Q8 (tradeoff). When would you run the completion model on-device vs in the cloud? On-device (JetBrains full-line, Zed Zeta, Continue+Ollama) wins on privacy (code never leaves the machine), offline use, zero network latency, and zero per-request server cost — but you're limited to tiny models (≤~3B), so raw quality and context size suffer, and you burn the user's CPU/GPU/battery. Cloud (Copilot, Cursor) lets you serve bigger/better models with large shared caches and central tuning, at the cost of network round-trips, server spend, and a privacy/compliance surface. The decision hinges on the customer: enterprises with strict data rules and offline needs favor on-device; consumer/cloud-native users favor the higher-quality cloud path. A common hybrid is a fast local model for the common case with cloud fallback for harder multi-line completions.
Q9. What is "neighboring tabs" and why does it matter more than the model sometimes? Neighboring tabs is Copilot's technique of pulling relevant snippets from your other open editor tabs (and recently viewed files) into the prompt, scored by cheap token overlap and packed into the token budget. It matters because most completion quality comes from context selection, not model size — the model can only complete processOrder( correctly if the function's definition or call sites are in the prompt. Better context routinely beats a bigger model in acceptance rate, which is why prompt assembly is the real engineering battleground.