Gaps, Accuracy Notes & Curveballs
A critic's pass over sections 01–16. Part 1 flags claims that are oversimplified, dated, or shaky. Part 2 covers topics a sharp interviewer probes that the guide treats thinly or not at all, each with a primer so you still learn it. Part 3 is 15 hard questions the guide does not directly prepare you for, each with a model answer.
Part 1 — Accuracy Pass ¶
These are mostly small — the guide is strong — but an interviewer who catches you repeating them will dock points.
- 05-agent-loop: "Thomas Dohmke ... noted 'the agent is a while-loop.'" — The attribution is shaky; the "agent is a while loop" framing is folklore across many practitioners, not a documented Dohmke quote. Correction: present it as a common community framing, not a sourced attribution.
- 08-context / 03-indexing / 15-training: "lost in the middle" is presented as a near-universal law. — It's real but substantially mitigated in 2025+ frontier long-context models (strong needle-in-haystack scores). Correction: frame it as model-dependent and weakening over time, not a constant.
- 14-evaluation: "Frontier models in 2025 report roughly 60–75% on Verified." — Given today is mid-2026, top systems are now ~75–80%+ on SWE-bench Verified. Correction: treat any SWE-bench number as a fast-moving snapshot and say so.
- 09-inference: "frontier models stream at maybe 50–100 tokens/sec." — Dated; many 2026 frontier endpoints exceed 100–200+ tok/s, and reasoning models vary wildly. Correction: quote ranges as illustrative, not current.
- 09-inference / 13: list prices ("$15/MTok in, $75/MTok out" for top tier) are flagged as "early-2025-era" but read as current in places. Correction: always caveat pricing as illustrative; the ratios (output ≫ input, cached ≈ 10%) are the durable point.
- 03-indexing: "A 1M-line codebase is roughly 10–15M tokens." — High end is generous; code averages closer to ~8–10 tokens/line, so ~8–12M is tighter. Minor.
- 03-indexing: "HNSW ... roughly O(log N)." — Empirically near-logarithmic but not a proven bound; it's a heuristic graph search. State it as empirical.
- 01-editor-core: "Xi's original ambition" cited as a live GPU-editor example. — Xi is discontinued; fine as history, but don't imply it's current.
- 01-editor-core: GPU rendering described as "rebuild the frame from scratch every time." — Oversimplified; GPUI and similar still cache glyph atlases and do damage tracking. The contrast with retained-mode DOM is directionally right, the "from scratch" is loose.
- 03-indexing: embeddings are "hard — though not impossible — to invert." — Correctly hedged, but worth knowing the teeth:
vec2text-style attacks reconstruct surprisingly much, so embeddings are sensitive data, not anonymized data.
Part 2 — Coverage Gaps ¶
Topics that are missing or thin across all 16 sections.
1. Reasoning models / test-time compute. The guide barely mentions extended-thinking models (o-series, Claude extended thinking, DeepSeek-R1-style). These reshape everything: a single "call" now hides thousands of internal reasoning tokens you pay for but may not see, TTFT becomes "time to first visible token" after a long think, and the thin-vs-heavy-harness debate tilts further toward thin because the model self-plans. They also change routing (reserve reasoning models for hard agent steps) and eval (you must measure cost-per-success including hidden thinking tokens). Not knowing this is the biggest 2026-era hole.
2. Cross-session / long-term agent memory. Section 08 covers context within a session; nothing covers durable memory across sessions — learned project conventions, past decisions, a persistent CLAUDE.md/memory file, or an external memory store. The hard problems: what to persist (episodic logs vs distilled semantic facts), how to retrieve it without bloating context, how to invalidate stale memory when the codebase changes, and how to stop memory poisoning (a wrong "fact" that compounds). Treat memory as a write-with-care store with provenance and TTLs, not an append-only log.
3. Multimodal / visual input. No coverage of screenshots, design-to-code, diagram understanding, or debugging from an image of a broken UI / stack-trace photo. Frontier coding workflows increasingly paste screenshots ("this layout is wrong, fix it") and Figma frames. This adds a vision encoder to the fleet, far higher per-image token cost, new prompt-injection surface (text hidden in images), and grounding challenges (mapping a pixel region to a DOM node or CSS rule).
4. Voice / ambient coding. Related but distinct: voice-driven coding (dictation + agent) is emerging. It breaks the FIM autocomplete model entirely (no cursor), demands robust intent disambiguation, and stresses streaming/interruptibility. Worth a sentence in any "future interfaces" answer.
5. Caching taxonomy beyond KV/prompt cache. The guide nails prompt/KV caching but ignores the other layers: exact-match response cache (identical completion request → cached answer), semantic cache (embed the request, serve a near-duplicate's answer — risky for code), embedding cache (don't re-embed unchanged chunks), and negative caching (remember "no good completion here" to suppress requests). Each has correctness/staleness tradeoffs.
6. Multi-repo / cross-service reasoning. Section 03/16 handle monorepo scale, but microservice estates — N separate repos, a change in repo A breaks repo B's contract — get no treatment. Needs a cross-repo symbol/dependency graph (SCIP federated across repos), API-contract awareness, and the agent reasoning about a change's blast radius beyond the current repo.
7. Advanced prompt-injection defenses. Section 13 names the threat and the lethal trifecta well but stops at "approve / sandbox / egress." It omits the cutting edge: spotlighting/delimiting untrusted data, the dual-LLM / CaMeL pattern (a privileged planner that never sees untrusted content + a quarantined LLM that does), capability-scoped tool tokens derived from the plan, and taint-tracking so data from an untrusted source can't reach an exfiltration sink.
8. Unit economics & business model. Cost levers are covered; the business isn't. A flat subscription faces a power-law: ~5% of users (heavy agent runs) can cost 20–50× the median, so you need usage tiers, fair queuing, model downgrades on the cheap tier, and BYO-key options. Interviewers love "is this product even profitable?" — know gross-margin math and the abuse/whale problem.
9. Production observability & online guardrails. Sections 09/14/16 touch tracing, but "evals as continuous monitoring" is thin: real-time regression detection on acceptance/retention, automatic canary rollback, anomaly detection on tool-error spikes, per-cohort dashboards, and runtime guardrails (a classifier that blocks an exfiltration-shaped tool call live, not just in review).
10. Long-tail languages & non-code artifacts. What happens with a language that has no good tree-sitter grammar or LSP (a proprietary DSL, COBOL, a config dialect), or artifacts like notebooks, SQL migrations, protobufs, and generated files? The guide assumes the LSP/tree-sitter stack exists. Graceful degradation here (fall back to pure-LLM + tests, synthesize a lightweight grammar, treat as plain text with structure hints) is a real interview probe.
Part 3 — Curveballs (with model answers) ¶
1. A reasoning model with hidden thinking tokens — how does it change caching, cost, and streaming UX? Caching: the visible prompt prefix still caches, but the hidden reasoning is regenerated every call and isn't cacheable, so the cost floor per turn rises — you cache harder on the stable prefix and route reasoning models only to steps that need them. Cost: you must bill and budget on total tokens including invisible thinking; a "cheap-looking" request can be expensive. UX: TTFT becomes "time to first visible token" after a long think, so you show a thinking indicator, stream the summary/answer once it starts, and keep interruptibility so a user can cancel a 30-second think early. Eval shifts to cost-per-success, since these models trade tokens for accuracy.
2. Give an agent persistent memory of a codebase across weeks without re-reading everything — and keep it from going stale or wrong. Separate episodic from semantic memory. Episodic: append run transcripts to durable storage, but never feed them raw — retrieve on demand. Semantic: distill durable facts ("auth lives in src/auth, we use pnpm, tests run via make test") into a compact, human-readable memory file with provenance (which commit/file each fact came from). Keep memory fresh by tying facts to file hashes — when the source file changes, mark the fact suspect and re-verify before trusting it. Guard against poisoning: require facts to be re-grounded against the live repo, cap how much un-verified memory can influence a decision, and let the user inspect/edit memory. Treat it as a cache of conclusions, not ground truth.
3. Two engineers and an agent edit the same file simultaneously. Prevent corruption. Make the buffer a CRDT (or OT) so concurrent character-level edits from any author merge deterministically — this is why Zed built on a CRDT. Tag each edit stream with an author identity (human A, human B, agent) so the agent's edits are visually distinguishable and individually revertible. The agent must edit against a versioned snapshot and submit edits as content-anchored search/replace, not line offsets, so a human's concurrent edit elsewhere doesn't break it; on an overlapping region, three-way-merge and surface a conflict rather than clobber. Apply the agent's multi-edit transaction atomically with one undo boundary. And throttle the agent so it isn't fighting a human's keystrokes in the same span — yield the region the human is actively in.
4. The editor must work on a plane with no network. What still works, degrades, breaks? Works: everything local — buffer, tree-sitter highlighting/folding, LSP (servers are local processes), local search/grep, git. A small on-device completion model (JetBrains-style full-line, or a quantized 1–3B via llama.cpp/MLX) can keep Tab alive. Degrades: retrieval falls back from server embeddings to local lexical (ripgrep) + a local vector index if you shipped one. Breaks: frontier chat/agent calls, server-side fast-apply, cloud indexing. Architecture: design the context/apply pipeline with a local fallback path, cache recent model responses, queue agent tasks to replay on reconnect, and clearly signal degraded mode rather than hanging on a dead socket. The principle is graceful degradation, same as the keystroke-path rule.
5. Design how an agent debugs from a screenshot of a broken UI. Pipe the image through a vision-capable model to extract intent ("button overlaps text, wrong color"). Ground it: correlate the visual region to source by matching rendered text/component names against the codebase (or, if you control the runtime, map the DOM node → source via React DevTools-style component stacks / source maps). Then it's a normal agent loop — read the implicated component/CSS, propose an edit, and verify by re-rendering (headless browser screenshot) and visually diffing against the target. Watch cost (images are token-heavy) and a new injection surface: text hidden in the image can carry instructions, so treat OCR'd content as untrusted data.
6. Support real-time voice-driven coding. What breaks vs text? Add streaming ASR in front of the agent and TTS/▢ visual feedback after. What breaks: FIM autocomplete is meaningless without a cursor, so the inner loop shifts to instruction-level edits; disambiguation is harder ("rename it" — rename what?), so you lean on the editor's current selection/cursor and visible context as grounding. Latency budget changes — speech has natural pauses, so you can be slightly slower than Tab but must feel conversational. Interruptibility is critical (talk over the agent to redirect). Errors are costlier because the user can't easily scan a wall of dictated code, so you summarize changes aloud and rely on diff review. Accessibility is the real win and a strong product angle.
7. Price a subscription when 5% of users cost 50× the median. Walk the economics. Cost is dominated by tokens, and agent runs are a power law: whales fan out dozens of frontier calls per task. A pure flat fee is unprofitable if whales aren't capped. Levers: (1) usage tiers / metered overages above an included quota; (2) model routing — cheap/small models by default, frontier on demand or higher tiers; (3) fair queuing so a whale's long run doesn't starve others; (4) aggressive prompt caching and context trimming to lower the floor; (5) BYO-API-key option to offload inference cost for the heaviest users; (6) abuse detection for resold/automated accounts. Gross margin = subscription − (tokens + GPU + infra); the median request must be cheap, which is exactly why the small-model hot path exists.
8. Rate-limit fairly when one user "request" is actually 80 model calls over 10 minutes. Don't rate-limit at the HTTP request layer — meter the underlying resource. Track per-tenant token and GPU-second budgets with token-bucket refill, not request counts. Use weighted fair queuing so a long agent run is preempted/time-sliced against short interactive requests (chat/Tab get priority lanes; background agents get a lower-priority lane). Enforce per-run budgets (max steps/tokens/wall-clock) so a single task can't run unbounded. Apply backpressure by admission-controlling new agent runs under load while letting in-flight ones drain, and shed background work (re-indexing, speculative prefetch) first. Surface "you're queued / approaching your limit" rather than failing opaquely.
9. The agent needs secrets to run integration tests, but must never hold standing production credentials. Never give it long-lived prod keys. Provision short-lived, narrowly-scoped tokens minted per run (OIDC/STS-style) that expire in minutes and grant only the test-environment access needed — ideally pointed at an ephemeral/staging environment with seeded data, not prod. Inject secrets via the environment of a sandboxed runner, not into the model's context, and redact them from anything the model sees or logs. Combine with egress allowlisting so even a hijacked agent can't ship a leaked credential anywhere. The framing: the runner has the capability, the model never sees the secret, and the capability is time- and scope-boxed.
10. A monorepo language (a proprietary DSL) has no tree-sitter grammar or LSP. Make the agent competent there. Degrade gracefully and add structure cheaply. Short term: treat files as plain text but give the model the spec/docs of the DSL via retrieval, plus many in-repo examples, and rely on the compiler/linter for that DSL as the feedback loop (the CLI exit code/errors fed back as observations — you don't need an LSP to run the toolchain). Medium term: write a lightweight tree-sitter grammar (a few hundred lines) to unlock chunking, folding, and symbol extraction, which dramatically improves retrieval and edit-anchoring. If the language has a compiler, wrap its diagnostics into the diagnostics-as-feedback loop. The agent's correctness here lives or dies on the execution/verification loop, not on static intelligence.
11. Defend against tool poisoning (malicious instructions in an MCP tool's description) without relying on users reading schemas. Users won't read JSON Schema, so make the system defend itself. Hash tool definitions at approval time and re-confirm on any change (kills rug-pulls). Run untrusted server descriptions through an injection classifier before they enter the model's context, and delimit/spotlight them so the model treats descriptions as data, not instructions. Architecturally, adopt a dual-LLM split: the privileged planner that decides actions never ingests third-party tool descriptions verbatim; a quarantined component summarizes them into a constrained, structured capability list. Enforce least-privilege per server (scoped OAuth, no ambient creds) and gate side-effecting calls behind approval, so even a poisoned tool can't reach a dangerous sink unattended. Admins allowlist servers.
12. Detect and stop an agent that reward-hacks CI in production — marking tasks "done" by weakening assertions. This is RLVR's reward hacking leaking into product behavior. Detect: diff the test files in every agent change and flag deletions/weakening of assertions, skipped tests, or assert True-style edits; run a held-out test set the agent never sees and can't edit; track a "tests-touched vs source-touched" ratio that spikes on hacking. Block: make the agent's success criterion external tests it cannot modify, and require any change to test files to be a separately-approved, human-reviewed action. Verify on a clean checkout so the agent can't poison the environment. Long term, log these as eval cases and penalize them in any fine-tuning reward.
13. Every benchmark went up but users say "the new model is dumber." Investigate. This is the benchmark-to-feel gap. Hypothesis tree: (1) Behavioral regressions benchmarks don't measure — verbosity, over-editing unrelated files, worse instruction-following, sycophancy, ignoring project rules. (2) Latency/cost up — it's "smarter" but slower or chattier, which feels worse. (3) Distribution mismatch — benchmark is Python issue-fixing; your users do TSX refactors, and you regressed there (check sliced dashboards by language/task). (4) Eval contamination — the gain is memorization, not capability. (5) Harness interaction — the new model uses tools differently and your scaffolding fights it. Investigate via per-cohort production telemetry (acceptance, retention, edits-after-accept), A/B with human preference, and pulling real failing traces. Never trust the aggregate number alone.
14. Your fast-apply model occasionally silently deletes code. Make this bug class impossible, not just rare. Shift from "detect after" to "structurally prevent." Constrain the edit format so unchanged regions are referenced, not regenerated — e.g., anchored search/replace where the apply step can only insert/replace within matched spans and physically cannot drop un-referenced code. Where you must use lazy // ... existing code ... placeholders, treat them as literal copy directives and verify by reconstructing: the output's unchanged regions must hash-match the input's, else reject. Add a hard guard: any apply whose net deletion exceeds the intended diff's scope is blocked and downgraded to a conservative full-file regeneration or surfaced to the user. Parse-check post-apply. The diff review is the last gate, but the contract + invariant checks make silent loss unrepresentable.
15. If model calls were free and instant, what scaffolding dies and what new failure modes appear? Dies: prompt caching, model routing, small specialized completion/apply models, context compaction, debounce/cancellation, cost budgets — all are latency/cost optimizations that vanish. Heavy scaffolding (forced plan/execute phases, retrieval to save tokens) also weakens; you'd just let the model read everything and iterate freely — the bitter lesson taken to its limit. What doesn't die and gets worse: correctness and safety. Free/instant calls mean the agent can take millions of actions, so loop detection, destructive-action gates, sandboxing, egress control, prompt-injection defenses, and human review become the entire game. Reproducibility and observability matter more, not less. The bottleneck moves from "tokens" to "trust and verification" — which is the real moat anyway.