14. Evaluation of Coding Agents
What This Subsystem Is and Why It Exists ¶
Evaluation is the discipline of answering one deceptively simple question: is the agent actually good, and did my last change make it better or worse?
Without it you are flying blind. A new model drops, you swap it in, and "it feels faster" — but does it resolve more bugs? You tweak the system prompt to fix one annoying behavior and silently break three others. You add a new tool and latency doubles. Evaluation is the instrumentation that turns "vibes" into numbers you can defend in a planning meeting or a launch review.
The reason this is its own hard subsystem — and not just "run some tests" — is that coding agents are non-deterministic, multi-step, and open-ended. A traditional unit test asserts f(2) == 4. An agent eval has to answer "given a messy GitHub issue and a 2-million-line repo, did the agent produce a patch that makes the real test suite pass, without breaking anything else, in reasonable time and cost?" The output space is enormous, the same input can yield different outputs run-to-run, and the thing you actually care about (developer productivity, trust) is only loosely correlated with anything you can cheaply measure.
Think of it like evaluating a junior engineer. You don't grade them on a multiple-choice quiz (HumanEval). You give them real tickets and check whether the feature ships, the tests pass, and they didn't set the codebase on fire (SWE-bench). And ultimately you watch them work over months (production telemetry).
How It Works: The Evaluation Stack, Layer by Layer ¶
Evaluation in a Cursor-style product is a pyramid, cheap-and-fast at the bottom, expensive-and-truthful at the top.
Layer 1: Offline benchmark harnesses (pre-merge, CI) ¶
The data flow for a benchmark like SWE-bench:
- Task instance = a real GitHub issue + the repo pinned at the commit before the fix, plus the "gold" PR's test patch (the tests that should pass after a correct fix).
- The harness spins up a clean, isolated container (Docker) with the repo at the broken commit and the project's dependencies installed.
- The agent receives the issue text (and tools to read/search/edit files, run commands). It works autonomously and emits a patch (a unified diff).
- The harness applies the patch, then runs two test sets:
FAIL_TO_PASS(tests that failed before and must now pass) andPASS_TO_PASS(tests that passed before and must still pass — the regression guard). - The instance is resolved only if all FAIL_TO_PASS and PASS_TO_PASS tests pass. Binary. No partial credit.
- Aggregate across hundreds of instances → resolve rate (e.g. "we resolved 62%").
The container isolation matters enormously: tests run arbitrary code, repos have conflicting dependency versions, and a flaky environment poisons your numbers. SWE-bench ships per-instance Docker images for exactly this reason.
Layer 2: Component / regression evals (pre-merge, fast) ¶
Not everything is a full agent loop. You also have narrow evals for individual pieces:
- Edit-application evals: did the model's proposed diff actually apply cleanly to the file? (More on this below — it's a top failure mode.)
- Autocomplete evals: given a code prefix, does the completion match what the developer actually typed next?
- Tool-use evals: given a task, did the agent call the right tool with valid arguments?
- Prompt regression suites: a fixed set of "golden" prompts whose outputs are checked when you change the system prompt or model.
Layer 3: Online A/B testing and telemetry (post-deploy, truthful) ¶
The benchmark says 62%. The user says "meh." Layer 3 closes that gap. You ship the change to a fraction of real users behind a feature flag and measure acceptance rate, retention, edits-after-accept, and latency on real traffic. This is the only layer that measures real-world feel, but it's slow (days to significance) and you can't run a hundred of these at once.
The Benchmarks That Matter ¶
SWE-bench and SWE-bench Verified ¶
- SWE-bench (2023): ~2,294 task instances from 12 popular Python repos (Django, sympy, scikit-learn, etc.). The defining benchmark for agentic coding because it requires repo navigation, multi-file edits, and passing real test suites — not toy function synthesis.
- The problem: the original set had noise — underspecified issues, broken/flaky tests, and gold tests that even a correct human fix wouldn't pass.
- SWE-bench Verified (2024, built with OpenAI): a 500-instance human-validated subset where engineers confirmed each issue is solvable and the tests are fair. This is now the headline number vendors quote. Frontier models in 2025 report roughly 60–75% on Verified depending on the scaffold; numbers climb fast, so treat any specific figure as a snapshot.
- Variants: SWE-bench Lite (300 cheap instances for quick iteration), SWE-bench Multimodal, and SWE-bench Multilingual (beyond Python).
Analogy: SWE-bench Verified is the "road test" of coding benchmarks — real car, real traffic, examiner checks you didn't run a light.
HumanEval and MBPP — the saturated, contaminated old guard ¶
- HumanEval (OpenAI, 2021): 164 hand-written Python problems; the model writes a function from a docstring, graded by hidden unit tests via pass@k.
- MBPP (Mostly Basic Python Problems): ~974 entry-level problems.
- Both are largely saturated (frontier models score 90%+) and almost certainly contaminated — they've leaked into training data, so a high score may reflect memorization, not capability. They measure single-function synthesis, not the repo-scale, tool-using work a real agent does. Useful as a smoke test, not a differentiator.
Aider polyglot ¶
- Aider's benchmark uses 225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python, and Rust. It specifically stresses two things agents fail at: writing correct code and applying edits in the exact diff/format the harness expects. A model that codes well but can't emit a clean edit scores poorly — which is exactly the real-world failure you want to catch.
RepoBench, LiveCodeBench, and contamination-resistant evals ¶
- RepoBench: evaluates repository-level code completion — retrieval + completion across files, closer to autocomplete-in-context than issue-fixing.
- LiveCodeBench / "live" evals: continuously scrape new problems (e.g. recent LeetCode/contest problems, or GitHub issues created after a model's training cutoff) so the test data couldn't have been memorized. Contamination resistance is the key idea: only by testing on data created after the cutoff can you trust the score reflects reasoning, not recall.
- SWE-bench Live / SWE-rebench: rolling sets of fresh GitHub issues for the same reason.
The Metrics That Matter ¶
| Metric | What it measures | Where used |
|---|---|---|
| pass@k | P(at least 1 of k samples is correct) | HumanEval/MBPP |
| Resolve rate | % of tasks fully resolved (all tests pass) | SWE-bench |
| Edit-application success | % of proposed diffs that apply cleanly | Apply pipeline |
| Autocomplete acceptance rate | % of suggestions the dev keeps | Production telemetry |
| Latency p50 / p95 / p99 | Tail response times | All layers |
| Cost per resolved task | $ (tokens) per success, not per attempt | Economics |
A few sharp points:
- pass@k vs pass@1:
pass@1is what users feel (you usually get one shot).pass@k(k=10, 100) rewards models that can get it right with sampling + a verifier. Reporting pass@100 to imply everyday quality is a common, misleading move. The unbiased estimator: sample n ≥ k completions, count c correct,pass@k = 1 − C(n−c, k)/C(n, k). - Cost per resolved task is the honest economic metric. An agent that resolves 65% at $2.50/success can be better than one resolving 70% at $18/success. Always normalize by success, not by attempt.
- Latency percentiles, not averages. For autocomplete the bar is brutal: a suggestion must appear in well under ~300 ms or the developer has already typed past it. p50 of 120 ms with p95 of 900 ms is a bad experience because the tail is where trust dies. For agentic tasks, users tolerate seconds-to-minutes, but you still track p95 to catch runaway loops.
- Acceptance rate is necessary but gameable. A model can boost acceptance by suggesting trivial, obviously-correct completions. Pair it with "survival" / retention — what fraction of accepted code is still present after N minutes/edits. Accepted-then-immediately-deleted is a false positive.
Engineering Challenges and Tradeoffs ¶
Latency ¶
Full SWE-bench Verified is 500 tasks × a multi-minute agent loop each → hours and serious money per run. So teams iterate on Lite (300) or a curated 50–100 subset for fast signal and reserve the full set for release gates. The tradeoff: small subsets have high variance and can be over-fit.
Cost ¶
A single Verified run can cost tens to hundreds of dollars in API tokens. Running it on every PR is infeasible, so it's gated to nightly or release builds, while cheap component evals run per-commit.
Quality vs. contamination ¶
The cruelest tradeoff: the best-known benchmarks are the most contaminated. Every public benchmark has a half-life — the moment it's popular, it leaks into the next training run. This forces continuous investment in fresh, private, held-out evals.
Privacy / scale ¶
You'd love to eval on your users' real repos and edits — that's the most representative data. But that's their proprietary code. Mitigations: opt-in telemetry, on-device/aggregated metrics, k-anonymity, and synthetic or internal repos that mimic real distributions. At scale you also can't manually inspect millions of sessions, which is why automated judges (next section) become tempting — and dangerous.
The benchmark-to-feel gap ¶
The persistent, unkillable problem: a model can climb SWE-bench while feeling worse day-to-day. Why? Benchmarks reward "eventually produces a passing patch after 40 tool calls"; users reward "got it right in one shot, fast, explained itself, didn't touch unrelated files." Benchmarks under-weight latency, verbosity, over-editing, sycophancy, and instruction-following nuance. Closing this gap is the whole game, and it's why no serious team ships on benchmark numbers alone.
LLM-as-Judge and Its Pitfalls ¶
For open-ended outputs (explanations, refactors, "is this code well-structured?") there's no unit test. So teams use a strong model to grade outputs against a rubric — LLM-as-judge.
It scales beautifully and correlates okay with human judgment. But the pitfalls are well-documented and you must name them in an interview:
- Position / order bias: judges favor whichever answer is shown first (or second). Mitigate by swapping order and averaging, or randomizing.
- Verbosity / length bias: judges reward longer, more confident answers even when wrong.
- Self-preference / self-enhancement bias: a model rates its own family's outputs higher. Mitigate by using a different model family as judge, or an ensemble.
- Sycophancy & rubric drift: judges can be talked into agreeing, and "score 1–10" is noisy. Prefer pairwise comparison (A vs B) over absolute scores, and binary checklists ("does it compile? does it address the ask?") over vague Likert scales.
- Not ground truth: always calibrate the judge against human labels on a sample, and report the judge's agreement rate with humans. A judge you haven't validated is just another model hallucinating confidently.
Best practice: use execution-based grading wherever a test can decide it (deterministic, cheap, trustworthy), and reserve LLM-as-judge for the genuinely subjective slice.
Eval-Driven Development of the Harness ¶
Mature teams treat evals like a test suite for the product, not the model:
- Regression suites for prompts and tools: every system-prompt change, tool-schema change, or model swap runs against a golden set before merge. Caught a bad behavior in the wild? Add it as a permanent eval case (like a regression test for a bug).
- Eval-driven development: write the eval first, watch it fail, then change the harness until it passes — TDD for agent behavior.
- Sliced dashboards: track resolve rate by language, by repo size, by task type so a gain in Python doesn't hide a regression in Rust.
- Trace logging: store full agent trajectories (every tool call, token, retry) so a failed eval is debuggable, not just a red X.
How Leading Products Actually Do It ¶
Internals are mostly proprietary; the following is "commonly understood / reported," not gospel.
- GitHub Copilot: pioneered the metric playbook for autocomplete — acceptance rate and, more meaningfully, "retained" / surviving code (suggestions still in the file after a window). GitHub has published research correlating Copilot usage with task-completion speed. For its agent (Copilot Workspace / coding agent), it leans on SWE-bench-style internal evals.
- Cursor: heavy on production telemetry for its custom Tab/autocomplete and "apply" models, plus offline evals on its fast-apply model (the model that turns an LLM's loose edit into a clean diff). Cursor specifically obsesses over edit-apply success and latency because Tab must feel instant.
- Aider: famously transparent — publishes its polyglot leaderboard publicly, and its benchmark explicitly measures edit-format adherence (whole-file vs unified-diff vs search/replace), which is why Aider numbers are a good proxy for "can this model actually edit code."
- Claude Code / Anthropic and OpenAI Codex: report SWE-bench Verified as the headline agentic number, with internal held-out evals and human preference evals to catch the feel gap. Both emphasize agentic, tool-using evals over HumanEval.
- Sourcegraph (Cody): evaluates retrieval/context quality — its differentiator is code search/embeddings, so its evals stress "did we fetch the right context" (recall@k on relevant files) as much as final answer quality.
- Zed: focuses on latency as a first-class metric (it's a performance-obsessed editor) plus edit-prediction acceptance for its Zeta model.
- JetBrains: ships its own full-line completion model evaluated on acceptance/retention, and has published Kotlin/Java-specific benchmarks (Mellum), since its strength is JVM-language depth.
- Continue / Windsurf (Codeium): Continue is open-source and lets teams plug in their own evals; Windsurf reports agentic benchmarks and leans on production acceptance metrics for its Cascade agent and Supercomplete.
The pattern: everyone quotes SWE-bench Verified for agents and acceptance/retention for autocomplete, but the real moat is each team's private eval set and production telemetry.
Failure Modes and Mitigations ¶
- Contamination → benchmark in training data → inflated scores. Mitigate: live/held-out evals on post-cutoff data; private internal sets; canary strings to detect leakage.
- Overfitting to the benchmark → scaffold tuned to SWE-bench quirks generalizes poorly. Mitigate: rotate/refresh task sets; diverse benchmarks; weight production metrics highest.
- Flaky test environments → non-deterministic pass/fail poisons resolve rate. Mitigate: Docker isolation, pinned deps, retries with flake detection, the PASS_TO_PASS regression guard.
- Edit-application failures → model writes good code but a malformed diff that won't apply (a top real-world failure). Mitigate: dedicated apply models, lenient/fuzzy patch matching, and an edit-application eval that catches it before users do.
- Reward hacking on metrics → acceptance rate gamed by trivial suggestions. Mitigate: pair with retention/survival; track edits-after-accept.
- LLM-judge bias → unreliable subjective scores. Mitigate: pairwise + position-swap, cross-family judges, human calibration, prefer execution-based grading.
- Single-number tunnel vision → "62%!" hides a Rust regression or a 3× latency hit. Mitigate: sliced dashboards; always report resolve rate with cost and latency.
- Benchmark-feel gap → ship a "better" model users hate. Mitigate: gate launches on A/B + human preference evals, never benchmarks alone.
Interview Talking Points ¶
- Eval is a pyramid: cheap offline benchmarks (CI) → component/regression evals → expensive-but-truthful online A/B + telemetry. Benchmarks gate merges; production data gates launches.
- SWE-bench Verified (500 human-validated real GitHub issues, binary resolve via FAIL_TO_PASS + PASS_TO_PASS tests in Docker isolation) is the agentic benchmark. HumanEval/MBPP are saturated and contaminated — single-function synthesis, smoke-test only.
- Aider polyglot stresses edit-format adherence; RepoBench stresses repo-level completion; LiveCodeBench / SWE-bench Live are contamination-resistant by using post-cutoff data.
- Key metrics: pass@k (report pass@1 for real feel), resolve rate, edit-apply success, autocomplete acceptance + retention/survival, latency p50/p95, and cost per resolved task (normalize by success).
- Autocomplete latency bar is ~sub-300ms; tail (p95) is where trust dies. Acceptance rate is gameable — always pair with survival.
- LLM-as-judge scales but has position, verbosity, and self-preference bias → use pairwise + order-swap, cross-family judges, human calibration; prefer execution-based grading.
- The benchmark-to-feel gap is the central problem: models climb SWE-bench while feeling worse because benchmarks under-weight latency, over-editing, and instruction-following. Never ship on benchmark numbers alone.
- Contamination is the eternal enemy: every popular benchmark leaks into training data, so investment in fresh private held-out evals is permanent.
Likely Interview Q&A ¶
Q1: What makes SWE-bench better than HumanEval for evaluating a coding agent? HumanEval grades single-function synthesis from a docstring against hidden tests — it tests one narrow skill, it's saturated (90%+), and it's contaminated by training data. SWE-bench gives the agent a real GitHub issue and the actual repo, requiring multi-file navigation, tool use, and a patch that makes the project's real test suite pass (both the previously-failing tests and the previously-passing regression tests). It evaluates the whole agentic loop — search, reason, edit, verify — which is what a Cursor-style product actually does. SWE-bench Verified is the human-validated 500-instance subset that removes broken/underspecified tasks, which is why it's the number vendors quote.
Q2: Explain pass@k and when pass@1 vs pass@100 matters. pass@k estimates the probability that at least one of k sampled completions is correct, using the unbiased estimator 1 − C(n−c,k)/C(n,k) over n samples with c correct. pass@1 reflects everyday user experience — you usually get one shot. pass@100 reflects a model's potential if you have a perfect verifier to pick the right sample. Quoting a high pass@100 to imply day-to-day quality is misleading; for product decisions, pass@1 (or pass@k with the actual k your product samples) is the honest number.
Q3: What is the "benchmark-to-feel gap" and how do you close it? A model can climb SWE-bench while feeling worse to users because benchmarks reward "eventually produces a passing patch" and ignore latency, verbosity, over-editing unrelated files, sycophancy, and instruction-following nuance. You close it by not gating launches on benchmarks alone: layer in human preference evals, online A/B tests measuring acceptance/retention/edits-after-accept, and sliced telemetry. You also feed real failure cases back into your private eval set so the suite evolves toward what users actually care about.
Q4: What are the pitfalls of LLM-as-judge and how do you mitigate them? Main biases: position/order bias (favors first answer), verbosity bias (favors longer answers), and self-preference bias (rates its own model family higher). Plus general unreliability — absolute 1–10 scores are noisy and judges are sycophantic. Mitigations: use pairwise comparison with order-swapping and averaging, use a different/ensemble model family as judge, prefer binary checklists over Likert scales, calibrate against human labels and report agreement rate, and use execution-based grading (real tests) wherever the question can be decided deterministically — reserve the judge for the genuinely subjective slice.
Q5: Why is "cost per resolved task" a better metric than raw resolve rate? Resolve rate ignores economics. Agent A resolving 65% at $2.50/success can beat Agent B resolving 70% at $18/success because B burns far more tokens (more tool calls, retries, longer context) per win. Normalizing cost by success (not by attempt) captures the real efficiency. In production you can't ignore this — token cost and latency directly bound how aggressive an agent loop you can afford to ship.
Q6: What is contamination and how do contamination-resistant benchmarks work? Contamination is when benchmark problems and solutions have leaked into a model's training data, so a high score reflects memorization rather than reasoning. Every popular public benchmark gets contaminated over time. Contamination-resistant evals — LiveCodeBench, SWE-bench Live, SWE-rebench — only use problems created after the model's training cutoff (e.g. fresh contest problems or GitHub issues), so they couldn't have been memorized. Teams also maintain private held-out sets and use canary strings to detect leakage.
Q7 (system design): Design an evaluation system for a new autocomplete (Tab) feature shipping to millions of developers. I'd build it as layers. Offline: a RepoBench-style corpus of real code where I hold out the next token/line and measure exact-match and edit-similarity, sliced by language and context length, run in CI on every model change. Latency harness: measure p50/p95/p99 generation time on representative prefixes — the hard bar is suggestions visible in well under ~300ms, so I'd gate merges on p95 latency, not just quality, and likely use a small specialized model with speculative decoding and caching. Online: ship behind a feature flag to a small % of users; primary metrics are acceptance rate and, more importantly, survival/retention (is the accepted code still there after N edits/minutes) to defeat the "accept-then-delete" false positive, plus per-keystroke latency from real clients. Guardrails: monitor cost per accepted suggestion and a regression dashboard sliced by language/editor. I'd treat acceptance as necessary-not-sufficient and weight retention + latency highest, because for autocomplete, speed and trustworthy suggestions matter more than raw model IQ.
Q8 (system design / tradeoff): SWE-bench Verified takes hours and hundreds of dollars per run. How do you build a CI/eval pipeline that gives fast signal without going broke or fooling yourself? Tier it. Per-commit: cheap, deterministic component evals — edit-application success, tool-call validity, prompt regression golden set, and HumanEval/MBPP as a smoke test (seconds, cents). Per-PR / nightly: a curated 50–150 instance subset of SWE-bench (Lite or a hand-picked stratified sample across languages/repo sizes) for a fast directional read — accept the higher variance as the cost of speed. Pre-release gate: the full Verified 500 plus contamination-resistant live evals, run once, with results sliced by language and reported alongside cost and latency. Anti-self-fooling measures: rotate/refresh the subset to avoid overfitting, keep a private held-out set the team never inspects in detail, use Docker isolation with pinned deps and flake retries to keep numbers trustworthy, and never let a single aggregate number gate a launch — require the sliced dashboard plus an A/B confirmation. The core tradeoff is variance/cost vs. truth: small fast subsets for iteration, expensive full + live + online runs for the decisions that actually matter.