Eval-Driven Agent Development

• Name the three evaluation layers — static benchmarks, custom offline, online production — and what each is for.
• Explain the evaluator-optimizer tight loop.
• Describe the 2026 best practice: evals live next to code, run in CI, gate PRs.
• Connect every Phase 14 lesson to the eval case it generates.

Agents pass demos. They fail in production in ways demos cannot predict. Benchmarks answer "is this model broadly capable?" not "is this agent shipping the right patches for my product?" The answer: evaluation at three layers, running continuously, with every guardrail and learned rule mapped to an eval case.

Three evaluation layers

1. Static benchmarks — SWE-bench Verified for code (Lesson 19), WebArena/OSWorld for browsing / desktop (Lesson 20), GAIA for generalist (Lesson 19), BFCL V4 for tool use (Lesson 06). Use for cross-model comparison and regression gating. Contamination is real: SWE-bench+ found 32.67% solution leakage. Always report Verified / +-audited scores.

2. Custom offline evals — your product's shape:

   • LLM-as-judge (Langfuse, Phoenix, Opik — Lesson 24).
   • Execution-based (run the patch, check tests).
   • Trajectory-based (compare action sequences against gold; OSWorld-Human shows top agents 1.4-2.7x over gold).

3. Online evals — production:

   • Session replays (Langfuse).
   • Guardrail-triggered alerts (Lesson 16, 21).
   • Per-step cost / latency tracking (Lesson 23 OTel spans).

Evaluator-optimizer (Anthropic)

The tight loop:

1. Proposer generates output.
2. Evaluator judges.
3. Refine until evaluator passes.

This is Self-Refine (Lesson 05) generalized. Any agent flow you care about can wrap in evaluator-optimizer for reliability.

2026 best practice

• Evals live next to code.
• Run in CI on every PR.
• Gate merge on eval scores (e.g. "no regression > 5% vs main").
• Every guardrail maps to an eval case.
• Every learned rule (Reflexion, pro-workflow learn-rule) maps to a failure case.

Tying Phase 14 together

Every lesson in Phase 14 generates eval cases:

If your eval suite has cases for each, you have covered Phase 14.

Where eval-driven development fails

• No baseline. Evals without a last-known-good are unreadable. Store baselines.
• LLM-judge without grounding. Judges hallucinate too. CRITIC pattern (Lesson 05) — judge grounds on external tools.
• Over-fitting to evals. Optimizing for the eval diverges from production usefulness. Rotate cases.
• Flaky evals. Non-deterministic cases cause false alarms. Pin seeds, snapshot state.

code/main.py is a stdlib eval harness:

• Case registry with categories (benchmark, custom, online).
• A scripted agent under test.
• Evaluator-optimizer loop: propose, judge, refine until pass or max rounds.
• CI gate: aggregate pass rate + regression against baseline.

Run it:

python3 code/main.py

Output: per-case pass/fail, regression flag, CI gate verdict.

• Write eval cases in the same repo as your agent code.
• Run them on every PR via CI.
• Fail the build on regression.
• Track pass rate over time.
• Tie every production failure to a new case.

1. Take one of your production failures. Write an eval case that reproduces it. Does your agent pass it now?
2. Build an LLM-judge rubric for your domain with three dimensions (factual, tone, scope). Score 50 sessions.
3. Wire the eval suite into CI. Fail the build on >=5% regression.
4. Add a trajectory-efficiency metric: how many steps did the agent take vs a gold trajectory?
5. Map every Phase 14 lesson to an eval case in your suite. Any missing? That's a gap to close.

• Anthropic, Building Effective Agents — "start simple, optimize with evals"
• OpenAI, SWE-bench Verified — the curated benchmark
• Berkeley Function Calling Leaderboard — tool-use benchmark
• Langfuse docs — evals + session replay in practice