Week 10 readings: Evaluating agents

The required reading is the notes, worked in your own copy of the fixture, and the studio. Attempt each prediction in the notes before you run the command that answers it. If you have already seen an output, mark your next attempt as informed practice.

The two sources below are optional. Each entry names the primary source, the part to read, and the question from this week to ask of it. The lesson's own tables are invented for practice; none of these sources is the origin of a number on the course pages.

• Source 1 — DebugML, "Finding Widespread Cheating on Popular Agent Benchmarks". Read "Mining git history (SWE-bench)" and "Prompt-injecting the verifier (Terminal-Bench 2)" in the article (about 15 minutes). For each case, ask this week's first two questions: what did the check actually examine, and what did the passing trajectory actually do? Then read "Takeaways" for the limit: the authors show specific trajectories and say that the true prevalence remains unknown, so the cases do not give a rate. (benchmarks/cheating-agents; as_of: 2026-09-05; Source 1)
• Source 2 — Inspect, "Welcome to Inspect". Read "Hello, Inspect" on the documentation page (about 10 minutes). Name the dataset, the solver and the scorer in its first example, and say which kind of scorer it uses: a text comparison, a model judgment or a custom algorithm. Then ask what you would need before trusting a model-judged scorer; the framework does not answer that for you. (tools/inspect-ai; as_of: 2026-09-05; Source 1; Source 2)

For one source, write two sentences in your work record: the claim you took from the named section, and the check in your own studio record that could support or challenge it. Do not report a source's example as your own run.