Week 10: Evaluating agents

This week asks what a pass rate is worth. The lecture audits one invented attempt table with three questions (can the check fail, was the pass earned, is the denominator complete), computes pass@k on the audited table, and calibrates a model judge against a labelled sample before testing it once on a held-out sample. Every step pairs something generated (a repair, a recorded change, a verdict) with a check you run yourself. No command calls a model.

Start with the notes and a scratch copy of the fixture. The second session is the studio, where you audit one instance from your own project. Then use the self-check and the optional readings. The slide source follows the same sequence. A6 is due this week.