Agentic Software Engineering · Week 10

Evaluating agents

This week

First session (lecture)
Can the check fail, was the pass earned, is the denominator complete; pass@k on a small table; calibrating a model judge.
Second session
Studio: audit one instance.
Due
A6

Materials

Film

Before you trust 25 points

Made for the earlier edition of this course, which ran model-written code through a course launcher. The current course does not require one.

8:58 · subtitles: English, 中文 · download

A pass rate is a claim about checks, passes and a denominator.