Skip to content

Week 10 self-check: Evaluating agents

Answer without an agent and before reopening the notes. Give one concrete input, count or command result in every answer. These questions are formative and carry no marks. All data in them are invented for practice. After your first attempt, compare with the public answer guidance.

Can the check fail?

  1. A scorer is return "pass" if output else "fail". Name one wrong answer it accepts and the only kind of output it rejects. What do four recorded passes from this scorer show?
  2. A proposed repair compares output.strip().lower() == expected. The contract asks for the exact string yes or no. Which two inputs separate this repair from the contract, and what should each be scored?
  3. A task statement says "raise ValueError when total is 0". Its check requires the message no tasks. A correct implementation raises ValueError("total is 0"). What does this check do to the pass rate, and what should the check require instead?

Was the pass earned?

  1. The only test is assert pass_rate(2, 4) == 0.5. The recorded change is return 0.5 and the record ends with 1 passed. Was the pass earned? Give one input that settles it, with the expected and the observed result.
  2. Same test. Another recorded change is return passed / total, with no handling of total == 0. It returns 0.25 for pass_rate(1, 4) and raises ZeroDivisionError for pass_rate(0, 0). The task asks for ValueError when total is 0. Is this an unearned pass, a weak check, or both? Is it a shortcut like the change in question 4? Say what you would change.
  3. A record shows git log -p -- src/versions.py before the fix, and a detector flags it. The task statement says "You may inspect the git history of src/versions.py". Write the three things your record should hold, and your judgment.

Is the denominator complete?

  1. A protocol declares five tasks with two attempts each. The record has 6 passes, 1 fail, 1 timeout and 1 completed attempt with no verdict; one declared attempt left no row. The report says "6 of 7: 86%". Give the count in each category, show that they add up to the declared number, and state the pass rate the record supports.
  2. A suite has four cases. Someone deletes one; the remaining three pass. What does a gate that counts the cases it finds report? What does a gate with the expected count written as a constant report, and why is that the right answer?

pass@k

  1. One task has five independent attempts: pass, fail, fail, pass, fail. Compute pass@1 and pass@2, showing n, c and the formula or the list of pairs.
  2. A report gives "pass@3 = 100%" for a task that was attempted once and then retried twice, each retry seeing the previous error message. Name two reasons the figure does not mean what it says. With one attempt per task, what is pass@2?

Calibrating a judge

  1. On five held-out cases the labels are fail, pass, fail, fail, fail and the judge's verdicts are fail, pass, pass, fail, fail. Compute observed agreement, expected agreement and kappa. What kind of error is the one disagreement?
  2. After that held-out run you add a clause to the rubric and run the judge on the same five cases again. It now agrees on five of five. What can you claim, and what do you need before you can claim more?
  3. Choose the simplest way to decide each criterion and say why: (a) whether an answer is exactly yes; (b) whether an explanation of a build failure is accurate; (c) the reference labels a model judge is compared with.

Deciding again

  1. Your audit concluded "6 of 16 declared attempts are earned passes, 2 unresolved". A colleague then replaces the scorer with a new version. Which steps do you run again before quoting the figure, and what is the status of your earlier decision until then?

Transfer decision

The repaired exact-answer scorer passes all nine public checks at one revision. State the bounded use that this evidence supports, one variation it has not been tested on, and the check that would be required before accepting JSON responses.

Spaced retrieval

R1. Retrieval 1. A model review of a change reports no findings. What does that support about the change, and what do you still check yourself?

R2. Retrieval 2. An unattended agent writes PASS in its report without running the tests. What kind of gate blocks this path, and who must control it?