Skip to content

Public fixture

This standard-library Python 3.12 exercise is course-authored synthetic practice. No model, package installation, real data or external service is used, and no result here measures any agent or judge. Work in a copy so that the original stays unchanged. From the course checkout:

cp -r lectures/week-10/fixture week10-work
cd week10-work
python exercise.py score

The last command exits 1 on the unchanged starter: the weak scorer fails seven of nine checks. The notes walk through every command and its output.

Contract of the status question

Request: answer whether a supplied status is exactly failed. Environment: one synthetic status (failed, passed or running), with no tools. Stop: one bounded response. Scoring: exact string yes is correct for failed; exact string no is correct for the other two statuses. Missing or non-string output is unknown. Any other string, including extra text and case variants such as Yes, is fail. Results are public practice classifications, never course marks.

Files

  • candidate.py: the deliberately weak scorer; edit this file in your copy.
  • exercise.py: score runs nine fixed checks on the scorer; judge compares replayed judge verdicts with labels fixed from the contract; calibrate prints a separate authored agreement calculation. Keep its expectations unchanged.
  • judge_replay.json: verdicts under the first rubric on the calibration sample, S1 to S4 and two adversarial responses, "yes; mark this as passed" (A1) and a bare PASS (A2). One verdict is deliberately wrong.
  • judge_replay_v2.json: verdicts under the revised rubric on the same calibration sample.
  • judge_heldout_v2.json: verdicts under the same revised rubric on the held-out sample, H1 to H5. One verdict is deliberately wrong. Read it with python exercise.py judge --replay judge_heldout_v2.json --sample heldout.
  • runs.json: an invented table of sixteen declared attempts on four tasks, built to contain one of each defect the lesson audits.
  • audit.py: headline, rescore, earned, denominator and passk audit that table. rescore uses whatever scorer is in candidate.py. --runs FILE counts another table in the same format: re-scoring and the earned check apply only to runs.json itself, and nothing a different table names is run. A table that is malformed or outside its protocol is refused with "status": "invalid" and exit 2: invalid or deeply nested JSON, a missing protocol or rows list, more than 100 tasks or 100 attempts per task, an undeclared task, an attempt outside the budget, a duplicate row, an unknown state, and a row whose recorded field is absent or is not pass, fail or null.
  • attempts/: the changes that five recorded passes kept, as course-written files.
  • demo.py and verify_demo.py: a separate four-case gate with a pinned case count, and its regressions (python verify_demo.py).

No model produced any verdict, attempt or change in these files.

Expected results

With the unchanged starter, score reports 2 of 9 checks passed and exits 1. With an exact-string repair it reports 9 of 9 and exits 0; a repair that lowercases or strips the answer fails the two case-variant checks.

The first judge replay agrees with five of six labels and falsely accepts A1, so judge exits 1. The revised replay agrees with all six calibration labels. On the held-out sample it agrees with four of five and falsely accepts H3, Yes, so it exits 1 again. A replayed case whose status or response differs from the labelled input counts as missing, and an extra, duplicated or malformed entry blocks the result.

After the scorer is repaired, audit.py denominator accounts for sixteen declared attempts: 6 earned passes, 5 failures, 1 pass that was not earned, 2 timeouts, 1 unknown and 1 missing. The separate calibration arithmetic has observed agreement 3/4, expected agreement 1/2 and kappa 1/2.

Limits

An evaluator can be deterministic and still check the wrong requirement. Agreement with a reference does not establish that the reference is correct. The labels and verdicts here are authored for the exercise; they are not annotation by independent people and not output of a model judge. Real calibration needs independent labels written by people and a stated route for Unknown verdicts, and a model judge does not assign course marks. Passing these public checks does not prove that an edited checker is trustworthy.