Skip to content

A6: Audit an evaluation instance and scorer

At a glance

  • Task: inspect one evaluation instance and scorer, then state exactly what its result supports and does not support.
  • Week: 10
  • Weight: 2.5 points
  • Due: Sunday 23:59 in the listed week
  • Hand in: an audit record, recomputed counts and report.json.
  • Done when: every denominator includes success, failure, timeout and missing outcomes, and one scorer weakness is demonstrated.
  • Time: published after a timed pilot; record your prediction and actual time.

Objectives

Check whether an evaluation can fail, whether its denominator is complete and how far a small result generalizes (LO6).

Prerequisites

Follow the safety practices in syllabus §9:

  • Run the agent in a dedicated directory or container.
  • Keep residency, employer and cloud credentials out of that environment.
  • Keep keys out of repositories, prompts, screenshots and logs.
  • Run injection demonstrations offline against the local machine only.

Use the mature coding-agent seat supplied for the course. Do not run an unknown generated command on a credential-bearing host.

Tasks

  1. Predict. Identify one check you expect to be weak and one outcome likely to be missing.
  2. Generate. Ask an agent to propose an audit plan. Keep it as claims, not truth.
  3. Audit the instance. Open starter/instance.json, including its prompt, inputs, checks and four declared attempts. Run a safe counterexample showing a check can fail or a wrong answer can pass.
  4. Audit the scorer. Run python starter/run_scorer.py starter/instance.json, then recompute the result with success, failure, timeout and missing rows visible. Compare the reported rate with a denominator that includes every declared attempt and record any difference you find.
  5. Claim limits. Write one supported claim and at least three unsupported claims, such as a ranking, unseen-task generalization or stable rate from one attempt. Revise after feedback from a peer or the teaching team during the assignment week; record who gave it and the date, or your self-review date if no feedback was available. Finish the decision line.

Generation is a proposed audit; verification is the counterexample and recomputation. Never copy a reference answer or treat a course fixture as an industry result.

Deliverables

Include the instance and scorer paths, audit output, counts and report.json. Counts must include declared, success, failure, timeout and missing. Run:

python practice/a6/check.py path/to/submission

Grading

Criteria are a reproducible counterexample; a complete denominator; a precise supported claim; at least three limits; and a revision backed by evidence. The public check catches inconsistent records but does not judge the scorer or make a statistical claim.

AI use rules

Green: drafting an audit plan and recomputation table. Yellow: suggesting counterexamples, followed by your own run and inspection. Red: deciding what the result supports, what it cannot support and the final decision. Attach one AI-use note ID.

Estimated time

Published after a timed pilot. Record predicted and actual time, including recomputation and revision.

Work record outline

A6 report
instance: TODO
scorer_path: TODO
supported_claim: TODO
audit_evidence: [TODO]
counts: {declared: TODO, success: TODO, failure: TODO, timeout: TODO, missing: TODO}
unsupported_claims: [TODO, TODO, TODO]
explanation: TODO
ai_use_note: TODO
decision: accept | narrow | recheck | stop
because: evidence ID(s)
next: TODO