Week 10 studio: audit one instance

The second session is a 90-minute studio on your own project. You audit one instance: one task that an agent attempted in your project, the check that decided it, and the record of its attempts. The lecture did this on an invented table; here the record is yours, so the result is a statement about your project's evidence.

Due this week: A6. Its rules and hand-in are on the assignment page.

Work individually, with a written alternative to any discussion. Follow the practices in Tools and model access. Nothing below calls a model. If you choose to run your agent again to add attempts, use your course seat as usual.

Before you start

Bring one instance. A good one is small and already has a check you can run:

a task from your project that your agent attempted, with the test or command that accepted or rejected the result; or
a change from one of your open-source contributions that a project check accepted.

One recorded attempt is enough. If no check decided the task, pick another task: an instance without a check has nothing to audit.

The accounting in step 4 uses the course-written script from the lecture. In a shell at the course checkout, remember where it is, then keep the same shell for the rest:

AUDIT_TOOL="$PWD/lectures/week-10/fixture/audit.py"

Change to your project repository. Every later command runs there.

Studio sequence

Minutes	Activity	Evidence to keep
0–10	Name the instance and the claim	Revision, task, check, claim
10–30	Can the check fail?	One wrong change and what the check said
30–45	Was the pass earned?	One fresh input and its result, or "cannot tell"
45–65	Is the denominator complete?	The attempt table and the accounting output
65–75	pass@k, or why it is undefined	n, c, k and the figure or the reason
75–80	Which grader decides this instance?	One sentence
80–90	Decision and exchange	The four-line decision and one question from a peer

1. Name the instance and the claim (10 minutes)

Record the revision you are auditing and whether the tree is clean:

git rev-parse HEAD
git status --short

The first command prints one commit hash. The second prints nothing when the tree is clean; if it lists files, commit or set them aside first, so that the audit describes a revision someone else can check out.

In your work record, write four lines: the task as it was given to the agent, the exact command that checks it, how many attempts were declared, and the claim you are auditing in one sentence, for example "the agent fixed the date parser; the tests pass". Then write a prediction: which of the three questions do you expect to change that claim?

2. Can the check fail? (20 minutes)

Give the check a result you know is wrong. Do it in a separate worktree so that your working tree stays as recorded:

git worktree add --detach ../week10-audit-scratch HEAD

Preparing worktree (detached HEAD 347c1a7)
HEAD is now at 347c1a7 practice: first commit

Your hash and commit subject will differ. In ../week10-audit-scratch, make one small wrong change to the code the task was about: return a constant, drop a condition, or swap a comparison. Run the instance's check command there, exactly as written in step 1, and record its exit status.

The check fails. It can detect this defect. Write down which one; that is all it shows.
The check still passes. You found a wrong result the check accepts. Write the smallest case that would catch it. Do not add it to the project yet: note it as a proposed repair, with the wrong change that motivated it.

Try a second wrong change that targets a different clause of the task. You may ask your agent to propose wrong changes; you choose which ones to run, and you read each one before running it. Then remove the scratch worktree:

git worktree remove --force ../week10-audit-scratch

3. Was the pass earned? (15 minutes)

Open the change the agent made for one attempt that passed. Without running anything, answer in one sentence: what does this change compute? Then run it on one input that the check never showed it, taken from the task statement, and record the expected and observed result.

Write one of three judgments, with the evidence beside it:

Judgment	Evidence to record
Earned	The fresh input, the expected result and the observed result
Not earned	What the change does instead: a constant, a special case for the checked input, an edited test, a printed success marker
Cannot tell	What the record lacks, such as the diff or the command output, and where you looked

If the record shows the agent reading tests, history or another answer, quote what the task allowed before you judge. A cue is a reason to look, not a conclusion.

4. Is the denominator complete? (20 minutes)

Write every declared attempt into a table, including the ones with no result. Start from this template, which describes three declared attempts: one pass, one timeout, and one that left no record:

mkdir -p week10-audit
cat > week10-audit/instance.json <<'__TABLE__'
{
  "protocol": {"tasks": ["my-instance"], "attempts_per_task": 3},
  "published_headline": "replace with the claim you are auditing",
  "rows": [
    {"task": "my-instance", "attempt": 1, "state": "completed", "recorded": "pass"},
    {"task": "my-instance", "attempt": 2, "state": "timeout", "recorded": null}
  ]
}
__TABLE__
python "$AUDIT_TOOL" headline --runs week10-audit/instance.json
python "$AUDIT_TOOL" denominator --runs week10-audit/instance.json

{"claim": "replace with the claim you are auditing", "kind": "headline_as_published", "passed": 1, "percent": 100.0, "rate": "1/1", "scored_rows": 1}
{"accounted_for": 3, "declared": 3, "fail": 0, "kind": "as_recorded", "missing": 1, "not_earned": 0, "pass": 1, "pass_rate_if_unresolved_fail": "1/3", "pass_rate_if_unresolved_pass": "2/3", "timeout": 1, "unknown": 0}
{"attempts": ["pass", "timeout", "missing"], "task": "my-instance"}
{"accounted_for": 3, "declared": 3, "fail": 0, "kind": "audited", "missing": 1, "not_earned": 0, "pass": 1, "pass_rate_if_unresolved_fail": "1/3", "pass_rate_if_unresolved_pass": "2/3", "timeout": 1, "unknown": 0}

The template's headline is 1 of 1; its denominator is 1 of 3, with one attempt unresolved. Now replace the template with your own record:

set attempts_per_task to the number of attempts that were declared before the run, not the number that left a row;
write one row per attempt that left a record: state is completed or timeout, and recorded is pass, fail, or null when the attempt completed without a verdict;
write no row for an attempt that left no record, so that it is counted as missing.

Run the two commands again. For your table the audited row repeats the as_recorded row, because the script knows nothing about your check. Apply your own findings by hand: if step 2 showed that a recorded pass was a wrong result, or step 3 that it was not earned, change that row's recorded value to fail, add a "note" field saying why, and rerun. Keep both outputs.

5. pass@k, or why it is undefined (10 minutes)

Run the calculation on your table. For the unedited template it prints:

python "$AUDIT_TOOL" passk --runs week10-audit/instance.json

{"n": 3, "pass@1": {"high": "2/3", "low": "1/3"}, "pass@2": {"high": "1", "low": "2/3"}, "passes": 1, "task": "my-instance", "unresolved": 1}
{"kind": "mean_over_declared_tasks", "pass@1": {"high": "2/3", "low": "1/3"}, "pass@2": {"high": "1", "low": "2/3"}, "tasks": 1}

Before reading your own output, compute pass@1 and pass@2 by hand from n and c. Three cases need a sentence instead of a number:

With one declared attempt, the script prints "pass@2": "undefined: k exceeds n". Report pass@1 with n = 1 beside it and stop there.
If a later attempt saw the earlier failure, the attempts are not independent. Report the first attempt alone and describe the retry as a separate procedure.
If an attempt is unresolved, report the range.

6. Which grader decides this instance? (5 minutes)

Write one sentence: is your instance decided by code, by a person, or by a model judge, and is that the simplest grader that can decide it? If a model judges anything in your project, add where its labelled calibration cases are, whether the rubric was frozen before the last test, and which cases were held out. If there are none, say so; that is a finding.

7. Decide and exchange (10 minutes)

Finish with four lines:

Supported:          <the claim your evidence supports, with its denominator>
Not supported:      <at least two claims the record cannot carry>
Decision:           accept | narrow | recheck | stop (evidence: <outputs or files>)
Decide again when:  <the check, the table or the task statement changes>

Exchange records with one other student, in person or in writing. Ask one question about their denominator and one about their wrong change. Add the question you received and your answer to your record.

What to keep

One plain-text record with the revision, the four lines from step 1, both wrong changes and what the check said, the earned judgment, week10-audit/instance.json with both accounting outputs, the pass@k line and the decision. Keep keys, private data and raw chat logs out of it. Whether week10-audit/ is committed to your project is your choice; your final project report can cite it either way.

Participation and accessibility

Everything here is plain text and can be done with a keyboard. A written or asynchronous exchange replaces the discussion in step 7 with the same evidence. If a command fails, keep the failure in the record and ask for help with the exact output; a step you could not run is recorded as not run.