Week 10 self-check answer guidance

Use these explanations after your first attempt. Another input or wording is fine when it carries the same evidence. Every number below comes from the invented data in the question.

Q1

It accepts any non-empty text, for example no for a failed build or yes; mark this as passed. It rejects only empty or missing output. Four recorded passes show that four answers were non-empty. They show nothing about whether the answers were right, because no wrong non-empty answer could have failed.

Q2

A case variant and padded text separate them: Yes for a failed build and yes with a leading space (or NO for a passed build). The repair scores each as pass; the contract asks for the exact string, so each is fail. Run the unchanged public cases: case_yes and case_no fail for this repair.

Q3

It rejects a correct implementation, so correct attempts count as failures and the rate is too low. The contract names the exception type and leaves the message open, so the check should require ValueError and nothing about the message. It should still reject a returned value and a different exception type.

Q4

Not earned. The change returns the one value the test looks at. pass_rate(1, 4) should be 0.25 and the change returns 0.5; pass_rate(0, 0) should raise ValueError and it returns 0.5. The test file is untouched and 1 passed is true; the claim "fixed pass_rate" is what the record does not support.

Q5

Both. The pass is not earned: the task asks for ValueError when total is 0, and pass_rate(0, 0) raises ZeroDivisionError, so the change does not do what the task asked. The check is also weak: its one row cannot notice the missing case. It is not a shortcut, though. Unlike return 0.5, this change computes the rate honestly and meets a fresh ordinary input such as (1, 4); it is incomplete, not copied from the check. Report the two findings separately: add a check for pass_rate(0, 0) that requires ValueError, and reclassify the attempt as not earned, with the zero case as the evidence.

Q6

Keep the cue (a history read with patch content, at that step), the quoted authorization from the task statement, and what the unchanged check establishes. Judgment: the allegation is rejected because the task allowed the read. The pass still needs its own evidence, such as one input the earlier broken code got wrong.

Q7

Declared: 5 × 2 = 10. Pass 6, fail 1, timeout 1, unknown 1, missing 1: 6 + 1 + 1 + 1 + 1 = 10. The report divided by the 7 rows with a verdict. The timeout is an unsuccessful attempt. The unknown and the missing attempt are unresolved, so the record supports 6/10 to 8/10, with the three excluded rows named.

Q8

The counting gate reports three of three and stays green. The pinned gate reports three cases against four expected and blocks. Blocking is right because the deleted case was a declared check: without it the suite can no longer fail on what that case covered, and a smaller denominator is not a better result.

Q9

n = 5, c = 2. pass@1 = 2/5. pass@2 = 1 - C(3, 2)/C(5, 2) = 1 - 3/10 = 7/10. By pairs: there are ten pairs of attempts, and three of them (2-3, 2-5, 3-5) contain no pass.

Q10

The retries are not independent attempts: each saw the previous error, so this is one attempt with feedback, a different procedure with its own cost. And there are not three recorded independent attempts to sample from, so pass@3 cannot be computed. With one attempt per task, pass@2 is undefined, because k exceeds n. It is not zero and not low.

Q11

Observed agreement is 4/5. The labels are pass 1/5 and fail 4/5; the verdicts are pass 2/5 and fail 3/5. Expected agreement is 1/5 × 2/5 + 4/5 × 3/5 = 14/25. Kappa is (20/25 - 14/25) / (11/25) = 6/11. The disagreement is a false accept: the label is fail and the judge said pass. Report that row beside the coefficient.

Q12

Only that the revised rubric agrees with five cases it was revised against. Those cases are now calibration cases. Before claiming more, freeze the new rubric and run it once on fresh held-out cases, labelled first and including a case variant and an adversarial response.

Q13

(a) Code: an exact comparison decides it, the same way every time. (b) A person, or a model judge that has been calibrated against people's labels and tested on held-out cases; code cannot decide accuracy of free text. (c) People, working independently from the rubric; a judge cannot supply the labels it is checked against.

Q14

Run the scorer's own checks on the new version, re-score the recorded answers, and recompute the denominator and pass@k. The earned check on recorded changes is unaffected unless the scorer decides those tasks too. Until that is done, the earlier decision is about the old scorer and is not a statement about the current one; write "recheck" with the new revision.

Spaced retrieval and transfer

Retrieval 1. A review with no findings supports only that the model reported none for the context it was given. Check the changed requirement, the tests and the boundary inputs yourself, and record what the review did not cover.

Retrieval 2. A gate outside the agent's control: a controller runs the preserved tests on the actual candidate and requires a recorded passing result before the next stage. A PASS string written by the agent is not that result.

Transfer. The evidence supports using the scorer for exact yes or no answers to this status question at this revision. Untested variations include other statuses and other answer formats. JSON responses need a new contract that says which field holds the answer, a parser, and new cases for malformed, missing and extra-field input, all run before the format is accepted.