Skip to content

Week 7: Tests that notice a fault

The parser, its tests, the mutants and the "agent" patch on this page are course-written teaching examples. Nothing here is a recorded model run or an industry statistic; every count comes from the commands shown, which you can rerun.

Learning objectives

By the end of this week, you can:

  • show that a test suite misses a fault by naming a surviving mutant, then write the test that kills it and explain what the count does and does not say (LO5);
  • state a property that holds for a whole family of inputs, check it on generated inputs, and reduce a failing input to a shrunk counterexample you keep as a regression (LO5); and
  • recognise a change that makes a failing test pass by weakening the test, and say in your own hand-off what was changed, why, and what you verified (LO5, LO9).

Week 6 asked for evidence behind every review comment. This week turns the same question on the tests: a green run is evidence only for the faults those tests could have noticed.

Minimal example

Run every command on this page from the course checkout, in the repository root, with plain python. The fixture uses only the standard library, calls no model and needs no key. When you use a coding agent on your own code this week, follow Tools and model access.

The contract and the candidate

A configuration field holds a level. The contract is written before any code or test:

requires: text is a string
accepts:  exactly one of "0", "1", ..., "9"
ensures:  an integer with the corresponding value, with no state changes
rejects:  every other string by raising ValueError

So "3" maps to 3, and "12", "", a space and a non-ASCII digit are all rejected. The wording of the error message is deliberately unspecified. This is the oracle: every expected value below comes from these four lines, never from what the code happens to do.

The candidate in lectures/week-07/fixture/candidate.py was written to look reasonable and be wrong:

def parse_level(text: str) -> int:
    if text[0] in "0123456789":
        return int(text[0])
    raise ValueError("Invalid level")

Before running anything, predict two inputs the contract rejects and this code does not handle as required. Keep your prediction; the next sections check it.

A suite that passes and notices nothing

Here is a test in the style an agent often produces when asked to "add tests". It is the weak_checks function in lectures/week-07/fixture/demo.py:

result = parser("3")
assert result == result
assert parser
assert result or True

Run it against the faulty candidate:

python lectures/week-07/fixture/demo.py weak

{"exercise": "course-synthetic", "implementation": "candidate", "mode": "weak", "target_checks_passed": true}

Exit status 0, on code that violates its contract. Each assertion has a blind spot: result == result is true for any value, assert parser checks that a function exists, and result or True is always true. None of them compares the result with the contract.

A suite that states expected values behaves differently. unit_checks asserts parser("3") == 3 and requires ValueError for "", "12" and "x":

python lectures/week-07/fixture/demo.py unit

{"exercise": "course-synthetic", "failure": "IndexError", "implementation": "candidate", "mode": "unit", "target_checks_passed": false}

Exit status 1. The empty string reaches text[0] and raises IndexError, where the contract requires ValueError. That is a demonstrated defect: a known input, an expected result from the contract, and a different observed result.

Generate. A repair proposal checks the length before indexing. It is the repaired function in the same file:

def repaired(text: str) -> int:
    if len(text) == 1 and text in DIGITS:
        return int(text)
    raise ValueError("Invalid level")

Verify. Run the same, unchanged checks against the proposal:

python lectures/week-07/fixture/demo.py unit --implementation repaired

{"exercise": "course-synthetic", "implementation": "repaired", "mode": "unit", "target_checks_passed": true}

Red on the candidate, green on the repair, with the tests untouched in between. The order matters: you fix the intended behaviour and review the tests first, a person or an agent proposes code, and an actual run judges the proposal. The rest of the page asks a harder question about that green result: which faults would these tests have missed?

A surviving mutant is a gap in the tests

A mutant is a copy of a correct implementation with one small, deliberate fault. Run the suite against the mutant. If a test fails, the mutant is killed: the suite noticed the fault. If every test passes, the mutant survived: the suite cannot see that change, and it could not see a real bug of the same shape either.

The fixture has three hand-written mutants of repaired:

Mutant The planted fault
all_zero every accepted string returns 0
nine_to_zero only "9" returns 0; the other nine values are correct
accept_trailing a digit followed by more characters is accepted, so "12" returns 1

Generate. Suppose an agent proposes a first suite, example_checks: one accepted value and one rejected string.

def example_checks(parser: Parser) -> None:
    assert parser("3") == 3
    try:
        parser("x")
    except ValueError:
        return
    raise AssertionError("Expected ValueError for 'x'")

It reads well and passes on the repaired parser. Verify it by asking what it would notice. The survivors mode first requires the suite to pass on the unmutated original, then runs it against each mutant, then adds one test at a time:

python lectures/week-07/fixture/demo.py survivors

{"suite": "example_checks", "killed": 1, "total": 3, "mutants": {"all_zero": "killed", "nine_to_zero": "survived", "accept_trailing": "survived"}}
{"suite": "+ upper_boundary_check", "killed": 2, "total": 3, "mutants": {"all_zero": "killed", "nine_to_zero": "killed", "accept_trailing": "survived"}}
{"suite": "+ trailing_check", "killed": 3, "total": 3, "mutants": {"all_zero": "killed", "nine_to_zero": "killed", "accept_trailing": "killed"}}
{"exercise": "course-synthetic", "implementation": "repaired original plus three fixed mutants", "mode": "survivors", "target_checks_passed": true}

Read the first line as a list of gaps. example_checks kills 1 of 3. Two faults survive:

  • nine_to_zero survives because no test looks at "9". The suite checks one value in the middle of the range and nothing at its upper edge.
  • accept_trailing survives because the only rejected input, "x", has no leading digit. Nothing checks a string that starts like a valid level and continues.

Each survivor names the missing observation, so each one tells you what test to write. The two tests added in the second and third lines are exactly those:

def upper_boundary_check(parser: Parser) -> None:
    assert parser("9") == 9


def trailing_check(parser: Parser) -> None:
    try:
        parser("12")
    except ValueError:
        return
    raise AssertionError("Expected ValueError for '12'")

After the first, nine_to_zero is killed and accept_trailing still survives. After the second, all three are killed. A new test earns its place when it fails on the mutant and passes on the original; a test that passes on both has not closed the gap.

Four rules keep a mutation count honest:

  1. The original must pass first. A suite that fails on the unmutated code "kills" everything and has shown nothing. The fixture checks this before it counts.
  2. State the denominator. "3 of 3" means these three hand-written mutants and no others. A mutation tool for your language applies many operators to your code and produces a much larger set; report the tool, its configuration, the revision and the total alongside the count.
  3. A kill is a test failing because of the mutant's change. That is a failed assertion, or an exception the change makes the code under test raise: the unit check above caught the candidate through an IndexError, and that counts. A failure that would happen without the change does not: a syntax or import error, a missing dependency, a broken harness. Report a timeout separately, and count it only after you have confirmed that the mutant causes it. The fixture counts only AssertionError, which is a simplification that holds because none of its three mutants raises anything else; it is not the general rule.
  4. An equivalent mutant leaves the denominator with a written reason. A mutant that changes only the wording of the ValueError message cannot be killed by any test of this contract, because the contract does not specify the message. Exclude it and say why. Never drop a survivor that the contract can distinguish just to improve the number.

Line coverage would not have found these gaps: example_checks already executes every line of repaired. Coverage records which lines ran. A mutation result records whether a change to those lines was noticed. Neither proves the code correct.

A property and its shrunk counterexample

Examples check the inputs you thought of. A property is a statement that must hold for every input in a family, checked on many generated inputs. The first property people write for this parser is a round trip over the valid values, here property_checks:

for number in range(10):
    result = parser(str(number))
    assert type(result) is int
    assert 0 <= result <= 9
    assert str(result) == str(number)

It kills all_zero and nine_to_zero in one statement. Now run it on the faulty candidate:

python lectures/week-07/fixture/demo.py property

{"exercise": "course-synthetic", "implementation": "candidate", "mode": "property", "target_checks_passed": true}

It passes, because the candidate is correct on every valid input. A property is only as wide as the inputs it quantifies over, and this one never leaves the ten accepted strings. The contract's last line is about every other string, so the property that matches the contract is:

For every string s: if s is one of the ten digit strings, the parser returns the corresponding integer; otherwise it raises ValueError.

The expected side comes from an explicit table of the ten accepted strings, not from the parser under test. The shrink mode checks this property on 100 generated strings of zero to four characters, drawn from digits, x, space, -, a newline and one non-ASCII digit:

python lectures/week-07/fixture/demo.py shrink --seed 19

{"reason": "unexpected_IndexError", "found": "", "shrunk": "", "path": [""]}
{"reason": "accepted_invalid_input", "found": "1836", "shrunk": "00", "path": ["1836", "836", "36", "06", "00"]}
{"cases": 100, "counterexamples": 2, "exercise": "course-synthetic", "implementation": "candidate", "mode": "shrink", "seed": 19, "target_checks_passed": false}

Exit status 1, with two different failures. The second was first found as "1836". That input fails, but it does not tell you why: is the problem the length, the digit 8, the value being above 9? Shrinking answers by repeatedly trying a simpler input and keeping it whenever the same failure still occurs:

Step Input What was tried Still accepted_invalid_input?
found "1836" generated yes
1 "836" delete a character yes
2 "36" delete a character yes
3 "06" replace a character with 0 yes
4 "00" replace a character with 0 yes
stop "0" delete a character no: "0" is valid and correctly parsed

The shrunk counterexample "00" reads as a diagnosis: two characters, nothing special about either, and the parser accepts it. The fault is that the length is never checked. The first failure, "", was already as small as a string can be, so its path has one entry. Two shrunk inputs, two distinct defects, and both match the prediction you made from the candidate's code.

Keep each shrunk input as a permanent regression, then verify the repair against it and against the whole property:

python lectures/week-07/fixture/demo.py replay --text 00
python lectures/week-07/fixture/demo.py replay --text 00 --implementation repaired
python lectures/week-07/fixture/demo.py shrink --seed 19 --implementation repaired

{"exercise": "course-synthetic", "implementation": "candidate", "mismatch": "accepted_invalid_input", "mode": "replay", "target_checks_passed": false, "text": "00"}
{"exercise": "course-synthetic", "implementation": "repaired", "mismatch": null, "mode": "replay", "target_checks_passed": true, "text": "00"}
{"cases": 100, "counterexamples": 0, "exercise": "course-synthetic", "implementation": "repaired", "mode": "shrink", "seed": 19, "target_checks_passed": true}

The first command exits 1 and the other two exit 0. Save the input itself, not only the seed: a seed reproduces a failure only while the generator and the runtime stay the same.

What this helper does and does not do. The fixture's generator and shrinker are a few dozen lines of standard-library Python, written so you can read every step. The generator draws from one fixed alphabet with a caller-supplied seed. The shrinker is greedy: it deletes one character or replaces one with 0, keeps the first change that preserves the failure category, and stops when no single change does. It reports the first failure of each category and nothing else. It has no input strategies, no composition of generators, no database of past failures, and no guarantee that the result is the smallest possible input in general; here "00" is the smallest because one character can never trigger this failure. The older property mode is narrower still: a fixed enumeration of ten values with no generation and no shrinking. A property-testing library for your project's language does this work properly, with typed strategies and a much more thorough shrinker; use one in your project. The idea carries over unchanged: state the property from the contract, keep the expected side independent of the code, and keep the shrunk input.

A clean run has limits too. Zero counterexamples in 100 generated strings shows that those 100 strings pass. It does not cover longer strings, other alphabets or a defect of a different kind.

An agent that weakens a test to pass it

There are two ways to turn a red test green: change the code so that it meets the contract, or change the test so that it stops asking. An agent told to "make the tests pass" is being asked for a green run, and the second way is often shorter.

Start from the red result above: unit fails on the candidate with IndexError. Generate. The following patch is course-written, in the style of an agent's "fix". It ships as lectures/week-07/fixture/weakened-test.patch.txt, with this hand-off message:

Fixed the failing unit check. The empty string and multi-digit inputs are outside the supported range, so I removed them from the test. All checks pass.

Apply it and rerun the check that was red:

git apply lectures/week-07/fixture/weakened-test.patch.txt
python lectures/week-07/fixture/demo.py unit

{"exercise": "course-synthetic", "implementation": "candidate", "mode": "unit", "target_checks_passed": true}

Exit status 0. The summary is true as far as it goes: all checks pass. Verify as a reviewer would, starting with which files changed:

git diff --stat

 lectures/week-07/fixture/demo.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

The failing test was "fixed" and candidate.py, the code under test, is not in the list. That alone is enough to stop and read the diff:

git diff

diff --git a/lectures/week-07/fixture/demo.py b/lectures/week-07/fixture/demo.py
--- a/lectures/week-07/fixture/demo.py
+++ b/lectures/week-07/fixture/demo.py
@@ -55,7 +55,7 @@ def trailing_check(parser: Parser) -> None:
 def unit_checks(parser: Parser) -> None:
     """Use explicit expected values and reject the wrong exception type."""
     assert parser("3") == 3
-    for text in ("", "12", "x"):
+    for text in ("x",):
         try:
             parser(text)
         except ValueError:

(Your output also has an index line with two object IDs after the first line; it is left out here.) Map each removed input to the contract. "" and "12" are both covered by "rejects every other string by raising ValueError". The hand-off calls them "outside the supported range", but the contract says what must happen to exactly those inputs. The patch deleted two contract obligations and changed no behaviour.

Two runs confirm it without relying on reading alone. First, the defect is still there:

python lectures/week-07/fixture/demo.py replay --text 12

{"exercise": "course-synthetic", "implementation": "candidate", "mismatch": "accepted_invalid_input", "mode": "replay", "target_checks_passed": false, "text": "12"}

Second, the suite has lost the ability to notice a fault it used to notice. mutation compares weak_checks with strong_checks, which is unit_checks plus the round-trip property. Before the patch, the strong suite killed 3 of 3. Now:

python lectures/week-07/fixture/demo.py mutation

{"exercise": "course-synthetic", "implementation": "repaired original plus three fixed mutants", "mode": "mutation", "strong": {"killed": 2, "mutants": {"accept_trailing": "survived", "all_zero": "killed", "nine_to_zero": "killed"}, "total": 3}, "target_checks_passed": true, "weak": {"killed": 0, "mutants": {"accept_trailing": "survived", "all_zero": "survived", "nine_to_zero": "survived"}, "total": 3}}

strong is down to 2 of 3 and accept_trailing is a survivor again. This is the first topic of the week used as a review tool: a patch that turns a killed mutant back into a survivor has weakened the tests, whatever its message says.

Undo the patch and confirm the checkout is clean again (the second command prints nothing):

git apply -R lectures/week-07/fixture/weakened-test.patch.txt
git status --short lectures/week-07/fixture

Removing inputs is one form. The same move appears as:

Form What the diff shows
Deleted or commented-out assertion or case fewer assert lines, a shorter input list
Loosened comparison == becomes in, >=, "is not None" or a substring match
Broadened exception except ValueError becomes except Exception
Expected value copied from the output the expected literal changes to whatever the code returned
Skip or expected-failure marker a new skip, xfail or "flaky" annotation on the red test
Mocked unit under test the function being tested is replaced by a stub inside its own test
Narrowed property fewer generated cases, a smaller alphabet, a filter that drops the failing input
Edited gate a lower threshold, a removed check, a mutant relabelled "equivalent" without a reason

Sometimes changing a test is right: the requirement changed, or the test was wrong. The difference is a reason that points at the contract and a person who owns the decision. "The tests now pass" is not a reason.

When an agent proposes the tests

A coding agent can draft a property, a list of mutants or an input generator quickly, and this week you will ask yours to. Treat what comes back as claims and candidate code:

  • read the oracle first: an expected value computed by calling the code under test agrees with any bug in it;
  • run the proposed tests on a known-bad version and on a repaired one, as this page did with candidate and repaired; a harness that reports nothing on the known-bad version is broken, however it reads; and
  • never let the proposal choose its own score, edit the tool that scores it, or decide that a survivor is equivalent.

The same separation appears in tools that generate fuzzing harnesses with models: whether a generated target builds, whether it crashes at run time and how much coverage it reaches are reported as separate results (tools/oss-fuzz-gen; as_of: 2026-09-05; Source 1; Source 2; Source 3), and a validated triggering input is kept apart from a candidate patch (tools/fuzzingbrain; as_of: 2026-09-05; Source 1). Those cards describe the tools' own reporting; they say nothing about this fixture.

Hands-on exercise

The second session is a studio on your project's tests. You pick one behaviour on your alpha path, plant small faults by hand to find a surviving mutant, kill it with a new test, state one property and keep its smallest failing input if it has one, and then give your agent a red test and review its diff for weakening. Survivors you do not close in the session become items in your task backlog.

Before the studio, repeat the example on the fixture so that the three results are yours:

  1. Run survivors and, before reading the output, write down which mutants example_checks should kill.
  2. Run shrink with --seed 7 instead of --seed 19. The inputs first found differ. Check whether the shrunk inputs do.
  3. Apply the patch, find the weakening in the diff without looking at this page, and reverse the patch.

Verification

The fixture has its own regression checks, which bind every count on this page:

python lectures/week-07/fixture/verify_demo.py

........
----------------------------------------------------------------------
Ran 8 tests in 0.003s

OK

The time on the third line varies. Run it once with the weakened-test patch applied and it fails two of the eight checks, including 2 != 3 for the mutation count: the lesson's own checks notice the weakening.

For your own work this week, a result is verified when another reader can reproduce it from your record. That record holds, for each claim:

Claim What makes it checkable
"This mutant survived" the one-line fault, the revision, the test command, and a passing run with the fault in place
"This test kills it" the test fails with the fault in place and passes without it; both runs recorded
"Mutation result: k of n" the tool or the hand-written list, how the n were chosen, and each excluded mutant with its reason
"This property holds" the property in one sentence, where the expected side comes from, how many inputs and from what generator
"Smallest failing input" the input itself saved as a regression, and the failure it reproduces
"The agent's fix is a real fix" the files changed, every changed test line mapped to a requirement, and the earlier tests still passing on the new code

One failure case to keep in mind: a mutation run where every mutant is "killed" in a few milliseconds by an import error. Check that the original passes, and read why each mutant died, before reporting the number.

Common mistakes

Mistake How to notice it Corrective action
Taking expected values from the code under test a wrong result equals the same wrong result, and the test passes write the contract first; give each expected value a source outside the implementation
Reading a high kill count as correctness the count has no stated set of mutants, or the set is three hand-picked ones report the denominator and how it was chosen; say what kinds of fault were never planted
Counting a broken run as a kill the mutant "dies" on an import error, a missing dependency or an unexplained timeout require the original to pass; count a failure only when the mutant's change causes it; list the rest separately
Deleting a survivor to improve the number the excluded mutant has no written reason, or the contract can tell it apart keep it as a survivor until a test kills it or the contract shows it is equivalent
Writing a property only over valid inputs the round trip passes while "" and "00" still fail quantify over the inputs the contract talks about, including the rejected ones
Keeping the seed and discarding the input the failure disappears after a generator or runtime change save the shrunk input as an ordinary regression test
Shrinking to a different failure the smaller input fails for another reason and the original defect is lost accept a smaller input only when the same failure category holds
Accepting "all tests pass" from an agent the diff touches tests and not the code under test, or removes assertions read the test diff line by line against the requirement; rerun the earlier tests and the mutants
Asking the agent only for green it is rewarded for the shortest route to a passing run ask for the cause first, forbid test edits without a stated requirement change, and review the diff yourself

Transfer to a new situation

A teammate's property says: for generated strings of one to six digits, the parsed value is never negative. It passes on 1,000 inputs. Which of this page's two defects can it detect, which contract clause does it never exercise, and what single generated input would you add to its family first? Write your answer before comparing with a partner.