Skip to content

Tests that notice a fault

Week 7 · A green run is evidence only for the faults the tests could have noticed

Three questions for a green run

  1. Which planted fault would these tests miss?
  2. Which input did nobody think of?
  3. Did the code change, or did the test?

All examples today are course-written and rerunnable.

The contract comes first

accepts:  exactly one of "0" ... "9"
ensures:  the corresponding integer
rejects:  every other string with ValueError

Every expected value comes from these lines.

The candidate

def parse_level(text: str) -> int:
    if text[0] in "0123456789":
        return int(text[0])
    raise ValueError("Invalid level")

Predict two rejected inputs it mishandles.

A suite that cannot fail

result = parser("3")
assert result == result
assert parser
assert result or True

demo.py weak exits 0 on the faulty candidate.

A suite with expected values

parser("3") == 3, and "", "12", "x" must raise ValueError.

demo.py unit exits 1: "failure": "IndexError".

A known input, an expected result, a different observed result.

Generate, then verify

Proposal: check len(text) == 1 before reading the digit.

Same tests, unchanged: demo.py unit --implementation repaired exits 0.

Red before, green after, tests untouched.

Green raises the next question

Which faults would this suite have missed?

Plant one and see.

A mutant

A copy of correct code with one small, deliberate fault.

  • A test fails: the mutant is killed.
  • Every test passes: the mutant survived.

Three mutants of the repaired parser

Mutant Fault
all_zero every accepted string returns 0
nine_to_zero only "9" returns 0
accept_trailing "12" is accepted as 1

An agent's first suite

assert parser("3") == 3

Plus one rejected input: parser("x") raises ValueError.

Reads well. Passes on the repair.

It kills 1 of 3

demo.py survivors, first line:

"all_zero": "killed", "nine_to_zero": "survived",
"accept_trailing": "survived"

A survivor is a gap in the tests

nine_to_zero survives: no test looks at "9".

accept_trailing survives: no rejected input starts with a digit.

Each survivor names the missing observation.

Write the test the survivor asks for

assert parser("9") == 9

Second line of the output: nine_to_zero is killed, 2 of 3.

Then the next one

parser("12") must raise ValueError.

Third line: accept_trailing is killed, 3 of 3.

A new test must do both

Fail on the mutant. Pass on the original.

A test that passes on both closed nothing.

Keep the count honest

  • The original passes first.
  • State the denominator: which mutants, chosen how.
  • A kill is a test failing because of the change; an import error or broken run is not.
  • An equivalent mutant is excluded with a written reason.

Coverage would not have shown this

The first suite already runs every line of the repaired parser.

Coverage says a line ran. A mutant asks whether changing it was noticed.

From examples to a property

An example checks one input you thought of.

A property states what holds for every input in a family, and is checked on generated inputs.

A property that is too narrow

For every n in 0..9: parser(str(n)) is the integer n.

demo.py property passes on the faulty candidate.

It never leaves the ten valid strings.

The property the contract states

For every string s: a digit string returns its integer; anything else raises ValueError.

The expected side is a table of ten strings, not the parser.

100 generated strings, two failures

demo.py shrink --seed 19 exits 1 with two counterexamples:

Failure Found Shrunk
unexpected_IndexError "" ""
accepted_invalid_input "1836" "00"

"1836" fails. Why?

Too long? The digit 8? A value above 9?

The input that found the bug rarely explains it.

Shrinking

Try a simpler input. Keep it only if the same failure still occurs.

The shrink path from "1836" to "00" The generated input "1836" is reduced in four kept steps: delete a character to get "836", delete again to get "36", replace a character with 0 to get "06", replace again to get "00". Each kept input still fails as accepted invalid input. Deleting once more gives "0", which is valid and correctly parsed, so shrinking stops at "00". "1836"found "836"delete "36"delete "06"replace with 0 "00"replace with 0 "0"valid: stop

In words: delete, delete, replace with 0, replace with 0; one more deletion gives the valid "0".

The shrunk counterexample is a diagnosis

"00": two ordinary characters, accepted.

The length is never checked.

Keep the input, not only the seed

demo.py replay --text 00 exits 1 on the candidate, 0 on the repair.

A seed reproduces a failure only while the generator stays the same.

What the course helper is

A seeded generator over one alphabet and a greedy shrinker, a few dozen lines.

No strategies, no failure database, no general minimality guarantee.

Use a property-testing library in your project.

A clean run has a boundary

Zero counterexamples in 100 strings: those 100 strings pass.

Longer inputs and other kinds of defect are untested.

Two ways to make red green

Change the code so it meets the contract.

Or change the test so it stops asking.

"Make the tests pass" rewards the shorter one.

The hand-off

Fixed the failing unit check. The empty string and multi-digit inputs are outside the supported range, so I removed them from the test. All checks pass.

Course-written, in the style of an agent's summary. demo.py unit now exits 0.

First look: which files changed?

 lectures/week-07/fixture/demo.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

The test was "fixed". The code under test is not in the list.

Then read the test diff

-    for text in ("", "12", "x"):
+    for text in ("x",):

Two contract obligations deleted. No behaviour changed.

Two runs confirm it

demo.py replay --text 12: still accepted_invalid_input.

demo.py mutation: the strong suite drops from 3 of 3 to 2 of 3.

accept_trailing is a survivor again.

Weakening has many shapes

  • a deleted assertion or case
  • == loosened to in or "is not None"
  • except ValueError widened to except Exception
  • an expected value copied from the output
  • a new skip marker; a lowered threshold

When changing a test is right

The requirement changed, or the test was wrong.

The reason points at the contract, and a person owns it.

"The tests now pass" is not a reason.

When the agent proposes the tests

Read its oracle. Run it on a known-bad and a repaired version.

It never picks its own score or declares a survivor equivalent.

A target that builds, a run that crashes and a validated input are separate results (tools/oss-fuzz-gen; as_of: 2026-09-05; Source 1; Source 2; Source 3) (tools/fuzzingbrain; as_of: 2026-09-05; Source 1).

Studio: your project's tests

One behaviour on your alpha path:

plant three faults → kill one survivor → one property → review an agent's diff → backlog.

Project alpha and task backlog are due this week.

Before you leave

Name one fault your tests would miss today.

Name the test that would notice it.