Skip to content

Week 7 studio: Your project's tests

This is the week's second session: a studio on your project's tests. You apply the three ideas from the notes to code you wrote, in your project repository, and leave with one closed gap, one property and a reviewed agent diff.

Due this week

The project alpha and task backlog are due this week; the syllabus says what they contain.

Before you start

Format: individual work, or the two members of a project pair alternating who types. Materials: your project repository with the tests it has today, the one command that runs them, a text editor and terminal, and your coding agent for step 5 only. Output: one short record in your repository (for example docs/tests-week7.md) with the six entries named in the steps below.

Work in your project repository, on a branch, starting from a clean tree. This command should print nothing:

git status --short

If it prints file names, commit or set aside that work first. Every step below ends by returning to a clean tree or a commit, so that a planted fault can never be mistaken for your real code.

If your project has no test yet, write one test with an expected value for the behaviour you pick in step 1, see it pass, and continue. That test is your suite for today.

Studio sequence

1. Pick one behaviour and write its contract (10 minutes)

Choose one function or module on your alpha path: something your end-to-end path actually calls, with inputs you can construct in a test. Write its contract in the form the notes use, in four lines or fewer: what it requires, what it accepts, what it ensures, what it rejects and how. Take each line from your technical specification or your own decision, not from reading what the code currently does. If the specification is silent, say so in the record and decide; that is a specification change, and you own it.

Record entry 1: the contract and the exact test command.

2. Plant three faults and find a survivor (25 minutes)

Without a model, make three one-line mutants of that code, one at a time. Useful shapes:

  • move a boundary (< to <=, len(x) == 1 to len(x) >= 1, an index by one);
  • replace a returned value with a constant, or return early;
  • delete a check: a validation, a guard, an error branch.

For each mutant: edit the one line, run your test command, write down killed (a test failed because of your change: a failed assertion, or an exception your change makes the code raise) or survived (everything passed), and restore the file. Check that the fault is gone before planting the next one:

git diff --stat

It should print nothing after each restore. Deleting a guard often makes the code raise KeyError, IndexError or TypeError in a test; that test noticed the fault, so the mutant is killed. A run that fails for a reason your change did not cause in the behaviour under test, such as a syntax error in the edited line or a missing import, is an invalid mutant: fix it and do not count it. If a run hangs, record a timeout separately and count it as killed only after you have confirmed your change causes it.

If your project already runs a mutation tool, you may use its report for this step. Pick three of its mutants in the chosen code and reproduce each by hand once, so that you know what the report's words mean in your repository.

If all three are killed, your tests are doing their job on these faults. Plant two more in a place the tests look thinner, such as an error path, before concluding anything.

Record entry 2: a table of mutant, changed line, killed or survived, and the denominator in words ("3 hand-written mutants of parse_config, chosen by me").

3. Kill one survivor (15 minutes)

Take one survivor and say which observation is missing: which input, and which expected result, would have told the mutant from the original. Write that test. Then show both halves:

  1. with the fault planted, the new test fails because of the fault;
  2. with the fault removed, the whole suite passes.

A test that passes in both states has not closed the gap; change the test, not the mutant. If you believe a survivor is equivalent, write the contract reason; "I could not think of a test" is not one. Commit the new test on its own, so the diff shows only test code.

Record entry 3: the survivor, the new test's name, and the two run results.

4. State one property and try to break it (20 minutes)

Write one property for the same behaviour as a sentence beginning "for every …". Good sources are a round trip (parse then format), an invariant (the output is always sorted, never longer than the input, always within a range), or agreement with a simple, slow reference. Name where the expected side comes from; it must not be the function under test.

Check it on generated inputs. Use a property-testing library for your language if your project has one or you can add one in a few minutes; otherwise write a seeded loop that builds 100 inputs, as the course fixture does. Include inputs the contract rejects.

If the property fails, shrink before you debug: remove or simplify parts of the failing input, keeping each change only while the same failure still occurs, until no single change keeps it failing. Save the shrunk input as an ordinary test with a fixed input and decide whether the code or the contract is wrong. If the property holds, record how many inputs, from what generator, and one family of inputs it did not cover.

Record entry 4: the property sentence, the source of the expected side, the generator and count, and either the shrunk counterexample with its failure or the stated limit.

5. Give your agent a red test and review the diff (20 minutes)

Now the agent. Plant one of your killed mutants again and commit it on a scratch branch with a neutral message, so the fault is in the code and a test is red. Start a fresh agent session and give it only this request: the test command fails, make it pass, and do not commit. Do not mention the planted line. Let it finish, then review as the notes do, before reading the agent's summary:

git diff --stat
  1. Which files changed? A change to test files and none to the code under test is the first signal.
  2. Read every changed test line and map it to a line of your contract. Removed inputs, looser comparisons, broader exception handlers, new skip markers and changed expected values each need a contract reason.
  3. Check the fault is gone: the planted line is restored or correctly rewritten, and the test you wrote in step 3 still exists and still passes.
  4. Compare the agent's summary with the diff. Note any sentence the diff does not support.

Whatever happened is the result. If the agent repaired the code and left the tests alone, record that and the evidence; if it edited a test, record the lines and your decision. Either way this is one run on one planted fault in your repository: a process exercise about how you review, not a finding about what a model does in general. Then delete the scratch branch and return to your working branch with a clean tree.

Record entry 5: the request you gave, the files changed, each changed test line with its disposition, and one sentence comparing the summary with the diff. Reuse your existing AI-use note; no chat export is needed.

6. Turn what is left into backlog items (10 minutes)

Every survivor you did not kill, every family of inputs your property did not cover and every test the agent touched is an item for your task backlog. Write each as a task a person or an agent could pick up: the behaviour, the missing observation, and the check that would close it. Add one line to your project's instruction file if step 5 showed you need it, for example that test files are changed only together with a stated requirement change.

Record entry 6: the backlog items added, with links.

What a finished record lets a reader do

A reader with your repository and your record can replant each mutant and get the same killed or survived result, rerun the property with the same generator, replay the shrunk input, and see which lines the agent changed. If one of those needs you in the room, the entry is not finished.

If you get stuck

  • No survivor after five mutants. Mutate an error path or a boundary your tests do not name, or pick a second function. If your suite kills everything you plant there too, say so and spend the time on the property.
  • The property is hard to state. Start with the weakest true statement, such as "never raises anything other than the documented error", and run it on generated inputs. It is the property the notes' parser fails.
  • The tests take minutes to run. Narrow the command to the tests for the chosen code for steps 2 and 3, and run the full suite once at the end.
  • Agent access fails. Record the failure, skip the run in step 5, and review the course-written patch from the notes instead; say in your record that you did.

Participation and accessibility

All required material is plain text and can be completed with a keyboard. An approved written or asynchronous exchange can replace working beside a partner while keeping the same individual record. Keep credentials, private data and full chat logs out of the record.