Week 5: Agent-ready codebases

Learning objectives

By the end of this week, you can:

locate a change point from a repository map and name the callers, rules and checks that constrain it (LO4);
ask an agent for a bounded proposal, then verify the proposal with independent tests, a diff and a local check (LO4, LO5);
score Structure, Documentation, Tests and Checks from observable anchors, name a disproving observation, and explain what evidence would change the score (LO4, LO5);
keep repository text and generated commands inside the approved safety boundary (LO7).

A repository is agent-ready when a fresh session can find the relevant code, learn the local contract, run a short check and report what it did not check. Readiness is a property of evidence at a named revision, not a feeling about how tidy the tree looks.

Minimal example

One task, two repositories

The task is deliberately small: replace a chain of status conditionals with a lookup while preserving whitespace, case and the unknown fallback. The public fixture contains the same function in three states:

before.py is the disordered starting point: the caller and contract are not written beside the function;
rejected.py is a tempting proposal that removes strip();
refactored.py is a structure change that keeps the contract.

The request is generated as text and inspected before it is copied into a candidate. The request is never executed as code. Verify the candidates from the root of your course checkout; the verifier reads the variant files that sit beside it, so it gives the same result from any directory:

python lectures/week-05/fixture/verify.py rejected
python lectures/week-05/fixture/verify.py refactored

Each command prints one line of JSON. The first exits with status 1 and reports the three padded inputs that fail:

{"cases": 10, "failures": [{"actual": "unknown", "expected": "passed", "input": " ok "}, {"actual": "unknown", "expected": "failed", "input": "\tfail\n"}, {"actual": "unknown", "expected": "passed", "input": " PASS "}], "if_statement_count": 0, "passed": 7, "scope": "course synthetic fixture; public feedback, not a grade", "variant": "rejected"}

The second exits with status 0:

{"cases": 10, "failures": [], "if_statement_count": 0, "passed": 10, "scope": "course synthetic fixture; public feedback, not a grade", "variant": "refactored"}

The result establishes those ten cases only. It does not prove behavior for every string or prove that a caller accepts the change.

Now run the same fixed cases against the starting point:

python lectures/week-05/fixture/verify.py before

{"cases": 10, "failures": [], "if_statement_count": 3, "passed": 10, "scope": "course synthetic fixture; public feedback, not a grade", "variant": "before"}

before and refactored both pass all ten cases; only if_statement_count differs (3 and 0). The cases cannot tell the two states apart, which is the point: behaviour is preserved, and what changed is how much a fresh session has to guess. In before.py a session must infer the contract from a chain of conditionals. In refactored.py the lookup table and the fallback are visible beside the change point. In a real project, a prepared repository would also name its entry point, contract and offline check in its context files; the agent still needs permission to read and edit, but the repository has removed avoidable guesses.

What the comparison can and cannot show

The fixture is one function, so it cannot show every kind of blocker. The rows below are illustrative: they are the kinds of observation you will record in the studio on a whole repository, not results of the commands above.

Observation	Supported claim	Missing evidence
The session found `status_label` in two file opens	The map points to this change point for this revision	Other behaviours may have different maps
The documented check exits zero on the refactored copy	These cases pass at this revision	Callers, permissions and unlisted inputs
The disordered copy has three test commands, two of which fail	Documentation is a blocker for a fresh session	Whether the implementation itself is correct
A hook rejects an unused import with exit 2	The hook can stop that class of change	Other checks and bypass paths

An agent may propose the score, but you choose it. For each dimension record the path or command, revision, observed result, the highest level whose anchors are all present, and one observation that would disprove the score.

Four observable readiness dimensions

Use levels 1–4. A level includes every lower-level anchor.

Structure

1: no map from a behaviour to its change point;
2: a map exists, but a checked entry is wrong or mixes unrelated responsibilities;
3: checked entries are accurate, the change point has one responsibility, and the probe reaches it in three file opens or fewer;
4: a deliberate boundary violation fails a test or check.

Evidence is a behaviour, the files opened in order, a path:line change point and, at level 4, the failing result.

Documentation

1: no context file or no runnable command;
2: commands exist but one fails as written, or no invariant names the check that protects it;
3: every listed command runs at the scored revision and at least three invariants name their checks;
4: an automated check runs the documented commands and fails when one goes stale.

Evidence is the context-file path, each command and exit code, and the invariant-to-check list.

Tests

1: no runnable test exercises the change point;
2: one run passes but a hidden input, nondeterminism or a deliberate break goes undetected;
3: two repeatable runs agree, stay within the chosen budget, use no hidden input, and a deliberate break fails;
4: failure messages identify the broken behaviour and mutation evidence accounts for surviving gaps.

Evidence is the two summaries, the hidden-input search, and the deliberate-break result.

Checks

1: no automatic gate;
2: a gate can be run by hand but is not required, or it ignores an unused import;
3: one offline command runs the required gates and exits non-zero for an unused import;
4: the gate runs before a commit is offered and a secret canary is stopped by the project scanner.

Evidence is the gate command, failing and passing output, and the point where the gate is required.

Hands-on exercise

Use the studio activity on your own project or contribution target repository. First freeze a baseline at a named commit. Give the agent only the files needed to answer one bounded question; inspect its proposal as untrusted text. Then run the repository's documented check and inspect the diff. If a command needs a credential, network access or an unknown install, stop and revise the boundary rather than running it.

Your first score is a four-row table with one evidence link and one disproving observation per row. Remove one blocker that is within your scope—for example, reconcile two test commands, add a map from a behaviour to its module, or make a check command deterministic. Run the same probe at the new commit and score again. A tidy directory without a changed observation does not raise a score.

Time-box the work: 15 minutes to choose the task and baseline, 25 minutes to score, 25 minutes to remove one blocker, 15 minutes to rerun the probes, and 10 minutes to write the decision. Ask for help at the first failed command; reserve the final ten minutes for revision.

Supported execution

Run the fixture commands on this page directly. Before running a project's own check, follow Tools and model access.

Verification

Run the public fixture before and after reading the candidate:

python lectures/week-05/fixture/verify.py rejected  # expected: exit 1, 7 of 10 passed
python lectures/week-05/fixture/verify.py refactored  # expected: exit 0, 10 of 10 passed

For your repository, run the exact command it documents, record its exit code and inspect the changed-file list:

git status --short
git diff --check
git diff --name-only

A passing check is evidence for the inputs it covered. A score changes only when the observed anchor changes at a named revision. If the same command passes because its tests were deleted, its scope was weakened, or the agent supplied a result without a command, keep the old score and record the finding.

Common mistakes

Mistake	How to detect it	Correction
Scoring from a polished README	A listed command fails or names a missing file	Run each command at the scored revision and lower the row until it passes
Treating an agent's score as evidence	No path, revision or output is attached	Ask for one observable anchor and one disproof before recording a number
Refactoring only for shorter code	The candidate passes happy paths but loses normalization or a caller contract	Add the distinguishing case, restore the invariant, and rerun the unchanged check
Calling contradictory docs a test failure	The same command is written three ways	Choose one bounded source of truth and verify the chosen command, leaving the conflict recorded
Running a generated command blindly	It asks for a key, network or an unpinned install	Stop, inspect the trust boundary, and run it only in a dedicated directory or container that holds no credentials; otherwise record the missing prerequisite
Raising a score after deleting tests	The test count or covered paths shrink	Compare the old and new test lists and keep the score unchanged

End the work record with one decision: accept, narrow, recheck or stop. Name the evidence that supports it and the observation that would change it.