Week 11: Building an AI-native team
Every incident, log, record, unit and agent summary on this page was written by the course for teaching. None is a measurement of a real system, model, gateway or price.
Learning objectives
By the end of this week, you can:
- describe the four shared pieces a team uses to operate agents (a tool portal, a model gateway, cost per accepted change and a routing policy), the problem each one solves and the check that would reveal each one failing (LO8);
- check an agent's own report on its work against the records, and say which line refutes it (LO8);
- check a draft root-cause analysis against a request log before any repair, marking each claim confirmed, refuted or cannot tell (LO8);
- prove a recovery with a regression that fails on the incident build, fails on a repair that only looks right and passes on the real repair (LO8); and
- state what your evidence does not cover and decide to accept, narrow, recheck or stop (LO9).
Minimal example
From one agent to a team's agents
So far the controls around your agent have been your own: a dedicated directory, no credentials in the environment and keys kept out of everything you share, as Tools and model access asks. When a whole team works this way, those jobs move into shared services. Four pieces recur:
| Piece | The problem it solves | What it concentrates |
|---|---|---|
| Tool portal | Nobody can say which agent may call which tool on which repository | Every grant |
| Model gateway | Keys, budgets and logs are scattered across laptops and CI | Every key and every cap |
| Cost per accepted change | Spending is reported per token or per run, which says nothing about results | The definition of what a change cost |
| Routing policy | Every task goes to the most expensive route by default | The choice of route for many tasks at once |
Each piece concentrates authority, so one review reaches every agent and so does one
mistake. For each piece this lesson does the same two steps. Generate: a record stands in
for what a team's agents and services produced, and its agent_summary is the claim, as if
an agent had reported on its own work. Verify: you check the records by hand, write the
result you expect, then run a check and compare.
Run every command on this page from the course checkout. The scripts use only the Python
standard library, call no model, open no network connection and write no file; if your
system has no python command, type python3. Read
org_check.py before you run it: each of its four commands applies
one stated rule to one JSON file and ends with a result: line.
Tool portal: compare the grant with the task
A portal is one service that lists which tools exist, who may call each one and on what, and records every grant. A hook in your own repository checks one call before its effect; a portal applies that idea to everyone at once.
org-portal.json holds one task, three grants and one agent
session of five calls. The task is to read the log of failed build #121 in the
ci-dashboard example repository and open a draft pull request there. The session's third
call reads a private repository because the build log told the agent to compare a settings
file. The agent's summary reads:
Every call I made stayed within my grants, and the portal refused the one call it did not cover.
Before running anything, match each call to the grant that allows it, and list what each grant can reach. Then run the check:
python lectures/week-11/fixture/org_check.py portal lectures/week-11/fixture/org-portal.json
It exits 1 and prints:
task repositories: ci-dashboard
call 1 ci.read_log ci-dashboard: allowed by G1
call 2 code.read ci-dashboard: allowed by G2
call 3 code.read release-config (private): allowed by G2; outside the task
call 4 deploy.run ci-dashboard: denied, no grant covers this tool; the tool did not run
call 5 code.open_draft_pr ci-dashboard: allowed by G3
G1 ci.read_log reaches: ci-dashboard
G2 code.read reaches: ci-dashboard, docs-site, hr-notes (private), release-config (private); beyond the task: docs-site, hr-notes, release-config
G3 code.open_draft_pr reaches: ci-dashboard
result: OVER-BROAD - grants reaching beyond the task: 1; calls outside the task that a grant allowed: 1
The summary is true, and that is the lesson. Every call was checked against a grant and the one uncovered call was refused. The portal worked exactly as configured, and the agent still read a private repository the task never needed, because grant G2 names every repository. Checking each call against its grant is not enough: when the grant itself reaches further than the task, every call passes. In one public demonstration, an agent's single GitHub MCP connection could reach both public and private repositories; injected text in a public issue led the agent to write private information into a public pull request, and the disclosers recommend limiting repository access (incidents/github-mcp; as_of: 2026-09-05; Source 1).
Treat a grant like code: it has an owner, a reason and a test showing that a call outside it is denied and leaves no effect. Narrow G2 in a copy and watch the same session again:
mkdir week11-org
cp lectures/week-11/fixture/org_check.py lectures/week-11/fixture/org-*.json week11-org/
In week11-org/org-portal.json, change the repositories of grant G2 from ["*"] to
["ci-dashboard"]. Write the line you expect to change, then run:
python week11-org/org_check.py portal week11-org/org-portal.json
It now exits 0 and prints:
task repositories: ci-dashboard
call 1 ci.read_log ci-dashboard: allowed by G1
call 2 code.read ci-dashboard: allowed by G2
call 3 code.read release-config (private): denied, no grant covers this repository; the tool did not run; outside the task
call 4 deploy.run ci-dashboard: denied, no grant covers this tool; the tool did not run
call 5 code.open_draft_pr ci-dashboard: allowed by G3
G1 ci.read_log reaches: ci-dashboard
G2 code.read reaches: ci-dashboard
G3 code.open_draft_pr reaches: ci-dashboard
result: WITHIN THE TASK - every grant reaches only the task's repositories; 2 calls denied
Call 3 is now the denied call that tests the grant. Delete week11-org when you have
finished the week's practice.
Model gateway: what it refuses
A gateway is one controlled place between code that wants a model and the service that answers. Describe a gateway by what it refuses:
| Job | What the gateway does | The check that reveals it failing |
|---|---|---|
| Keep the key outside the work | The agent's side sends a prompt; only the gateway holds the provider key | Search everything on the agent's side for the key |
| Fix the destination | It sends only to routes on its list | A request on an unlisted route appears as sent |
| Cap each run | It refuses, before sending, a request past the run's request or token cap | A request past the cap appears as sent |
| Survive an outage | When the provider is unavailable, a stated fallback answers | The record shows a different fallback, or none |
| Keep logs minimal | It records counts and status, not prompt or reply text | Read one log record |
Products exist for this. In LiteLLM, for example, virtual keys control model access and track spending, key settings can bound budgets, budget periods and request and token rates, and one logging setting keeps message content away from logging providers while request metadata is still tracked (tools/litellm; as_of: 2026-09-05; Source 1; Source 2). A setting is a claim until a record shows it enforced.
org-gateway.json holds a policy (at most 4 requests and 3000
tokens per run, two listed routes, a fallback named replay that answers from a recorded
reply, and the placeholder key sk-EXAMPLE) and six runs, each with the gateway's decisions
and the lines the agent's side logged. The agent's summary reads: "All six runs followed the
gateway policy." Replay the policy over each run by hand, adding each request's reservation
before it is sent, and search the logged lines for the key. Then run:
python lectures/week-11/fixture/org_check.py gateway lectures/week-11/fixture/org-gateway.json
It exits 1 and prints:
policy: at most 4 requests and 3000 tokens per run, each request reserving its worst case before it is sent; routes: route-large, route-small; fallback: replay
R1 route-small: 3 requests sent, 900 tokens used; matches the policy
R2 route-small: request 5 was sent; the policy refuses it before sending (request cap 4); DIFFERS
R3 route-large: request 3 refused before sending (2000 used + 1200 reserved > 3000); matches the policy
R4 route-preview: request 1 was sent; the policy refuses every request on a route it does not list; DIFFERS
R5 route-large: provider unavailable at request 1; fallback replay used; matches the policy
R6 route-small: 1 request sent, 250 tokens used; matches the policy
sandbox-side log: the key appears in R6 line 2
result: POLICY NOT ENFORCED - 2 runs differ from the policy (R2, R4); the key reached the sandbox side in R6
Three separate failures refute the summary: a fifth request that was sent, a request on a route nobody listed, and a key in a log on the agent's side (the check calls that side the sandbox side). R3 is the gateway working: 1000 tokens remained, and the third request was still refused because it could have used 1200. R6 matches the policy on every request and still leaks the key, so a policy replay and a key search are two different checks.
A budget reserves before it spends
R3 shows the rule that makes a cap protect anything: the gateway reserves a request's worst
case before sending it and settles to the actual amount after the reply. A gateway that only
reported spending afterwards would protect nobody's money. The whole rule is the Budget
class at the top of budget_ledger.py, about twenty lines; read
reserve and settle now.
The units are course-made numbers, not prices. A run's cap is 10 units. Each request reserves 4, the worst case for its prompt and output limits, and each reply actually costs 1. Predict how many requests are sent, then run:
python lectures/week-11/fixture/budget_ledger.py
request 1: sent, reply cost 1; spent 1 of 10
request 2: sent, reply cost 1; spent 2 of 10
request 3: sent, reply cost 1; spent 3 of 10
request 4: sent, reply cost 1; spent 4 of 10
request 5: sent, reply cost 1; spent 5 of 10
request 6: sent, reply cost 1; spent 6 of 10
request 7: sent, reply cost 1; spent 7 of 10
request 8: refused before sending (Request exceeds the remaining budget.) spent 7 of 10
Three units remain, and the eighth request is refused because it could cost four. Now let request 2 lose its reply. The gateway cannot tell whether the provider charged for it:
python lectures/week-11/fixture/budget_ledger.py --lost-reply 2
request 1: sent, reply cost 1; spent 1 of 10
request 2: sent, reply lost, reservation of 4 kept; spent 5 of 10
request 3: sent, reply cost 1; spent 6 of 10
request 4: sent, reply cost 1; spent 7 of 10
request 5: refused before sending (Request exceeds the remaining budget.) spent 7 of 10
One lost reply cost the run three requests. A timeout marks where observation stopped; it does not show that nothing happened. The incident later in this lesson turns on the same sentence.
A cap is a hard stop, not a warning. Design an agent task so that its required work finishes inside the cap, or so that it stops and reports. Splitting the work across runs to get around a cap defeats the protection.
Cost per accepted change
A run's total measures one run. What a team needs to know is what an accepted change cost: every run that led to it, including failed and retried attempts, plus the time people spent reviewing it. Runs spent on changes that were rejected count too, because the team paid for them. Cost per token rewards cheap output that nobody can merge.
org-changes.json records the runs and reviews behind four
changes:
| Change | Runs (units) | Decision | Review minutes |
|---|---|---|---|
| fix-118 | a1 6 (checks failed), a2 6 | accepted | 20 |
| fix-119 | a3 3 | accepted | 10 |
| fix-120 | a4 3, a5 2 | rejected | 15 |
| fix-121 | a6 2 (timed out, retried), a7 2 | accepted | 15 |
The agent's summary reads: "A change costs about 3 units: that is the trace total of run a3." Compute the cost per accepted change by hand, then run:
python lectures/week-11/fixture/org_check.py cost lectures/week-11/fixture/org-changes.json
It exits 0 and prints:
change fix-118 accepted: runs a1 6 (checks failed), a2 6; 12 units; review 20 minutes
change fix-119 accepted: runs a3 3; 3 units; review 10 minutes
change fix-120 rejected: runs a4 3, a5 2; 5 units; review 15 minutes
change fix-121 accepted: runs a6 2 (timed out, retried), a7 2; 4 units; review 15 minutes
all runs: 7 runs, 24 units; review: 60 minutes; accepted changes: 3
trace total of one run, for comparison: a3 3 units
result: cost per accepted change is 8 units of model use and 20 review minutes
Eight units, not three: 24 units over 3 accepted changes. Run a3 belongs to the one change that was accepted on its first run, which is why quoting it flatters the total. Report review minutes beside the units in the same way and do not convert them into a price. Exit 0 here means only that the figure is defined; the check exits 1 when no change was accepted.
Route to a cheaper option only when your evaluator says it is good enough
A routing policy sends each task to the cheapest route that passes the evaluator and escalates when a check fails. A cheaper route may be a smaller model, fewer requests, a smaller output limit or no model call at all. Whether it is good enough is a measurement: run the same frozen cases on both routes and compare them case by case.
org-routes.json holds a policy, "Send each task to the cheapest
route that passes at least 90% of its frozen evaluator cases", three route choices and the
evaluator results behind them. The route names are placeholders. The agent's summary reads:
"Routing moved all three tasks to a cheaper route without losing quality." Decide for each
task whether the chosen route passes every case the current route passes, on the revision
in use. Then run:
python lectures/week-11/fixture/org_check.py route lectures/week-11/fixture/org-routes.json
It exits 1 and prints:
policy: Send each task to the cheapest route that passes at least 90% of its frozen evaluator cases.
status-is-failed: 12 frozen cases; current route route-large; chosen route no-model
route-large r2: 12 of 12
no-model r1: 11 of 12; fails case 7
route-small r2: 10 of 12; fails cases 7, 11
no-model: NOT SUPPORTED - fails case 7, which route-large passes
summarize-log: 8 frozen cases; current route route-large; chosen route route-small
route-large r2: 8 of 8
route-small r2: 8 of 8
route-small: supported - passes every case route-large passes, at 1 unit per request instead of 4
label-flaky-test: 10 frozen cases; current route route-large; chosen route route-small
route-large r2: 9 of 10; fails case 4
route-small r1: 9 of 10; fails case 4
route-small: NOT SUPPORTED - the results for route-small are for r1 and the route is now r2; rerun the frozen cases
result: 1 of 3 route choices supported by the evaluator
All three choices satisfy the 90% rule, and only one is supported. In the first task the cheaper route passes 11 of 12 and loses the one case the current route gets right: a rate hides which case was lost. In the third the two routes fail the same case, but the cheaper route's results are for an older revision; a route's name does not guarantee unchanged behaviour, so the frozen cases are rerun whenever a route or a model changes. Even a fresh, complete pass says nothing about inputs that are not among the frozen cases.
Rolling it out: what each piece can hide
Rolling model access out to a whole team brings the shared pieces' failure modes with it. For each risk, name the check that would detect it:
| Risk | Check that would reveal it |
|---|---|
| Keys copied into repositories and CI instead of held by the gateway | Secret scanning and key rotation; the gateway is the only holder |
| Groups bypass the gateway with personal tools, so caps and logs miss that work | Compare the model use you can see with the work being delivered |
| Logs quietly become a store of prompts holding code or customer data | Minimal records by default; read one record |
| One cap for everyone either blocks real tasks or protects nothing | Set caps per task from measured runs; every refusal says what to do next |
| A gateway or provider outage stops everyone at once | A tested fallback, such as a recorded reply or a path that needs no model |
| Adoption is judged by usage rather than results | Evaluator results and incident records, not token counts |
Apply the boundary test from the security week to each piece: a denied call stays denied through every route, and permitted work still succeeds.
An incident: observe before you repair
Operating agents includes the day something breaks. On the ci-dashboard example, revision
R2 went live at 10:05:00, and a user then reported that the project page was slow. Before
anyone changes code, an agent drafts a root-cause analysis. The draft in
agent_rca.json was written for this lesson; no model produced it.
It cites lines of request_log.jsonl, ten minutes of request
records. Each line is one request with its time, request ID, method, path, status and
duration, and nothing from the query string, form or body:
{"timestamp":"2026-11-24T10:05:41+0000","event":"request","request_id":"w11-23","method":"GET","path":"/domains/ci_dashboard/projects/2","status":200,"duration_ms":884.1}
| ID | The draft claims | Cites lines |
|---|---|---|
| H1 | R2 broke imports: the import endpoint has returned server errors since the deploy. | 6, 14, 31 |
| H2 | From 10:05 the project page's p95 rose from about 60 ms to about 900 ms; other pages held. | 23, 28, 34 |
| H3 | The cause is R2's project page running one database query per build. | 23, 28, 34 |
It proposes: "Roll back R2, and add automatic retries to imports."
Predict before running. For each claim, write confirmed, refuted or cannot tell, and the
field or line you expect to decide it. Then run check_rca.py. The
second command summarises only the paths that begin with the prefix you give:
python lectures/week-11/fixture/check_rca.py
python lectures/week-11/fixture/check_rca.py --path /domains/ci_dashboard/projects/
The first command exits 0 and prints:
request log: 40 request records from 10:00:02 to 10:09:58 UTC
window requests 5xx error rate p95 ms
10:00-10:05 20 2 10.0% 58.9
10:05-10:10 20 0 0.0% 884.1
p95: nearest rank, the ceil(0.95 x n)-th smallest duration in the window
H1: R2 broke imports: the import endpoint has returned server errors since the deploy.
line 6 10:01:12 POST /domains/ci_dashboard/import 500 14.2 ms w11-06
line 14 10:03:40 POST /domains/ci_dashboard/import 500 13.8 ms w11-14
line 31 10:07:51 POST /domains/ci_dashboard/import 400 9.6 ms w11-31
H2: From 10:05 the project page's p95 rose from about 60 ms to about 900 ms; other pages held.
line 23 10:05:41 GET /domains/ci_dashboard/projects/2 200 884.1 ms w11-23
line 28 10:07:05 GET /domains/ci_dashboard/projects/5 200 912.6 ms w11-28
line 34 10:08:40 GET /domains/ci_dashboard/projects/5 200 856.7 ms w11-34
H3: The cause is R2's project page running one database query per build.
line 23 10:05:41 GET /domains/ci_dashboard/projects/2 200 884.1 ms w11-23
line 28 10:07:05 GET /domains/ci_dashboard/projects/5 200 912.6 ms w11-28
line 34 10:08:40 GET /domains/ci_dashboard/projects/5 200 856.7 ms w11-34
The script counts and prints; it does not judge. Give each claim one outcome, as you did for review findings in Week 6:
- Confirmed: the log shows the claim as stated, for the path, window and effect it names.
- Refuted: the log shows something the claim rules out, such as a cited line from before the time the claim names, or a window without the effect it asserts.
- Cannot tell: no field in this log could show the claim either way. Name the missing observation and the check that would supply it.
Work the three claims yourself before the studio. Three cautions apply to any request log:
- A cited line supports a claim only if its time, path and status say what the claim says. Open every one; a draft that cites lines has not yet been checked.
- Over a few requests, a percentile is the slowest one or two. Each window here holds six project-page requests, so their p95 is the slowest of the six. Read the lines behind a summary.
- A change that starts when a revision goes live confirms the timing, not the cause. Timing can justify a reversible mitigation; it does not tell you which code to repair.
Act only on what you confirmed. A mitigation that rests on timing alone, such as rolling back a revision, must be reversible, state its rollback condition and end with a check of the user path. A code repair waits until a check that can see its cause has run. Drop any part of a proposal that rests on a refuted claim, but keep the claim's lines if they show something else worth following up.
The continuing dashboard case
A second incident on the same example. The dashboard acknowledges a retry request for build
42, then the reply is lost. The operator retries and a second job is created.
retry-B42-7 identifies one user operation; req-a and req-b identify two delivery
attempts for that same operation.
| Time | Request | Operation key | Observation | Stored effect |
|---|---|---|---|---|
| 00:00 | req-a | retry-B42-7 | Server receives retry for build 42 | None yet |
| 00:01 | req-a | retry-B42-7 | Transaction commits | Job J81 queued; build 42 retrying |
| 00:02 | req-a | retry-B42-7 | Reply is lost; client times out | J81 still queued |
| 00:03 | req-b | retry-B42-7 | Client repeats the same operation | Unprotected handler creates J82 |
| 00:04 | query-c | retry-B42-7 | Operator reads stored state | Two jobs: J81 and J82 |
Decision: which evidence separates a failed request from a successful effect whose reply was lost? The client's timeout cannot: a timeout describes an observation boundary; it does not prove nothing happened. The persisted job and its operation key can.
Write the requirement before any repair. A repeated request with the same key and the same build returns the existing job and creates none. The same key with a different build is a conflict and creates nothing. A new intended retry uses a new key and gets a new job. Two deliveries of the same request that arrive together still create one job. These are the four rows below, and the last column is what a repair must show:
| Row | Replay input | Required replies | Required jobs |
|---|---|---|---|
| 1 | req-a commits, its reply is lost, then req-b repeats | J81 J81 | 1 |
| 2 | req-a and req-b arrive together with the same key and build | J81 J81 | 1 |
| 3 | The same key is reused for build 43 | J81 conflict | 1 |
| 4 | A later intended retry uses retry-B42-8 |
J81 J82 | 2 |
Generate. Ask for a bounded repair tied to the event chain. Suppose the agent replies with the obvious one: look the key up, and insert a job only if none is found. Its summary says the duplicate is fixed, and on the incident as it happened (row 1) that is true.
Verify with a regression. retry_replay.py plays the four
rows against an in-memory database with one of three handlers: unprotected (the incident
build), check-then-insert (the agent's proposal) and repaired. It exits 0 only when
every row meets the requirement. Run it on the incident build first:
python lectures/week-11/fixture/retry_replay.py unprotected
It exits 1 and prints:
mode: unprotected
row replies jobs (build/attempt) build 42 shows
1 J81 J82 J81 42/7, J82 42/7 J82
2 J81 J82 J81 42/7, J82 42/7 J82
3 J81 J82 J81 42/7, J82 43/7 J81
4 J81 J82 J81 42/7, J82 42/8 J82
rows meeting the repaired requirement: 1 of 4
The check can fail, and it fails on the build that had the incident. Now the proposal:
python lectures/week-11/fixture/retry_replay.py check-then-insert
It exits 1 and prints:
mode: check-then-insert
row replies jobs (build/attempt) build 42 shows
1 J81 J81 J81 42/7 J81
2 J81 J82 J81 42/7, J82 42/7 J82
3 J81 conflict J81 42/7 J81
4 J81 J82 J81 42/7, J82 42/8 J82
rows meeting the repaired requirement: 3 of 4
Row 1, the incident as reported, now passes. Row 2 still stores two jobs: both requests looked the key up before either one inserted, so both found nothing. A regression that replayed only the reported sequence would have accepted this repair.
The real repair puts identity where the database can enforce it: a unique
(pipeline, operation_key) stored with the job in one statement, so that a repeated matching
request returns the existing job, the same key with another build is refused, and a batch
with one conflicting row saves nothing. A check followed by a separate insert can still
race, so uniqueness and atomicity belong to the persistence boundary.
python lectures/week-11/fixture/retry_replay.py repaired
It exits 0 and prints:
mode: repaired
row replies jobs (build/attempt) build 42 shows
1 J81 J81 J81 42/7 J81
2 J81 J81 J81 42/7 J81
3 J81 conflict J81 42/7 J81
4 J81 J82 J81 42/7, J82 42/8 J82
rows meeting the repaired requirement: 4 of 4
In rows 1, 3 and 4 the first reply answers req-a, and in row 1 that is the reply the client never saw. Row 2 lists its two replies in sorted order, because either request may write first. The last column is the user's view: the dashboard shows the job of a build's greatest attempt, so in row 4 it shows J82. Check the stored jobs as well as the replies, then the user's view.
What makes this a regression that proves the recovery. Three runs, kept together:
- It fails on the incident build (exit 1, one row of four).
- It fails on the repair that only looks right (exit 1, three rows of four).
- It passes on the repair (exit 0, four rows of four), and stays in the test suite.
A test that has only ever passed cannot show that it would catch the defect.
A regression for a race must force the order. Row 2 runs its two requests in two
threads, and each handler pauses just before its first write, so both finish any lookup
before either one writes. Remove that pause and the race appears only sometimes. In a copy
of the file with the self.between() line of the check-then-insert branch deleted, 20
runs on the author's machine printed 3 of 4 three times and 4 of 4 seventeen times. Your
counts will differ; the point is that seventeen green runs would have accepted a broken
repair. The fixture guide gives the steps.
Acceptance decision. Declare recovery only after the user can observe the required state and duplicate effects are excluded for the tested path. Roll back if the change rejects a new authorised operation or corrupts the existing job mapping. The replay runs this lesson's reasoning on an in-memory table; it is not evidence about any service you run. Evidence that your own project recovered comes from a regression in your own repository.
Agents that keep running after deployment
An agent that acts in a deployed system, for example by triaging alerts or proposing a rollback, is one more service to operate. Give it credentials scoped to its task, a spending and action budget, and a switch that stops it without a deploy. Log what it read, what it proposed and what it changed, so that a postmortem can separate the agent's action from the incident it responded to. Keep irreversible operations behind a person's approval. In Replit's public account of one incident, its agent deleted data from a user's application database while development changes could still reach production, and the user restored the database through a rollback feature the agent did not know about (incidents/replit-db; as_of: 2026-09-05; Source 1). Put the agent in the runbook: who is paged when it misbehaves, how to disable it, and how to recover what it changed. Retried agent actions need the same idempotency you just examined for retried requests, and an agent's grants and caps are the portal and gateway from the first half of this lesson.
Hands-on exercise
The second session opens with CP2; the studio guide covers the rest. In the studio you work in your own project repository: you compare what your agent can reach with what one task needs, count what one accepted change cost you in runs and review minutes, and write one regression that fails before a fix of yours and passes after it.
Before the studio, finish the predictions this page asked for: the five portal calls, the six gateway runs, the two ledgers, the cost per accepted change, the three route choices, the three claims of the draft analysis and the four replay rows for each handler. Where an observation differs from your prediction, keep both and find the line you misread.
Verification
For the shared pieces, the evidence is your prediction beside the check's output:
| Check | Expected evidence | What a failure looks like |
|---|---|---|
portal on the supplied file |
Exit 1, result: OVER-BROAD, with G2 reaching three repositories beyond the task |
Exit 0 on your narrowed copy means the grant now fits; exit 2 means the JSON you edited no longer parses or lacks a field |
gateway |
Exit 1, R2 and R4 differ from the policy and the key appears in R6 line 2 | Calling R3 a failure: a refusal before sending is the policy working |
budget_ledger.py |
Seven requests sent and the eighth refused; with request 2's reply lost, four sent and the fifth refused | A prediction of ten sent requests ignores the reservation |
cost |
Exit 0, 8 units and 20 review minutes per accepted change | Quoting 3 units repeats the summary's single run |
route |
Exit 1, one of three choices supported | Counting two supported means a stale revision was accepted |
For the log check, every claim carries exactly one outcome and the line, or the missing field, that decides it. A claim the log cannot decide stays cannot tell, with its next check written beside it; your rewritten repair acts only on confirmed claims and names its rollback condition and the user-path check that ends it.
For the recovery, run the regression in all three modes and keep the three results together:
python lectures/week-11/fixture/retry_replay.py unprotected
python lectures/week-11/fixture/retry_replay.py check-then-insert
python lectures/week-11/fixture/retry_replay.py repaired
Expected evidence: exit 1 with 1 of 4, exit 1 with 3 of 4, then exit 0 with 4 of 4.
The failure case to recognise is the middle one: a repair that passes the reported sequence
and fails the forced race.
These checks apply stated rules to course-made records. A result is evidence about the rule you applied; it says nothing about how a real portal, gateway, route or model behaves.
Common mistakes
| Mistake | Better next action |
|---|---|
| Accepting an agent's summary because every call it made was allowed | List what each grant can reach and compare it with what the task needs |
| Treating a refusal before sending as a gateway fault | Replay the policy: reserved worst case plus what is already used, against the cap |
| Reading spend from the program's own output | Read the gateway's record, and search the agent's side for the key separately |
| Treating a timed-out request as free | Count its whole reservation until the outcome is known |
| Quoting one run's total as what a change costs | Divide all runs and all review minutes, rejected work included, by the accepted changes |
| Calling a cheaper route good enough because its pass rate is close | Compare case by case on the same frozen cases and the revision in use |
| Accepting a draft's claim because it cites log lines | Open each cited line and compare its time, path and status with the claim |
| Treating a slowdown that starts with a deploy as its cause | Record the timing as confirmed and the cause as cannot tell until a check that can see the cause has run |
| Deduplicating retries with a lookup followed by a separate insert | Put a unique operation key and the job insert in one statement or transaction, then force the concurrent order and count stored jobs |
| Calling a recovery proved because the new test passes | Show the same test failing on the incident build, and on the repair you rejected |
| Declaring recovery because the health endpoint is green | Check the state transition and the displayed result for the affected operation |
A source citation, a fluent explanation or a successful demonstration does not supply a missing observation. State the next check and why it would change your decision.