Skip to content

Team operation and incident practice fixtures

Small course-authored files for the shared pieces a team operates, the log check and the incident replay. Every record, unit, log line and agent summary is synthetic. None of the scripts calls a model, opens a network connection or writes a file, and each uses only the Python standard library. Read each one as text before running it. Run the commands from the course checkout; if your system has no python command, type python3.

Files

  • org_check.py checks one JSON record with one of four commands and ends with a result: line.
  • portal replays each call through the grants: a call is allowed when a grant for its tool names its repository, or every repository with *. It then lists what each grant can reach and marks the repositories the task does not need. It exits 1 when any grant reaches beyond the task.
  • gateway replays each run through the policy: a request is refused before it is sent when it would exceed the request cap, when the tokens already used plus its own reservation would exceed the token cap, or when its route is not listed; when the provider is unavailable, the policy's fallback must answer. It also searches every line logged on the agent's side (sandbox_log in the file) for the key. It exits 1 when a recorded decision differs from the policy or the key appears.
  • cost adds the units of every run, including failed, retried and rejected ones, and every review minute, then divides each total by the number of accepted changes. It prints one run's total beside the result. It exits 1 only when no change was accepted.
  • route supports a chosen route only when its evaluator results are for the route's current revision and it fails no case that the current route passes. It exits 1 when any choice is not supported.

A command exits 2 when its file is missing, is not JSON, lacks a field or holds a value of the wrong kind. - org-portal.json, org-gateway.json, org-changes.json and org-routes.json hold the records: a portal's grants and one agent session, a gateway policy and six runs, the runs and review minutes behind four changes, and a routing policy with its evaluator results. Each holds an agent_summary that stands in for an agent's report; no model wrote it. The route names are placeholders and the only key is the placeholder sk-EXAMPLE. - budget_ledger.py replays a per-run ledger through the small Budget class at the top of the file: reserve a worst case before sending, settle to the actual cost after the reply, and keep the whole reservation when a reply is lost. Its units are course-made numbers, not prices. Change the numbers with --cap, --reserve, --actual and --lost-reply N. - check_rca.py reads request_log.jsonl, forty request records, and agent_rca.json, a course-authored stand-in for an agent's draft root-cause analysis that no model wrote. For each five-minute window it prints the request count, the number and share of 5xx responses and the 95th-percentile duration by nearest rank; --path limits that summary to paths with one prefix. Then it prints every log line each claim cites. It never decides whether a claim is confirmed, refuted or cannot tell. - retry_replay.py plays the four replay rows from the notes against an in-memory SQLite database with one of three handlers: unprotected, check-then-insert (look the key up, then insert it if absent) and repaired. Row 2 forces the race: its two requests run in two threads, and each handler pauses just before its first write, so both finish any lookup before either one writes; put any lookup you add before that pause. The connection runs one statement at a time and no lock spans a lookup and an insert. In the repaired handler an operation key is unique within its pipeline, the whole batch is one statement so a conflicting row saves nothing, and the dashboard shows the job of a build's greatest attempt.

Expected distinction

portal on the supplied file exits 1: every call but the deploy is allowed, and grant G2 reaches three repositories the task does not need, one of which the session read. With G2 narrowed to the task's repository it exits 0 and that read is denied. gateway exits 1: runs R2 and R4 differ from the policy and the key appears in a line of R6. cost exits 0 with 8 units and 20 review minutes per accepted change, beside a single run of 3 units. route exits 1 with one of three choices supported.

The default ledger (cap 10, reserve 4, actual 1) admits seven requests and refuses the eighth with three units unspent. A lost reply keeps its full reservation, so with request 2's reply lost the fifth request is refused.

The log check summarises 40 records, 20 in each window: 2 server errors before 10:05 and none after, and a p95 of 58.9 ms before and 884.1 ms after. It exits 0 and leaves every outcome to you.

The unprotected replay stores two jobs for a repeated or concurrent retry and a job for build 43 under a reused key; one row of four meets the repaired requirement and it exits 1. Check-then-insert handles the sequential repeat and the reused key but stores two jobs in row 2, because both lookups ran before either insert; three rows of four meet the requirement and it exits 1. The repaired replay returns J81 to both matching requests, even in row 2, answers the reused key with a conflict that writes nothing, creates J82 for the new key and exits 0. In row 4 the dashboard shows J82, the job of attempt 8.

A result is evidence about the rule a script applied to synthetic records. It is not evidence about a real portal, gateway, route or model, and the replay is not evidence that any service of yours recovered. The request log's durations say nothing about how fast any real build is.

Optional variation

Test what the forced order adds to the evidence. Copy the replay into a new directory:

mkdir week11-replay
cp lectures/week-11/fixture/retry_replay.py week11-replay/

In week11-replay/retry_replay.py, delete the self.between() line in the check-then-insert branch of Store.handle, so the two requests in row 2 are no longer held until both lookups finish. Predict whether row 2 still fails on every run. Then run your copy about 20 times and read the last line of each run:

python week11-replay/retry_replay.py check-then-insert

Record how many runs printed 3 of 4. How often the race appears depends on the machine and on thread timing; any single run may print 4 of 4 and exit 0, so one pass proves nothing. Explain which line of the handler lets the duplicate through when it does appear, then delete week11-replay.