Skip to content

Week 11 studio: An operating note and a regression for your project

Due this week: CP2

CP2 takes the first 30 minutes of the second session, on paper, without AI.

Guest or studio

After CP2 the session is offered to a practitioner guest for at most 45 minutes, on how their team runs agents. Write one question for the guest beforehand. In a week without a guest, or in the time that remains, the session is the studio below. You work on your own project while staff circulate; what you do not finish in the room, finish before next week.

Studio sequence

Format: individual work in your project repository, then a short written exchange with a partner. Materials: your project repository, your agent's session records for one change you accepted, and a text editor. The studio produces one operating note, kept in your project repository. It has three parts, one for each half of the lecture and one decision.

The git commands below read your repository and, in step 3, swap one file to an earlier version and back. Commit or set aside your current work before step 3. If your project is not written in Python, use its own test command where a step says python -m pytest.

1. Reach against need (15 minutes)

Pick one task from your project backlog that you would hand to your agent. Write the repositories, directories, tools and network destinations that task needs. Then list what the agent can actually reach in the environment where you run it. Start in your project repository with:

git status --short
git remote -v
env | grep -i -E 'key|token|secret|password|credential' | cut -d= -f1

The last command prints only the names of environment variables whose name suggests a credential, never their values. Any name it prints is something every command your agent runs can read. Add what the first two do not show: other repositories and directories the agent's working directory or container can see, the tools and MCP servers it has enabled, and whether its permission mode asks before a command runs.

Write two columns, need and reach, and mark every entry of reach that the task does not need. For each marked entry decide: narrow it now, or keep it with an owner and a reason. This is the portal check from the lecture, applied to a team of one.

2. What one accepted change cost (15 minutes)

Choose the most recent change in your project that you accepted from your agent. Find it:

git log --oneline -10
git show --stat HEAD

Replace HEAD with the commit you chose if it is not the latest. From your agent's session records, count every run that led to this change, including the ones that failed, timed out or were abandoned, and any run spent on an approach you rejected along the way. Use the unit your agent reports (requests, tokens or credit) and do not convert it into a price. Add the minutes you spent reviewing and checking the change yourself.

Write one line: runs, total in your agent's unit, and review minutes for this one accepted change. If your records cannot show an earlier run, write cannot tell for that part and the record you would keep next time. One change is one observation about your own process; it is not a finding about any model.

3. One regression for one fix (25 minutes)

Choose a defect you have already fixed in your project, with the fix committed. Find the commit before the fix:

git log --oneline -- FILE

Here FILE is the source file the fix changed. Write a test that states the required behaviour for the input that showed the defect, and commit it. Then show that the test can fail: put the file back as it was before the fix, run the test, restore the file and run the test again. BEFORE is the commit before the fix and TEST is your test file:

git checkout BEFORE -- FILE
python -m pytest TEST -q
git checkout HEAD -- FILE
python -m pytest TEST -q
git status --short

Expected evidence: the first test run fails on the assertion about the defect, the second passes, and git status --short prints nothing for FILE, which shows the file is back at your current version (untracked files such as a test cache may still be listed). Keep both outputs beside their revisions. Three results need a second look:

  • The test passes on the earlier file. It does not reach the defect. Change the input, not the claim.
  • The test fails on the earlier file for another reason, such as an import error. That is not evidence about the defect. If the fix changed several files, name them all in both git checkout commands.
  • The defect depends on timing or order. A test that sometimes passes on the earlier file proves nothing; force the order, as row 2 of the lecture's replay does.

If you have no fixed defect yet, take one open item from your backlog instead: write the test first, keep its failing output, and record the regression as half done.

4. Decide and exchange (10 minutes)

Finish the operating note with one decision line: accept, narrow, recheck or stop, and the evidence behind it. Then exchange notes with a partner in writing. Each of you asks one changed-condition question, for example "what would your regression show if two requests arrived together?", and the other answers with the input, the expected result and the check that would decide it.

Your operating note holds:

Task and what it needs
Reach: what the agent can reach; entries beyond the need, each narrowed or kept with owner and reason
One accepted change: runs, total in the agent's unit, review minutes; what the records cannot show
Regression: defect, BEFORE and current revisions, failing output, passing output
Not covered: one condition the regression does not test
Decision: accept | narrow | recheck | stop, with the evidence

After class: fixture practice

If you did not finish the lecture's predictions, do them now from the course checkout. Each takes a prediction first and a run second; the commands and their expected output are in the notes.

Part Predict by hand Then run
Portal The grant that allows each of the five calls, and what each grant can reach org_check.py portal, then the narrowed copy
Gateway For each of the six runs, the request where the policy stops it; the line that holds the key org_check.py gateway
Budget How many requests are sent with cap 10, reserve 4 and actual 1; then with request 2's reply lost budget_ledger.py, then --lost-reply 2, then one variation of your own
Cost Units and review minutes per accepted change org_check.py cost
Route Supported or not for each of the three choices, with the deciding line org_check.py route
Replay Replies and stored jobs for each of the four rows under each handler retry_replay.py in all three modes

Observe before you repair

The draft, the log and the three outcomes are in the notes.

Minutes Action Retained work
0-7 Read agent_rca.json and one log line. For each claim, predict confirmed, refuted or cannot tell, and the field or line that would decide it Three predictions
7-17 Run check_rca.py once without options, then once with --path for each page a claim names The summaries you used
17-27 Open every cited line. Give each claim one outcome with the line, or the missing field, that decides it Three outcomes with evidence
27-35 For a claim you cannot decide, write the next check and the result that would confirm or refute it. Rewrite the proposed repair so that it acts only on confirmed claims, with a rollback condition and a user-path check Next check and bounded mitigation

From the course checkout:

python lectures/week-11/fixture/check_rca.py
python lectures/week-11/fixture/check_rca.py --path /domains/ci_dashboard/projects/
python lectures/week-11/fixture/check_rca.py --path /domains/ci_dashboard/import

With the first prefix the summary reads 12 records, with a p95 of 61.4 ms before 10:05 and 912.6 ms after. With the second it reads 8 records, with 2 server errors in 4 requests before 10:05 and 0 in 4 after. The script never gives an outcome; that part is yours.

Transfer

Leave the job table behind. While a service runs revision R7, an on-call agent asks the deployment tool to "roll back one revision". The call times out, so the agent sends the same request again. The first call had in fact completed, and the service already runs R6.

State what the second call must return and which effect it must not cause. Write the recovery assertion you would check afterwards. Then describe a later rollback that the tool must still carry out, and say what tells it apart from a repeat.

Participation and accessibility

All required material is plain text and can be completed with a keyboard. An approved written or asynchronous exchange can replace the live exchange while preserving the same individual note. Planned times organise support; they do not penalise extra time or an approved arrangement. Keep one operating note; do not copy private chat, personal data or credentials into it, and record a variable's name, never its value.