Skip to content

Dissect a recorded agent session

The loop you finish in the Week 1 notes shows the idea in miniature. A mature coding agent runs the same loop, but every request carries far more: long instructions, a set of tool definitions and the whole conversation so far. In this activity you read one recorded session request by request. You predict first, check each prediction against the record, and then compare what you saw with your own Week 1 loop.

The session shipped with this page, sample-session.jsonl, is course-made: the course team wrote it by hand in the same format a recording of a real agent uses. It shows what to look for. It does not describe how any product behaves, and it contains no measured numbers.

Purpose

Optional practice. It earns no marks and has no submission. Week 1 uses the questions of step 3 when you read your own session for A1, and Week 2 uses the record to look at how a request grows.

Estimated time: about two and a half hours for all five steps: predictions, the request-by-request reading, the comparison with your loop and the short write-up. This is an estimate, not a limit; steps 1-4 stand on their own if you have less time.

Order:

  1. Do steps 1-4 on the published record.
  2. Swap tables with a partner as described under Review, and revise.
  3. Step 5 needs the completion.json your own Week 1 loop wrote, so do it once your run() passes its checks. Then write up the records.

Materials:

Grouping: work alone or in pairs. You may discuss the record with a partner, but each person writes their own predictions, dissection and comparison.

Cost: everyone dissects the same published record. The activity makes no model call and needs no API key or agent seat.

AI use: red. Write your predictions, dissection and comparison without AI; the point is your own reading.

Which record to use

The header line of each record names its source. "provenance": "course-made-sample" is the hand-written sample. "provenance": "teacher-recording" is a real session that the teaching team captured; its header gives the agent's name and version, the recording date and a list of what was redacted. When a teacher recording sits beside the sample in session/, dissect the teacher recording and use the sample only to practise the steps. Otherwise, dissect the sample.

Step 1: predict before you open the record

Read only the task in the header line: the agent must change a status label in a small repository and keep its tests passing, which touches a source file and a test file. Then, before running the viewer, write down your answers:

  1. How many requests will the agent send to the model to finish this two-file change?
  2. Which tool will it call first?
  3. Will the instructions and tool definitions be sent once, or with every request?
  4. Will any tool call fail? If one does, what will the agent do next?
  5. How will the size of the request change from the first request to the last? Will anything sent earlier be removed or rewritten?
  6. Why will the loop stop, and what evidence will the agent have for its final message?

Keep these answers unchanged. You compare them with the record in step 4.

Step 2: read the record request by request

Run the viewer from the course root. It uses only Python's standard library, reads the file and prints text. It never runs anything found in the record and never uses the network.

python3 lectures/week-02/session/show_session.py lectures/week-02/session/sample-session.jsonl

The summary prints one block per request: whether the instructions and tools were sent again, which messages are new since the previous request, which earlier messages changed, and what came back. To see everything one request sent and received, including the full instructions, every tool schema and every message, name the request:

python3 lectures/week-02/session/show_session.py lectures/week-02/session/sample-session.jsonl --request 4

Expected: the summary starts with a Session: line naming the source and ends with an End of record block. A file the viewer cannot read as a session record gives one error: line naming the problem and its line number.

Step 3: dissect

Fill one row per request. Keep each cell short and cite what you saw.

Request New or changed instructions Tools offered (count; any change) What came back: text, tool calls or both How the results return What was re-sent, cleared or rewritten Continue or stop, and why

Then answer these questions. Label every answer [observed] when you can point to it in the record (give the request number and message index, for example request 5, [11]), or [inferred] when you are reasoning from what you read without having seen it happen. Both kinds are acceptable. A wrong label is the mistake to avoid.

A. The instructions. Open the instructions in request 1. Sort each sentence into the four groups from the notes: role or goal, tool contract, stopping or escalation rule, request for caution. Pick one sentence from each group and say which failure it guards against. Then say what, if anything, enforces it: a tool schema, the tool's own checks, a sandbox, an approval step, or nothing. Where does the description of the environment (working directory, date) sit in the request? Find a message that was added in the middle of the conversation rather than at the start. Who wrote it, which role does it use, and why might an agent add it there instead of repeating it in the instructions?

B. The tools. How many tools were offered, and did the set change during the session? Choose two tools that do different kinds of work, for example a file-editing tool and the planning tool, or the command tool. For each one: copy the relevant part of its schema; say which parameters are required, which are optional and what the tool deliberately does not let the model choose; find one sentence in its description that looks as if it was added because a model kept making the same mistake, and name that mistake. Which constraints does the schema enforce, and which does the description only ask for?

C. What the model returned. For each response, was it text, tool calls or both? Find a response with more than one tool call. Were those calls independent of each other?

D. How results return. How does each tool result find its call? What did the agent see when a tool call failed, and what did it do in the next request? Count the requests between the failure and the recovery, and quote the error text exactly.

E. Where and why the loop stopped. Which response ended the loop, and how can you tell from the record alone? Find the observation that supports the final message. Is the final claim verified by an observation in the record, or only stated? Would the record let you detect a final message that claimed more than the agent had checked?

F. What was re-sent, cleared or rewritten. How did the request size change across the session, and what accounts for most of it? Which earlier messages changed between two requests, what replaced them, and what information did that remove? Did the agent need that information again, and how would it get it back?

G. Plans and task state. Where does the agent keep track of which steps are done? If it records a plan or task list, list every place the plan appears: tool calls, tool results and any message added in the middle of the conversation. What does the result of a plan update tell the agent, and what does it not check? Find a request where the plan, as last recorded, is behind what the tool results already show, or say there is none. For each item marked completed at the end, cite the observation that shows the step was done. If the agent keeps no plan, say where its task state lives instead.

Step 4: check your predictions

Put each prediction from step 1 next to what you observed. For the prediction that was furthest off, write two or three sentences: what in the record surprised you, and what you would now expect from your own loop in the same situation.

Step 5: compare with your Week 1 loop

Open the completion.json that demo.py wrote in your copy of the Week 1 loop (the notes use the directory week01-loop). It is one JSON object: the whole message history of one scripted run, the stop reason and the status of the last check. Open your loop.py beside it.

Answer from the two records side by side:

  • What does your loop send as its first message, and what does the recorded agent send? Where do your tool names and argument rules live: in a schema the model sees, in your validation code (checked), or both?
  • Does your loop pass the whole history on every turn? Compare that with what the recorded agent re-sent or cleared.
  • How does each loop decide to stop, and what evidence does each have at that point?
  • Where does each loop keep track of which steps are done, and what would happen to that record if old messages were dropped or cleared?

Then choose:

  • one behaviour you would copy into your loop, and
  • one behaviour you would change if you built the recorded agent,

each with its evidence (request number and what you saw) and the check that would show it working: a test, a trace you would inspect, or a failing case that should now stop correctly. A behaviour you copy without a check is a claim, not an improvement.

Records

Nothing is handed in. Keep, in your own notes:

  • your six predictions as you wrote them before reading the record;
  • the request table and your answers to A–G, each labelled and cited;
  • the prediction check and your copy/change choices with their checks.

Name the record you dissected by its file name and provenance. Cite request numbers and quote only the lines your answer depends on. The labels and the citation style are the ones A1 asks for on your own session; if you reuse an answer there, check every quoted line for keys, personal data and private code first. Never include an API key, a header or an environment value.

Review

Swap your table with a partner. The partner checks one [observed] answer against the record using its request number, and marks any place where a final message or success claim has no supporting observation. Revise that answer. The partner's check is feedback; your explanation remains your own.

Common mistakes

Mistake How to notice it Small correction
Counting tool calls instead of requests Your request count differs from the number of Request N blocks One response can hold several calls; count model requests
Treating the final message as evidence Your answer to E quotes only the last message Find the tool result that supports it, or say there is none
Treating a plan item marked completed as evidence Your answer to G cites only a plan update Find the tool result that shows the step was done
Assuming an instruction is enforced You cannot name what would stop the model from ignoring it Look for a schema rule, a tool check or a sandbox; otherwise it only asks
Missing what the agent wrote into the conversation You attribute every user message to the person Read the message itself; an agent can add messages in any role
Labelling an inference as observed Your citation points to a message that does not show the claim Change the label or find the request that shows it

All materials are plain text. The viewer output, tables and record can be read with a screen reader or keyboard-only editor; no diagram or colour is needed for any step.