Skip to content

Week 1: The internals of coding agents

Learning objectives

By the end of this week, you can:

  • LO1: explain a coding agent's loop, system prompt and tool definitions;
  • LO1: predict a failure caused by missing context, invalid tool arguments or a stop condition, then test that prediction; and
  • LO9: label a claim as observed or inferred, cite its evidence and revise your decision when a check supplies new information.

You need Python functions, dictionaries, basic tests and a terminal. The first session builds a minimal agent live and inspects a request. The second session is a studio using your own coding agent. A1 is due this week.

Minimal example

The examples in this lesson are course-constructed. The supplied session is hand-authored; your studio capture provides evidence of your own real agent run.

One repair from contract to regression

The requirement says to ignore surrounding whitespace and ASCII letter case, map ok to passed, fail to failed, and other strings to unknown. A plausible candidate is:

def status_label(raw):
    return {"ok": "passed", "fail": "failed"}.get(raw.lower(), "unknown")

Generate: propose the smallest patch. Verify: before editing, derive the expected result for " OK " from the requirement. It must be passed; the candidate returns unknown. Adding .strip() before .lower() repairs that case. Retain the ordinary and unknown-status checks as well. A plausible patch needs a completed check on the changed code.

Build the agent around a proposer

Open loop.py. It supplies checked, tool and last_check; run() is deliberately unfinished. A model supplies proposals, while the controller executes permitted calls and feeds observations into the next request. For the live build, a scripted proposer lets us inspect this behavior without making a model request.

Tool Accepted arguments What actually happens
read path: result.txt Return the file's text
write That path, plus text of at most 32 characters Replace the file; return written
bash command: verify Run a fixed check for the exact bytes 4\n

Despite its name, bash does not accept arbitrary shell commands. For example, echo 4 > result.txt is refused. This restriction is in the tool's code; a prompt asking the model to be careful would not enforce it.

The instructor builds run() from its public five-step contract:

  1. Begin with a user message holding the task and an empty set of used call IDs.
  2. On each turn, send a copy of history to the proposer. Apply checked() and reject a reused call ID before executing any call from that reply.
  3. Append the assistant reply. With no tool calls, return the history and final.
  4. For every call in order, execute tool() and append one role: tool message with its own tool_call_id. Convert missing files and timeouts into failed observations that the next request receives.
  5. At the turn budget, return the history and step_limit.

Generate: the scripted proposer reads, writes 4\n, requests verification and then sends a reply with no calls. Verify: the public checks independently assert what the controller did. A successful scripted demonstration has this sequence:

Proposed action Tool observation
c1: read result.txt 5\n
c2: write 4\n written
c3: verify verified
final reply with no calls Controller stops with final

A second run writes 5\n instead. Its write still returns written, but the check returns check_failed. The proposer can still stop with final. Completion of the loop and correctness of the change are separate questions. A later write also makes an earlier passing check stale; last_check reports not_run until another check runs.

Read a real agent's instructions and your visible request

A request can carry system or developer instructions, user context, assistant calls and tool results, with tool definitions beside the messages. A tool definition names the operation, describes its intended use and declares argument shapes. A system prompt describes the agent's role and workflow; it does not execute tests.

Source 1 and Source 2 are pinned published instruction and tool files from open-source agents. Read one stopping instruction and one schema constraint. Then inspect the corresponding information exposed by your agent during the studio. Source files describe components, while only a captured request can establish which components that run sent. A visible transcript may omit system messages or the wire request; record that limit.

Practice the request-reading method with the supplied sample:

python3 lectures/week-02/session/show_session.py \
  lectures/week-02/session/sample-session.jsonl --request 4

Identify a role/goal instruction, a tool argument rule, a stopping rule and a request for caution. Locate calls and the results matched by call ID. Request 4 proposes an edit; request 5 carries its failed result. Request 6 shows a retry and request 7 carries a failing test. Predict the next action before revealing each response. Finally locate the verification supporting the last response.

Observed: the record contains a particular call, result or instruction at a citable location. Inferred: you explain why it happened or what might happen next. An error message is observed; an explanation of the model's private reasoning is inferred. Use the questions in the existing request dissection and prompt dissection for this reading; this week's activity and due item are on this page.

Hands-on exercise

Finish and check the live loop

From the course root, make a fresh draft copy. The scripts import the existing model client for definitions, but these commands use only scripted practice.

mkdir week01-loop
cp lectures/week-02/fixture/*.py week01-loop/
PYTHONPATH="$PWD/scripts" python3 week01-loop/verify_loop.py

Before implementing run(), expect the nonzero message Write run() in loop.py first. Edit only your copy's run() using the five steps above, then rerun that same command. With the controller complete, expect Ran 15 tests and OK. Until then, the named failures tell you which behavior still needs work.

At the class help checkpoint, trace one call/result pair on paper if the controller still fails. Then complete the missing behavior and rerun the checks before the demonstration:

(cd week01-loop && PYTHONPATH="../scripts" python3 demo.py)

Expect a first line with generation: scripted-practice, stop_reason: final, check: verified and eight messages. The two COURSE_ARTIFACT lines after it, one long, are encoded copies of the two output files; you can ignore them. Inspect completion.json and result.txt in your copy. Change only CANDIDATE to "5\n" in the copied demo.py, predict, and rerun. Expect a nonzero exit and check: check_failed. Preserve the result before restoring "4\n" and rerunning. Do not use the fixture's hosted mode for this demonstration.

Studio: capture and read your own session

Follow the studio to create a fresh status-label task, capture a real coding-agent session with a forced failing check, and verify the repair. Keep the original failure, the later check and a short explanation. At the revision checkpoint, check one observed claim against its citation and revise it if needed.

Use the safety practices in Tools and model access. Run the agent in a dedicated draft directory or container without employer or cloud credentials. Seats or credit are provided by the course; students do not pay.

Verification

Keep the verification tied to the candidate it tested:

  • Loop: all public checks pass in your copied controller; wrong writes fail, denied calls have no effect, and every result matches its call ID.
  • Request: point to an instruction, a tool schema, a returned result and a stop in the record. If the capture omits one, state that it was not visible.
  • Studio: retain the baseline failure, the repaired check and a bounded diff. A test on the untouched original fixture cannot verify your studio candidate.
  • Decision: end A1 with accept, narrow, re-check or stop and the specific evidence supporting it. Keep one AI-use note with the work and reuse its ID.

The loop's turn budget guarantees a bound on requests. It does not prove progress, validate the task's meaning or establish a secure process boundary around editable Python. These are properties you must distinguish when interpreting a trace.

Common mistakes

Mistake Detection Correction
Treating written or final as correctness No current passing check Run the check after the latest write
Losing a refused call Next request has no result under that ID Return the refusal as an observation
Reusing a call ID Two different actions have the same identifier Reject before effects and use fresh IDs
Treating a prompt rule as enforcement No schema, tool check or runtime check stops a violation Name the enforcing component or say it only asks
Claiming invisible system prompts were observed Capture has only visible chat messages Mark the request content unavailable
Counting calls as model requests A response has several calls Count request blocks separately
Calling scripted practice a real model run Proposer is demo.scripted Preserve the actual generation mode
Accepting a stale passing test Another write occurred afterward Verify the current revision