Skip to content

course_llm: generate once, then verify

This page describes the earlier fourteen-week layout and is being rewritten; where it differs, the syllabus applies.

course_llm is the shared offline interface used by the course launcher. It returns text and tool-call requests; it never executes generated code or tools. Student programs send only a prompt through the launcher’s prompt-only IPC. The host-side broker applies the approved course configuration and keeps any credential on the host; the sandbox receives neither credentials nor provider connection details. Generated code and shell commands must be checked and run through the launcher.

Follow Week 0 setup for admission, launcher use and the approved access or replay route. This page deliberately documents the package contract, not host configuration.

Start offline

The default scripted provider needs no network or credential:

uv run course-llm ping --provider fake --trace /tmp/course-ping.jsonl
uv run course-llm cost /tmp/course-ping.jsonl
uv run pytest src/course_llm -q

A successful ping prints OK without printing a response, configuration or request body. cost validates each trajectory line and prints USD <amount>. These commands create only synthetic evidence; remove temporary traces after checking them.

from course_llm import Client, FakeProvider, TracePolicy, Tracer
from course_llm.trajectory import validate_record
import json
from pathlib import Path

path = Path("/tmp/course-example.jsonl")
with Client(
    FakeProvider([{"content": "synthetic answer"}]), tracer=Tracer(path, TracePolicy())
) as client:
    response = client.chat([{"role": "user", "content": "synthetic question"}])
    assert response.content == "synthetic answer"  # Verify the generated result.
for line in path.read_text().splitlines():
    validate_record(json.loads(line))  # Verify the recorded evidence separately.

Replay and the launcher boundary

Replay is an equal learning route when a hosted run is unavailable. Use the reviewed record and the launcher command supplied by Week 0; choose replay without explaining why. Write a prediction first, inspect the returned text, run the local checks, and record that the result came from replay. A replay is not a new model call.

A typical replay invocation has this shape; copy the concrete version, digest and reviewed record path from the setup page or the teaching team:

sh scripts/course-sandbox \
  --profile approved-hosted \
  --version VERSION \
  --digest DIGEST \
  --model replay \
  --replay-file REPLAY_FILE \
  --replay-digest REPLAY_DIGEST \
  --workspace example-work \
  -- python generate.py

The broker accepts a bounded prompt and returns text only. It rejects extra fields, direct URLs, credentials, tools, shell commands and requests that exceed the per-run limits. The sandbox cannot read the host broker’s configuration. Inspect the proposed result and run the verifier as a separate step; a generated answer never counts as its own verification.

Public interface and tool calls

Client.chat(messages, tools=None, **opts) -> Response is synchronous and non-streaming. Response contains content, usage.input_tokens, usage.output_tokens, tool_calls and decimal cost_usd. Response.message() returns the assistant message needed for the next request. Close the client with with Client(...) or close().

Messages use {"role": "user", "content": "..."}. Supported roles are system, user, assistant and tool; every message must have string content. An assistant message containing only tool calls still has an explicit empty string. A tool definition is {"name": "read", "description": "...", "parameters": {"type": "object", ...}}. A tool request is {"id": "call-1", "name": "read", "arguments": {"path": "fixture.txt"}}. After a request, the caller validates the name, arguments and workspace permission before the launcher executes anything. Package tests use fixed tool results and perform no shell or file operations.

The scripted fake provider accepts a JSON/YAML script, a Python list or a callable. Script items can assert exact messages and tools before returning a fixed response. An exhausted script or failed assertion raises an error without echoing the hidden content. Fixed scripts make local checks reproducible and require no network.

Budget and evidence

The launcher bounds prompt size, request count, output size and wall-clock time for each run. Budget also prevents a client from reserving more output than the run allows. A rejected reservation sends no request. A completed call records usage and status; an uncertain call is recorded as uncertain and is not silently retried. These are conservative local checks, not a promise about a remote bill.

Use the supplied FakeProvider or a reviewed replay in tests and ordinary offline work. Do not add another model route, call a provider directly, or put connection details in a student script. Any real route is selected by the host-side broker and documented in the central setup page.

Minimal traces and filtering

TracePolicy() defaults to minimal. Each message, argument and tool result keeps only its state, encoded length and digest. An explicit redacted policy may retain fixed synthetic content; every string outside its allowlist is replaced in full, as are sensitive scalar values. Never build an allowlist dynamically from a user message or a model response.

The tracer always filters credential patterns, personal identifiers, long identifiers, course canaries and other registered sensitive values. Raw requests, responses and tool results are application data, not safe logs. Minimal trajectories cannot reconstruct a conversation, and a digest cannot recover the original value.

JSONL contract

The canonical schema is evals/trajectory.schema.json. The package includes a byte-identical copy for offline validation. Each line represents one call, not a cumulative snapshot:

  • schema_version is "1"; run records a random id, sequence, model label, status, timestamp, trace policy and cost status.
  • steps records the input snapshot and returned assistant message. Only the returned step has this call’s usage and latency; other steps use null, which is not zero usage.
  • A missing value is distinct from a redacted value. Consumers must preserve that distinction.
  • Unknown cost is recorded as unknown and cannot be reported as a completed total.

validate_record checks the schema and state rules. It does not prove that a log is authentic or that a human has reviewed it. Keep only the minimum synthetic evidence needed to explain the work and run the local verifier again after any change.

Safe log handling

Keep traces in a controlled course workspace, restrict access, and delete temporary files and known copies after verification. Do not submit environment files, credentials, raw interviews, or full tool output. If private content appears, stop, remove the copy and use the course reporting route. Passing schema validation does not replace a privacy check or an explanation of what was verified.