Skip to content

Read Trajectories Safely: Generate, Then Verify

This page describes the earlier fourteen-week layout and is being rewritten; where it differs, the syllabus applies.

The evaluation framework contains the trajectory schema, cheating-pattern detection, benchmark audits and evaluation-task templates. Results must be reproducible and report both correctness and execution evidence.

These tools read the trajectory JSONL v1 contract. They do not call models, execute commands recorded in trajectories or send trajectories over the network. Students can check their own evidence, and instructors/TAs can inspect process evidence. Generated or suggested code must still use the course safety launcher; these tools parse data only.

Offline Use

Course build repository: After completing make setup at the repository root, generate a minimized copy from a fully synthetic example, then validate and inspect it:

uv run --locked python -m evals.trajectory.validate evals/trajectory/examples/synthetic-tools.jsonl
uv run --locked python -m evals.trajectory.redact evals/trajectory/examples/synthetic-tools.jsonl -o /tmp/course-safe.jsonl
uv run --locked python -m evals.trajectory.validate /tmp/course-safe.jsonl
uv run --locked python -m evals.trajectory.stats /tmp/course-safe.jsonl
uv run --locked python -m evals.trajectory.viewer /tmp/course-safe.jsonl -o /tmp/course-view.html
uv run --locked pytest tests/test_trajectory_tools.py -q

Extracted student package: The package excludes the build repository's Makefile, root project and instructor-side test directory. Use the standalone route below from the student-package root in a POSIX shell, after installing the course-designated uv:

uv venv --python 3.12.13 evals/trajectory/venv
uv pip sync --python evals/trajectory/venv/bin/python --require-hashes evals/trajectory/requirements.lock
export PYTHONPATH="$PWD/src/course_llm/src"
evals/trajectory/venv/bin/python -m evals.trajectory.selfcheck
evals/trajectory/venv/bin/python -m evals.trajectory.validate evals/trajectory/examples/synthetic-tools.jsonl
evals/trajectory/venv/bin/python -m evals.trajectory.redact evals/trajectory/examples/synthetic-tools.jsonl -o /tmp/course-safe.jsonl
evals/trajectory/venv/bin/python -m evals.trajectory.stats /tmp/course-safe.jsonl
evals/trajectory/venv/bin/python -m evals.trajectory.viewer /tmp/course-safe.jsonl -o /tmp/course-view.html

The standalone requirements.lock is exported from the course_llm dependency closure in the root uv.lock. It pins versions and hashes without resolving floating versions or installing project build tools. PYTHONPATH points to the course_llm source included in the student package. Environment preparation may download Python/locked dependencies; instructors can preinstall them or supply a matching offline wheelhouse, adding --no-index --find-links <wheelhouse> to uv pip sync. After preparation, all trajectory commands and selfcheck disable model calls and run offline without pytest or API keys. The self-check verifies synthetic trajectories and HTML round trips; it is not the build repository's full security regression tests.

Open the generated single-file HTML in a browser. Tab to step headings and expand them with Enter or Space. Tool calls use native mark highlighting; roles do not rely on color alone. The page needs no server, JavaScript, CSS, font downloads or external images. CSP blocks scripts, external resources, network connections, forms and object loading. The viewer observes records; it cannot prove model output correct. Check the task's actual test results as well.

All four CLIs accept an input path and optional -o/--output. Without an output path, the complete result goes to stdout. After successful publication, redact writes its redaction report to stderr: only call counts, removed content-field counts and removed-identifier counts, with no original text, filenames or reversible mappings. Failure returns nonzero with fixed English errors and no partial report. Existing output files remain unchanged on failure; new destinations are not created. Successful publication atomically replaces output with owner-only read/write permissions. Identical input/output paths, final-path symlinks and special files are rejected; callers must use trusted parent directories.

Minimization Boundary

Reuse course_llm's TracePolicy and mandatory filtering. By default, remove every value, including original user text, assistant bodies, tool arguments, tool results and original call IDs. Unlabeled names, credentials, personal identifiers, absolute paths and long tool outputs are therefore removed in full rather than guessed with regular expressions. There is no dynamic allowlist or switch to retain bodies. Collected content receives fresh private random HMAC digests; keys are not written to disk. Existing not_collected fields retain their lengths and HMACs for correlation within one run. HTML and statistics do not display digests.

Model/tool identifiers retain only the built-in fake-v1/read/write/bash; all others become [REDACTED] and remain subject to the trajectory secret filtering. Custom tools share one redacted category, so their names cannot be distinguished. This reduces detail to support safe sharing by default. Finer categories require a separately auditable public classification contract. The viewer checks actual fields and identifiers rather than trusting an input's claimed redacted status; bodies or unknown identifiers require running redact first. Successful validation proves structure only, not authentic provenance, absence of privacy risk or billing accuracy.

Statistics and Rejection Rules

  • One line represents one call. steps counts returned assistant steps with usage measurements; snapshot_steps separately counts all context-snapshot steps. tools counts tool requests only in the current returned step, without adding repeated history.
  • Tokens and latency come only from the current returned step. Latency is the sum of observed call latencies, not end-to-end wall-clock time. Known costs are summed with Decimal, without inferring prices from model names. Amounts in the three examples are synthetic arithmetic inputs only.
  • Provider failures preserve missingness: cost: null, unknown_cost_calls, usage_complete: false and latency_complete: false. known_cost is a known subtotal, not the full cost.
  • Reject unknown versions, negative/non-finite numbers, duplicate JSON keys, malformed/blank JSONL, duplicate or reversed sequence numbers within a run, usage/latency repeated in historical snapshots, and inconsistent status/cost. Excerpts may have sequence gaps. v1 has no totals field; the schema rejects added summaries, and every report recomputes results.
  • Limits: 1 MiB per line including the newline; 16 MiB and 10,000 lines per file. Exceeding any limit rejects the entire input. Tokens, sequence numbers and latency cannot exceed 2**53-1; cost text is at most 64 characters, the absolute Decimal exponent at most 18, and the amount at most 1e18. These are tool resource limits, not model or industry data.

Three Fully Synthetic Example Trajectories

Every identifier, timestamp, amount and digest is fixed teaching data constructed by the course, with no real student or user information.

Access, Storage, Deletion and Sharing

Follow the trajectory collection responsibilities without adding course policy. Creators retain only minimal excerpts needed for their work, in controlled directories inside the course workspace that are excluded from Git. Instructors/TAs read necessary evidence only through authorized course channels. Peer sharing uses self-checked minimized copies only. HTML is also process data; redaction does not make it suitable for public release.

Creators clean up original trajectories, minimized copies, reports and HTML; course administrators handle controlled copies and backups. Retention and deletion/withdrawal follow approved course arrangements. These tools set no retention period and do not automatically remove backups. If arrangements are unresolved, use only synthetic material and temporary files, delete them after verification, and record that deletion completed without retaining original content. The /tmp paths above serve synthetic demonstrations only; creators delete both temporary files afterward.

Share in order: confirm authorization and the minimum necessary data; generate a copy with redact; use validate and stats to check missingness; generate HTML with viewer; manually inspect the copy and complete the course secret scan; share through a controlled channel. Never submit environment configuration, raw interviews, credentials, complete tool output or instructor answers. If sensitive values appear, stop sharing, replace/delete affected copies and report through the course process. Automated validation does not replace manual self-checks.

Trajectory Anomaly-Pattern Practice

The detector and synthesizer provide seven fixed lexical clues, deterministic regression reports and an offline read-only execution entry point. Matches enter a human review queue only; they do not determine authorship, misconduct or grades.