Skip to content

Trajectory Anomaly Patterns: Generate, Detect, Explain, Verify

This page describes the earlier fourteen-week layout and is being rewritten; where it differs, the syllabus applies.

For LO6/LO9, first predict which tool requests will trigger clues, then run detection and explain false positives and false negatives. Begin with the detection limits and human review route. Everything here is a public, course-authored exercise, with no hidden answers, scorers or instructor-only question bank. Commands in the synthesizer are JSON strings and are never executed.

Current detection covers synthetic course trajectories that retain supported tool arguments. Ordinary minimal/redacted collection does not retain analyzable commands, paths or content; reports explicitly return insufficient_evidence. These tools provide no content-detection route for real student trajectories and do not require collection-policy changes.

Prepare a Trusted Parsing Environment

After make setup in the course build repository, prepare the image. This step may download locked dependencies:

docker build -f evals/cheat_detector/Dockerfile -t course-cheat-detector:1 .

The image copies only the parser, course_llm and bundled schemas, not the workspace or instructor assets. The base image has a pinned digest; Python comes from .python-version; dependencies reuse the hashed trajectory dependency lock, adding no third-party dependencies. The local image tag must identify the image built from the reviewed source for this run. Do not use an image of unknown provenance merely because it has the same name.

Generate Once, Then Verify Once

These commands only produce/parse synthetic course data; they do not execute shell commands or code inside trajectories.

uv run --locked cheat-synth --pattern read_tests --pattern history_patch --seed 17 > /tmp/course-synthetic.jsonl
uv run --locked python -m evals.trajectory.validate /tmp/course-synthetic.jsonl
uv run --locked cheat-detect /private/tmp/course-synthetic.jsonl
uv run --locked cheat-regression --seed 17
uv run --locked pytest evals/cheat_detector -q
uv run --locked python -m evals.cheat_detector.verify_isolation

On Linux, replace /private/tmp/course-synthetic.jsonl with /tmp/course-synthetic.jsonl. On macOS, /tmp is a system symlink and the CLI rejects symlinks in any input-path component, so use the real path. By default, cheat-detect <trace.jsonl> mounts only that explicit file read-only at /input/trace.jsonl, disables networking, uses a read-only root filesystem, drops capabilities, prevents privilege escalation and limits resources to 1 CPU, 256 MiB and 32 processes. The client deletes the container after a 45-second timeout. It reads private files as the caller (with the caller's UID/GID, or as container root under rootless Docker, which the engine maps to the caller), without changing file permissions or mounting parent directories, the Docker socket, instructor assets or credentials. It never pulls an image automatically and offers no host fallback. The Docker client requires a trusted runtime environment and local Docker permissions. Callers must control parent directories and must not replace files during execution. Paths containing commas, symlinks or special files are rejected.

worker is a passive parsing-library entry point inside the container. Calling it directly on the host does not provide isolation and is not a route for real trajectories. Inputs are limited to 1 MiB per line, 16 MiB and 10,000 lines per file. Shared strict v1 JSONL validation rejects the entire input for duplicate keys, blank lines, broken final lines or version/usage contradictions. Dependency code and bundled schemas are fixed parsing resources. Input never causes other files to be read, commands to run or models to be called. Success writes only JSON to stdout; failure returns 1 with fixed English diagnostics on stderr, without echoing input or detailed runtime errors. Matches still return 0 and do not act as disciplinary gates.

--pattern may be repeated; --seed is required. --kind positive|negative|benign|minimal selects an exercise target, absence of that clue, the same action with legitimate authorization, or missing body content. A fixed seed produces fixed JSONL bytes. Metadata, timestamps, HMAC keys and numbers are synthetic and must not be reused with real data. Callers redirect and clean up the output.

Read Regression Reports

Read coverage.conclusion before findings:

Value Meaning
no_tool_requests The supplied snapshot has no assistant tool requests; this says nothing about behavior outside the trajectory.
insufficient_evidence Requests exist, but none has analyzable arguments for a supported tool; action clues cannot be checked.
partial_evidence Only some requests or required arguments are analyzable; existing matches do not close evidence gaps.
available_arguments Every request in the supplied snapshot has visible, nonempty arguments required by its supported tool shape; lexical-rule limitations still apply.

coverage.argument_availability assigns each request to exactly one of available, partial, unavailable or unsupported; their sum equals tool_requests. Shell tools require command/cmd, reads require path/file_path, write/write_file/edit require a path and content, apply_patch requires content, and network tools require url. See detection limits for aliases and coverage. Missing, non-string or empty arguments and arguments containing redaction markers are not analyzable. Unknown/redacted tool names count as unsupported. Partially available write arguments can still provide clues. The compatibility count unavailable_arguments counts only requests whose arguments were entirely uncollected or contain any redaction marker. It does not cover unknown shapes or missing keys and cannot alone establish coverage. It may overlap the four categories and is not included in their sum.

Every coverage state retains disposition: manual_review_only. Empty findings cannot substitute for the coverage conclusion; available_arguments does not establish complete detection, complete logs or a verified artifact. These are detector-report fields, without changes to the trajectory schema, collection fields or privacy contract.

cheat-regression --seed 17 reports expected/observed results per example and tp/fp/tn/fn per rule. The current fixed course regression set has 7 patterns x 4 contexts; each rule has TP=1, FP=1, TN=25 and FN=1. A positive label marks the predefined investigation target in a synthetic exercise, never an integrity violation. Authorized examples that still match count as FP; targets hidden by body minimization count as FN. These are repository-reproducible exercise counts, not industry performance, real-student false-positive rates or probability calibration. Adding samples or changing rules requires rerunning reports, updating the rule version and reviewing every item.

Read the full report manually, check overlapping rules and repeated snapshots, then explain why a match cannot prove misconduct. verify_isolation actually starts containers to verify disabled networking, read-only roots and inputs, invisible sibling files and absence of model credentials. It then detects all seven clues through the public CLI and checks that trajectories and a mock gradebook are unchanged. Missing Docker/images cause failure rather than a skip.

Student Distribution

The student package contains detection/synthesis/regression source and these guides, but excludes the build repository's project entry points, Dockerfile and tests. Instructors must first supply a local course-cheat-detector:1 image built from reviewed source. Prepare the fixed dependency environment and PYTHONPATH using the student-package commands in the trajectory guide, then run modules with that environment's Python, without a root project:

evals/trajectory/venv/bin/python -m evals.cheat_detector.synth --pattern read_tests --seed 17 > /tmp/course-synthetic.jsonl
evals/trajectory/venv/bin/python -m evals.cheat_detector.cli /tmp/course-synthetic.jsonl
evals/trajectory/venv/bin/python -m evals.cheat_detector.regression --seed 17

Use /private/tmp for input paths on macOS here as well. If the instructor image is unavailable, stop detection; do not switch to a host worker. Synthesis and regression process only inert fixed data in memory and remain available for understanding the rules.