Skip to content

A5: Score agent readiness from observations

At a glance

  • Task: score a repository with four observable anchors and report what would change each score.
  • Week: 6
  • Weight: 2.5 points
  • Due: Sunday 23:59 in the listed week
  • Hand in: a four-row scorecard, verification evidence and report.json.
  • Done when: every level names an anchor and evidence, and a proposed agent score has been checked.
  • Time: published after a timed pilot; record your prediction and actual time.

Objectives

Use observable evidence to judge structure, documentation, tests and checks, and explain uncertainty (LO4, LO5).

Prerequisites

Score the revised A4 starter repository that you copied and changed, at the immutable commit or local snapshot named in your report. Do not score the course repository or an unrelated project. Read the revision from that repository and, if you score a snapshot, make the copy from the same commit:

git -C path/to/a4-repo rev-parse HEAD
mkdir -p path/to/a5-submission/snapshot
git -C path/to/a4-repo archive --format=tar HEAD | tar -x -C path/to/a5-submission/snapshot

git clone path/to/a4-repo path/to/a5-submission/snapshot also works when you want the history. Put the printed commit in the revision field of your report.

Follow the safety practices in syllabus §9:

  • Run the agent in a dedicated directory or container.
  • Keep residency, employer and cloud credentials out of that environment.
  • Keep keys out of repositories, prompts, screenshots and logs.
  • Run injection demonstrations offline against the local machine only.

Use the mature coding-agent seat supplied for the course. Do not run an unknown generated command on a credential-bearing host.

Published anchors

Give each dimension the highest level whose observable anchor you can show at the named revision. These anchors are the complete vocabulary for this assignment.

Dimension Level 1 Level 2 Level 3 Level 4
Structure No behaviour-to-file map A map exists but an entry is wrong Map is accurate and the change point has one responsibility A check catches a deliberate boundary violation
Documentation No runnable context file Commands are listed but one fails or an invariant is absent Commands run and three invariants name their checks A check detects a stale documented command
Tests No runnable coverage for the changed path A run passes but is slow, nondeterministic or misses a deliberate break Two repeatable runs pass and a deliberate break fails Mutation evidence explains every surviving mutant
Checks No automatic gate A gate is manual or already failing One offline gate command stops a bad change A pre-commit gate stops both an unused import and a secret canary

Tasks

  1. Choose a revision. Record the commit printed by git -C path/to/a4-repo rev-parse HEAD, and the snapshot directory if you made one.
  2. Generate. Ask the agent to suggest a level and evidence for each dimension. Treat each suggestion as claims.
  3. Disprove. For every claim, choose an observation that could show it false; run commands and open files at the named revision.
  4. Score. Record the highest level whose anchors are all evidenced for Structure, Documentation, Tests and Checks. Name the observation that would change each score.
  5. Review. Recheck one proposed agent claim yourself and record whether the evidence confirms, refutes or leaves it unresolved. Finish the decision line.

Generation is a proposed score; verification is your command output, file reading and deliberate break where safe. A score without evidence stays unsupported.

Deliverables

Submit the four-row scorecard.md, evidence files or excerpts, the checked agent proposal and report.json; set scorecard_path to that file. Run:

python practice/a5/check.py path/to/submission

Grading

Criteria are reproducible anchors; correct distinction between evidence and inference; a concrete change condition for each dimension; and a checked proposal. The public check does not authenticate commits or assign a readiness status.

AI use rules

Green: proposing levels, probes and prose. Yellow: using generated probe plans after reading and adapting them. Red: choosing the final levels, evidence interpretation and decision. Attach one AI-use note ID.

Estimated time

Published after a timed pilot. Record predicted and actual time, including the time spent rechecking one claim.

Work record outline

A5 report
revision: TODO
scorecard_path: TODO
dimensions: [{name: structure|documentation|tests|checks, level: 1..4, anchor: TODO, evidence: [TODO], would_change: TODO}]
review_evidence: [TODO]
explanation: TODO
ai_use_note: TODO
decision: accept | narrow | recheck | stop
because: evidence ID(s)
next: TODO