WEBVTT

1
00:00:00.000 --> 00:00:03.851
Here is the complete record of an AI agent repairing a bug.

2
00:00:03.851 --> 00:00:04.830
It read a file.

3
00:00:04.830 --> 00:00:05.874
It wrote a file.

4
00:00:05.874 --> 00:00:09.137
Then it replied: done, and asked for nothing more.

5
00:00:09.483 --> 00:00:11.132
So, is the bug fixed?

6
00:00:11.467 --> 00:00:13.576
Nothing in this record can tell us.

7
00:00:13.576 --> 00:00:17.552
By the end of this video, you'll see which observation is missing,

8
00:00:17.552 --> 00:00:20.987
and why an agent saying "done" is a claim, not a verdict.

9
00:00:22.033 --> 00:00:25.945
First, let's separate two things we usually blur together.

10
00:00:26.283 --> 00:00:28.435
A language model produces text.

11
00:00:28.435 --> 00:00:29.755
That's all it does.

12
00:00:29.755 --> 00:00:35.518
Sometimes that text is shaped like a request: call this tool, with these arguments.

13
00:00:35.518 --> 00:00:37.393
We'll call that a proposal.

14
00:00:37.733 --> 00:00:42.394
The program around the model, the controller, decides what happens next.

15
00:00:42.394 --> 00:00:46.731
It checks the call: a tool it knows, the argument names it expects,

16
00:00:46.731 --> 00:00:49.320
a path it allows, and text it can write.

17
00:00:49.320 --> 00:00:52.234
Only then does the proposal become an action.

18
00:00:52.583 --> 00:00:55.546
When a call fails those checks, nothing runs.

19
00:00:55.546 --> 00:00:59.365
The controller sends back a refusal as that call's result,

20
00:00:59.365 --> 00:01:01.933
and the model sees it on its next turn.

21
00:01:01.933 --> 00:01:06.608
The model can ask for anything; the controller decides what is allowed.

22
00:01:07.648 --> 00:01:12.263
Let's watch the loop on the smallest task we could think of.

23
00:01:12.263 --> 00:01:15.725
A file, result.txt, contains the number five.

24
00:01:15.725 --> 00:01:17.494
It should contain four.

25
00:01:17.832 --> 00:01:20.339
On turn one, the model calls read.

26
00:01:20.339 --> 00:01:24.984
The controller runs it, and sends back the actual result: five.

27
00:01:24.984 --> 00:01:28.966
Every later request carries it: the file, not a guess.

28
00:01:29.315 --> 00:01:31.578
On turn two, the call is: write four.

29
00:01:31.578 --> 00:01:35.004
The controller writes it, and the tool reports: written.

30
00:01:35.348 --> 00:01:36.494
Should we stop now?

31
00:01:36.494 --> 00:01:37.338
It's tempting.

32
00:01:37.338 --> 00:01:39.026
The file probably says four.

33
00:01:39.026 --> 00:01:42.101
But "written" only tells us a side effect happened.

34
00:01:42.101 --> 00:01:45.356
It doesn't tell us the result meets the specification.

35
00:01:45.698 --> 00:01:48.259
On turn three, the call is: verify.

36
00:01:48.259 --> 00:01:53.160
A fixed check compares the file to the spec, and reports: verified.

37
00:01:53.160 --> 00:01:57.111
That result, under the call's own ID, is the evidence.

38
00:01:57.448 --> 00:02:02.626
On turn four, the model asks for nothing more, so the loop stops with final.

39
00:02:02.626 --> 00:02:06.441
The model proposes, the controller decides, a tool acts,

40
00:02:06.441 --> 00:02:09.848
and the observation goes back for the next choice.

41
00:02:09.848 --> 00:02:12.437
That circle is the core of this agent.

42
00:02:13.480 --> 00:02:16.559
Now rewind, and let the model make a mistake.

43
00:02:16.559 --> 00:02:19.569
On turn two, it writes five instead of four.

44
00:02:19.569 --> 00:02:21.280
The write still succeeds.

45
00:02:21.280 --> 00:02:23.332
The tool still says "written".

46
00:02:23.332 --> 00:02:25.590
Only the check notices: it fails.

47
00:02:25.930 --> 00:02:30.795
Now the model replies with no calls: "Verified." The loop stops with final,

48
00:02:30.795 --> 00:02:32.158
as the contract says.

49
00:02:32.158 --> 00:02:36.244
But that reply cannot change the result under the check's call,

50
00:02:36.244 --> 00:02:38.968
however convincing its explanation sounds.

51
00:02:39.314 --> 00:02:41.388
A message can describe the result.

52
00:02:41.388 --> 00:02:43.036
Only the check observes it.

53
00:02:43.036 --> 00:02:47.429
So report verified only when a passing check comes after the last write.

54
00:02:48.462 --> 00:02:50.143
So how does this loop end?

55
00:02:50.143 --> 00:02:54.151
There are only two stop reasons, and neither one is the check.

56
00:02:54.496 --> 00:02:57.028
Final: a reply had no tool calls.

57
00:02:57.028 --> 00:03:01.171
Step limit: the turn budget, four by default, ran out.

58
00:03:01.171 --> 00:03:05.544
The contract has no repeat detector, so without a budget,

59
00:03:05.544 --> 00:03:09.380
a model that keeps asking could go around forever.

60
00:03:09.729 --> 00:03:14.755
The check is read separately, from the history: the newest verify result,

61
00:03:14.755 --> 00:03:17.715
or not run, if a later write made it stale.

62
00:03:17.715 --> 00:03:22.534
Here, the check failed, the model wrote again, and the budget ran out.

63
00:03:22.534 --> 00:03:24.186
Stop reason: step limit.

64
00:03:24.186 --> 00:03:25.219
Check: not run.

65
00:03:25.219 --> 00:03:28.317
And that failure must be kept in any summary.

66
00:03:28.662 --> 00:03:31.053
Final means the conversation stopped.

67
00:03:31.053 --> 00:03:32.991
Verified means a check passed.

68
00:03:32.991 --> 00:03:36.674
The two are independent: in the notes' budget experiment,

69
00:03:36.674 --> 00:03:40.873
a three-turn run stops at the step limit with its check verified.

70
00:03:40.873 --> 00:03:42.618
Keep those two words apart.

71
00:03:43.662 --> 00:03:45.463
Now look at the tools.

72
00:03:45.463 --> 00:03:47.673
Each has a narrow contract.

73
00:03:47.673 --> 00:03:49.883
Read takes only result.txt.

74
00:03:49.883 --> 00:03:55.040
Write takes at most thirty-two characters, only for result.txt.

75
00:03:55.040 --> 00:03:58.314
Bash runs only the fixed verify command.

76
00:03:58.662 --> 00:04:04.016
Any other tool, argument name, path or command is refused before any effect,

77
00:04:04.016 --> 00:04:07.538
and the refusal goes back under the call's own ID.

78
00:04:07.538 --> 00:04:13.104
In our run of the fixture's tool function, each of these six came back refused,

79
00:04:13.104 --> 00:04:15.288
and result.txt still held five.

80
00:04:15.629 --> 00:04:18.152
So the name bash does not grant a shell.

81
00:04:18.152 --> 00:04:21.305
The adapter runs the literal verification program,

82
00:04:21.305 --> 00:04:24.270
and never puts model text into a shell command.

83
00:04:24.270 --> 00:04:27.991
And this dispatcher only limits calls that pass through it:

84
00:04:27.991 --> 00:04:30.640
the launcher supplies the outer isolation.

85
00:04:31.678 --> 00:04:35.209
When a project publishes its agent's instructions,

86
00:04:35.209 --> 00:04:40.859
read them as an engineer reads a configuration file, not as guaranteed behavior.

87
00:04:40.859 --> 00:04:45.660
Sort every sentence into four bins: a role or goal, a tool contract,

88
00:04:45.660 --> 00:04:49.686
a stopping or escalation rule, and a request for caution.

89
00:04:50.028 --> 00:04:52.897
Then ask of each one: what enforces this?

90
00:04:52.897 --> 00:04:54.997
A schema, the tool's own code,

91
00:04:54.997 --> 00:04:59.405
or a sandbox or permission check outside the model can enforce.

92
00:04:59.405 --> 00:05:02.274
A sentence that begins "never" only asks.

93
00:05:02.274 --> 00:05:07.032
Our loop makes the same split: the checks before any effect enforce,

94
00:05:07.032 --> 00:05:10.671
while the contract text in the prompt only proposes.

95
00:05:11.012 --> 00:05:12.449
Two habits carry over.

96
00:05:12.449 --> 00:05:15.063
A tool definition is part of the prompt.

97
00:05:15.063 --> 00:05:19.834
And long instructions are not evidence of safety: count what is enforced,

98
00:05:19.834 --> 00:05:21.140
not what is written.

99
00:05:21.140 --> 00:05:25.845
And one more difference: your four-turn loop never compacts its history;

100
00:05:25.845 --> 00:05:27.544
a production agent has to.

101
00:05:28.577 --> 00:05:34.430
So far, every request has carried the whole history: the task, every reply,

102
00:05:34.430 --> 00:05:36.147
and every tool result.

103
00:05:36.494 --> 00:05:40.351
A context window caps how much one request can carry,

104
00:05:40.351 --> 00:05:43.627
and each turn sends every earlier byte again.

105
00:05:43.627 --> 00:05:46.756
Four turns over a one-line file fit easily.

106
00:05:46.756 --> 00:05:48.940
A repository repair would not.

107
00:05:49.277 --> 00:05:52.303
Three kinds of content take most of the room.

108
00:05:52.303 --> 00:05:56.809
Tool output, like file reads and test logs, is usually the largest,

109
00:05:56.809 --> 00:05:58.624
and goes out of date first.

110
00:05:58.624 --> 00:06:01.314
Failed attempts stay, with their errors.

111
00:06:01.314 --> 00:06:03.735
Repeated reads add a copy each time.

112
00:06:03.735 --> 00:06:08.039
And the instructions and tool definitions go with every request.

113
00:06:09.077 --> 00:06:12.109
Overflowing the window is the obvious failure.

114
00:06:12.109 --> 00:06:13.691
Three others come first.

115
00:06:13.691 --> 00:06:17.250
Here is a longer history, from this week's self-check.

116
00:06:17.593 --> 00:06:18.851
Stale observations.

117
00:06:18.851 --> 00:06:24.545
The first read of the status file stays in the history after the agent has changed it.

118
00:06:24.545 --> 00:06:28.915
Nothing marks it out of date, so a later step can act on old text.

119
00:06:28.915 --> 00:06:30.703
Contradictions are similar:

120
00:06:30.703 --> 00:06:34.741
a failed attempt and the decision that replaced it both stay.

121
00:06:35.077 --> 00:06:36.264
Buried instructions.

122
00:06:36.264 --> 00:06:40.480
"Do not edit the view file" arrived once, inside a file the agent read.

123
00:06:40.480 --> 00:06:43.329
It now sits under everything that came after it,

124
00:06:43.329 --> 00:06:47.604
and nothing in the loop makes the model give it weight many turns later.

125
00:06:47.943 --> 00:06:51.294
Now clear every older tool result to a short stub,

126
00:06:51.294 --> 00:06:53.572
and keep the agent's own messages.

127
00:06:53.572 --> 00:06:57.526
The failure survives, but only in the agent's own sentence.

128
00:06:57.526 --> 00:06:59.738
The view-file constraint is gone.

129
00:06:59.738 --> 00:07:03.222
And nothing in what's left shows that it is missing.

130
00:07:04.260 --> 00:07:07.981
Summarising replaces older turns with written text.

131
00:07:07.981 --> 00:07:13.014
A useful summary keeps six things: the task and its acceptance check;

132
00:07:13.014 --> 00:07:18.923
each constraint and decision, with its source; what was tried, and how it failed;

133
00:07:18.923 --> 00:07:24.322
open failures, with the file version the check ran on; the files involved,

134
00:07:24.322 --> 00:07:26.875
and their state; and the next step.

135
00:07:26.875 --> 00:07:31.544
It drops raw tool output, which the agent can read or run again.

136
00:07:31.893 --> 00:07:33.044
Now generate one.

137
00:07:33.044 --> 00:07:36.023
Here is a summary written from that history.

138
00:07:36.023 --> 00:07:38.256
It reads well, and it is a claim.

139
00:07:38.256 --> 00:07:42.792
It kept the task, the edited file, and a next step: fix the lookup.

140
00:07:42.792 --> 00:07:47.056
It dropped the constraint, and the attempt that already failed.

141
00:07:47.393 --> 00:07:50.445
So the next step repeats the strip-only edit.

142
00:07:50.445 --> 00:07:55.532
We ran that edit on scratch files, and the same check failed on "OK" again.

143
00:07:55.877 --> 00:07:58.382
Check a summary like any generated work.

144
00:07:58.382 --> 00:08:03.268
For each item it should keep, find the message in the transcript it came from.

145
00:08:03.268 --> 00:08:08.217
Two have no match: the constraint, from the notes file, and the failed attempt,

146
00:08:08.217 --> 00:08:09.908
with the version it ran on.

147
00:08:09.908 --> 00:08:12.351
Add them back, each tied to its source.

148
00:08:12.693 --> 00:08:15.305
Then rerun the same check on the next step.

149
00:08:15.305 --> 00:08:19.436
Our next edit strips and lowercases, and leaves the view file alone.

150
00:08:19.436 --> 00:08:20.468
The check passes.

151
00:08:20.468 --> 00:08:24.234
But the check alone would not have caught the lost constraint.

152
00:08:24.234 --> 00:08:26.057
The transcript comparison did.

153
00:08:27.093 --> 00:08:29.681
When should you compact, and when restart?

154
00:08:29.681 --> 00:08:34.116
Compact when the task continues and the newest raw results still matter.

155
00:08:34.116 --> 00:08:40.707
Restart with a fresh context and a written handoff when most of the history is stale or contradicts itself,

156
00:08:40.707 --> 00:08:45.820
when one phase of the work has ended, or when the agent keeps failing the same way.

157
00:08:46.160 --> 00:08:49.019
A handoff holds the same items as a summary,

158
00:08:49.019 --> 00:08:52.268
written so that a new session can act on it alone.

159
00:08:52.268 --> 00:08:56.946
After either one, reread the files the next edit touches before editing.

160
00:08:57.993 --> 00:09:02.254
So, back to where we started: read, write, done.

161
00:09:02.593 --> 00:09:06.684
The missing observation was a passing check after the last write.

162
00:09:06.684 --> 00:09:08.760
With it, you may report verified.

163
00:09:08.760 --> 00:09:13.228
Without it, "done" is just a claim, and so is a summary nobody checked.

164
00:09:13.577 --> 00:09:16.867
One question to try before you compare with a partner.

165
00:09:16.867 --> 00:09:20.705
An agent writes the status file, and the acceptance tests fail.

166
00:09:20.705 --> 00:09:23.812
It writes the file again, and the turn budget ends.

167
00:09:23.812 --> 00:09:25.335
The new file looks right.

168
00:09:25.335 --> 00:09:28.685
What stop reason and check status does the record show?

169
00:09:28.685 --> 00:09:30.208
What must a summary keep?

170
00:09:30.208 --> 00:09:33.315
Which observation would let you report it verified?

171
00:09:33.315 --> 00:09:36.483
Write your answer, and the next check you would run.