WEBVTT

1
00:00:00.000 --> 00:00:02.515
A developer opens a build dashboard.

2
00:00:02.515 --> 00:00:06.287
The build service reports "OK", with spaces around it.

3
00:00:06.287 --> 00:00:08.382
The dashboard shows "unknown".

4
00:00:08.717 --> 00:00:13.019
Someone proposes a patch, and both of its existing tests pass.

5
00:00:13.019 --> 00:00:14.615
So, is the patch right?

6
00:00:14.950 --> 00:00:18.414
By the end of this video, you'll derive one more test,

7
00:00:18.414 --> 00:00:23.739
from the contract instead of the patch, and see why two green tests weren't enough.

8
00:00:24.783 --> 00:00:27.232
Start with the contract, not the code.

9
00:00:27.232 --> 00:00:29.230
The status arrives as a string.

10
00:00:29.567 --> 00:00:32.962
Ignore surrounding whitespace and letter case.

11
00:00:32.962 --> 00:00:34.290
"OK" means passed.

12
00:00:34.290 --> 00:00:35.766
"Fail" means failed.

13
00:00:35.766 --> 00:00:39.900
Anything else, including an empty string, means unknown.

14
00:00:40.233 --> 00:00:45.260
Each rule is something we can test, before we ever look at a patch.

15
00:00:46.299 --> 00:00:48.153
Here is the proposed patch.

16
00:00:48.153 --> 00:00:51.106
It lowercases the status, then looks it up.

17
00:00:51.106 --> 00:00:56.737
Unpadded "OK" becomes passed, "fail" becomes failed, and anything else is unknown.

18
00:00:57.083 --> 00:01:00.241
Two existing tests check "ok" and "fail".

19
00:01:00.241 --> 00:01:01.011
Both pass.

20
00:01:01.011 --> 00:01:03.399
Those are real, passing checks.

21
00:01:03.733 --> 00:01:05.856
But which rules do they exercise?

22
00:01:05.856 --> 00:01:09.201
Neither input has a space or a capital letter in it.

23
00:01:09.201 --> 00:01:11.581
The whitespace rule was never tested.

24
00:01:12.616 --> 00:01:14.975
So derive a test from the contract.

25
00:01:14.975 --> 00:01:17.740
The input is "OK", with spaces around it.

26
00:01:17.740 --> 00:01:22.055
The contract says the answer is passed, whatever the patch does.

27
00:01:22.399 --> 00:01:22.903
Run it.

28
00:01:22.903 --> 00:01:28.088
The patch lowercases the input, keeps the spaces, finds no matching key,

29
00:01:28.088 --> 00:01:29.528
and returns unknown.

30
00:01:29.528 --> 00:01:30.609
The test fails.

31
00:01:30.609 --> 00:01:32.697
The plausible patch is wrong.

32
00:01:33.032 --> 00:01:34.651
Not every new input helps.

33
00:01:34.651 --> 00:01:39.195
"Cancelled" returns unknown from the faulty patch and from a correct one.

34
00:01:39.195 --> 00:01:42.680
It exercises the fallback, but it can't tell them apart.

35
00:01:43.714 --> 00:01:48.916
The repair is one call: strip the surrounding whitespace before lowercasing.

36
00:01:49.264 --> 00:01:53.442
Now compare both versions on six cases taken from the contract.

37
00:01:53.442 --> 00:01:56.027
The first patch passes four of the six.

38
00:01:56.027 --> 00:01:57.751
The repair passes all six.

39
00:01:58.098 --> 00:02:02.525
Keep the two original tests, and keep the new ones as regressions.

40
00:02:02.525 --> 00:02:05.812
Six cases are observations about those behaviors,

41
00:02:05.812 --> 00:02:08.361
not a proof for every possible string.

42
00:02:09.397 --> 00:02:11.529
Why does this matter for agents?

43
00:02:11.529 --> 00:02:15.527
Suppose a model writes the patch, and also writes its tests.

44
00:02:15.864 --> 00:02:19.576
If both assume unpadded input, they agree with each other,

45
00:02:19.576 --> 00:02:21.368
and both miss the same rule.

46
00:02:21.368 --> 00:02:26.231
That's a possible shared omission, not a claim about how often models do it.

47
00:02:26.580 --> 00:02:31.760
Independent expected results come from somewhere else: the product contract,

48
00:02:31.760 --> 00:02:33.941
and cases you choose on purpose.

49
00:02:34.980 --> 00:02:37.800
Suppose a model had proposed this patch.

50
00:02:37.800 --> 00:02:39.492
How would it produce it?

51
00:02:39.492 --> 00:02:45.342
A language model processes tokens: word fragments, punctuation and code characters.

52
00:02:45.342 --> 00:02:50.911
At each step, it computes a distribution for the next token, given the context,

53
00:02:50.911 --> 00:02:52.039
and selects one.

54
00:02:52.039 --> 00:02:55.634
That token joins the context, and the step repeats.

55
00:02:55.980 --> 00:02:59.127
This can produce a useful multi-step algorithm.

56
00:02:59.127 --> 00:03:02.007
It does not mean the model has executed it.

57
00:03:02.007 --> 00:03:07.230
Valid Python can still implement a different rule from the one the user needs.

58
00:03:07.580 --> 00:03:13.503
And a token's likelihood is not a probability that the program satisfies the specification.

59
00:03:13.503 --> 00:03:18.644
The patch's behavior stays unobserved until an appropriate check actually runs.

60
00:03:19.680 --> 00:03:24.697
During ordinary inference, the workflow supplies a request and other context.

61
00:03:24.697 --> 00:03:30.496
Pasting a file or correcting an answer does not ordinarily update the model's parameters.

62
00:03:30.496 --> 00:03:32.841
Training changes learned parameters.

63
00:03:32.841 --> 00:03:36.816
Prompting changes what is available for the current response.

64
00:03:37.163 --> 00:03:38.909
Imagine two context packets.

65
00:03:38.909 --> 00:03:42.276
One holds only "fix the label" and the implementation.

66
00:03:42.276 --> 00:03:46.704
The other also holds "ignore surrounding whitespace" from the contract.

67
00:03:46.704 --> 00:03:49.385
The second supplies an explicit constraint.

68
00:03:49.385 --> 00:03:53.127
That does not mean the model will certainly succeed with it.

69
00:03:53.463 --> 00:03:55.394
Training is a different matter.

70
00:03:55.394 --> 00:04:00.438
Pretraining, supervised fine-tuning and reinforcement learning change parameters,

71
00:04:00.438 --> 00:04:02.369
and can improve coding ability.

72
00:04:02.369 --> 00:04:04.237
They still leave requirements,

73
00:04:04.237 --> 00:04:07.662
versions and execution conditions to check at use time.

74
00:04:08.696 --> 00:04:12.543
A coding agent combines a model with an execution workflow.

75
00:04:12.543 --> 00:04:14.434
The model proposes an action.

76
00:04:14.434 --> 00:04:19.063
The controller checks the tool name, arguments, permissions and budget.

77
00:04:19.063 --> 00:04:23.627
And an allowed tool performs the operation and returns an observation.

78
00:04:23.963 --> 00:04:26.724
Each event establishes something different.

79
00:04:26.724 --> 00:04:29.613
An allowed read passed a permission decision.

80
00:04:29.613 --> 00:04:31.604
A write means a write occurred.

81
00:04:31.604 --> 00:04:35.585
And a model saying the work is finished is a completion claim.

82
00:04:35.930 --> 00:04:39.410
A permission decision does not establish correctness,

83
00:04:39.410 --> 00:04:42.036
and a write result is not a test result.

84
00:04:42.036 --> 00:04:45.057
If a test request is denied or never finishes,

85
00:04:45.057 --> 00:04:49.850
a later "all tests passed" message cannot supply the missing observation.

86
00:04:49.850 --> 00:04:54.118
A safe next step is to inspect the refusal or the missing result,

87
00:04:54.118 --> 00:04:55.760
and get an allowed check.

88
00:04:56.096 --> 00:05:00.931
Today's loop has one model, one controller decision and one tool result.

89
00:05:00.931 --> 00:05:05.229
A production coding agent wraps the same loop in more machinery:

90
00:05:05.229 --> 00:05:09.661
tool definitions and budgets, a compacted record of earlier steps,

91
00:05:09.661 --> 00:05:12.750
context files and skills, hooks and a sandbox,

92
00:05:12.750 --> 00:05:15.369
and a specification with a staged plan.

93
00:05:15.369 --> 00:05:18.726
Weeks two to four take these layers one at a time.

94
00:05:18.726 --> 00:05:23.091
Each layer changes what the model sees, or what an action may do.

95
00:05:23.091 --> 00:05:25.710
None of them checks the result for you.

96
00:05:25.710 --> 00:05:30.276
A richer harness still needs a completed check on the current files.

97
00:05:31.312 --> 00:05:32.725
Now look one level up.

98
00:05:32.725 --> 00:05:37.092
The developer also needs a failed-build count that matches the rows.

99
00:05:37.092 --> 00:05:39.853
The caller labels each row with the helper,

100
00:05:39.853 --> 00:05:43.770
but it counts failures by comparing the raw status to "fail".

101
00:05:44.112 --> 00:05:47.364
So repairing only the helper leaves a disagreement.

102
00:05:47.364 --> 00:05:52.592
For a padded, mixed-case failure, the row says failed, while the count stays zero.

103
00:05:52.592 --> 00:05:55.142
Even all six helper tests would miss it.

104
00:05:55.479 --> 00:05:58.867
The proposed repair counts the labels already computed.

105
00:05:58.867 --> 00:06:01.763
A full-output regression then checks the count,

106
00:06:01.763 --> 00:06:04.104
and the unchanged IDs and their order.

107
00:06:04.104 --> 00:06:07.553
These are hand-derived expectations, not a recorded run.

108
00:06:07.553 --> 00:06:08.786
And notice the task:

109
00:06:08.786 --> 00:06:13.591
"make tests pass" alone would not have said which product behavior to protect.

110
00:06:14.629 --> 00:06:16.393
So, back to the dashboard.

111
00:06:16.393 --> 00:06:19.719
Two green tests did not show the patch was right.

112
00:06:19.719 --> 00:06:23.452
One test derived from the contract showed it was wrong.

113
00:06:23.795 --> 00:06:25.032
One question to try.

114
00:06:25.032 --> 00:06:29.299
A new service sends "completed", and our contract maps it to unknown.

115
00:06:29.299 --> 00:06:30.660
Should it mean passed?

116
00:06:30.660 --> 00:06:35.545
Before you change anything, write the question you would ask the product owner.