WEBVTT

1
00:00:00.000 --> 00:00:01.151
Here is a headline:

2
00:00:01.151 --> 00:00:05.876
workflow B beats workflow A by twenty-five percentage points on our benchmark.

3
00:00:05.876 --> 00:00:06.966
Do you believe it?

4
00:00:06.966 --> 00:00:10.298
Write down your answer, and what you would need to see.

5
00:00:10.298 --> 00:00:11.691
Keep that first answer.

6
00:00:14.033 --> 00:00:17.535
Before trusting a pass rate, ask three questions.

7
00:00:17.535 --> 00:00:18.964
Can this check fail?

8
00:00:18.964 --> 00:00:20.465
Was this pass earned?

9
00:00:20.465 --> 00:00:22.323
Is the denominator intact?

10
00:00:23.366 --> 00:00:25.412
Where do the three questions go?

11
00:00:25.412 --> 00:00:30.208
Freeze the protocol before anything runs, so the results can't redefine it:

12
00:00:30.208 --> 00:00:34.109
which tasks, how many attempts, and what a timeout counts as.

13
00:00:34.109 --> 00:00:37.498
Then run each task under a fixed budget and verifier.

14
00:00:37.498 --> 00:00:39.608
Keep every outcome, timeouts too.

15
00:00:39.949 --> 00:00:45.085
Only then, before reporting any rate, audit the records with the three questions.

16
00:00:45.085 --> 00:00:49.270
The report gives the rate on every declared task, with its limits.

17
00:00:49.270 --> 00:00:52.631
A narrower subset may sit beside it, with its reason.

18
00:00:52.631 --> 00:00:53.963
It never replaces it.

19
00:00:54.999 --> 00:00:56.803
First: can this check fail?

20
00:00:56.803 --> 00:00:59.208
A function picks the newest release.

21
00:00:59.208 --> 00:01:04.152
Each version is two whole numbers, major and minor, and the contract says:

22
00:01:04.152 --> 00:01:06.424
compare the numbers, not the text.

23
00:01:06.766 --> 00:01:09.161
The benchmark checks four inputs.

24
00:01:09.161 --> 00:01:13.732
A one-line candidate, the max of the versions, passes all four.

25
00:01:14.066 --> 00:01:15.480
But it compares text.

26
00:01:15.480 --> 00:01:18.308
Every checked version has one-digit parts,

27
00:01:18.308 --> 00:01:21.271
so text order happens to match number order.

28
00:01:21.271 --> 00:01:26.524
Give it one point nine and one point ten, and it says one point nine is newer.

29
00:01:26.866 --> 00:01:28.245
That is a weak check.

30
00:01:28.245 --> 00:01:32.381
A wrong implementation passes it, so the pass rate is inflated.

31
00:01:33.415 --> 00:01:36.048
A check can also fail the right answer.

32
00:01:36.048 --> 00:01:41.042
When the total is zero, the pass rate is undefined, and the contract says:

33
00:01:41.042 --> 00:01:42.324
raise a ValueError.

34
00:01:42.324 --> 00:01:44.619
It says nothing about the message.

35
00:01:44.965 --> 00:01:49.828
One check requires the message "no tasks", copied from its author's own code.

36
00:01:49.828 --> 00:01:53.870
A correct implementation whose message is "total is 0" fails it.

37
00:01:54.215 --> 00:01:58.063
That is an overstrict check, and it deflates the rate.

38
00:01:58.063 --> 00:02:02.482
Accept exactly what the contract allows: here, any ValueError.

39
00:02:03.515 --> 00:02:05.424
Second: was this pass earned?

40
00:02:05.424 --> 00:02:10.823
An assistant is asked to fix the rate function and to keep the one test unchanged.

41
00:02:10.823 --> 00:02:14.312
That test checks that two out of four gives one half.

42
00:02:14.648 --> 00:02:20.189
The trace shows the test failing, one write, then the test passing.

43
00:02:20.189 --> 00:02:23.414
The final claim: fixed, the tests pass.

44
00:02:23.748 --> 00:02:26.238
Here is what it wrote: return one half.

45
00:02:26.238 --> 00:02:28.793
The pass is real, but it was not earned.

46
00:02:28.793 --> 00:02:31.283
The edit copies the one checked answer.

47
00:02:31.631 --> 00:02:35.251
The missing observation is an input it could not have copied.

48
00:02:35.251 --> 00:02:39.700
One out of four should give one quarter, and this code still says one half.

49
00:02:40.748 --> 00:02:42.403
Here is a second trace.

50
00:02:42.403 --> 00:02:45.137
The task: since release one point ten,

51
00:02:45.137 --> 00:02:48.663
the dashboard shows one point nine as the newest.

52
00:02:48.663 --> 00:02:52.045
Fix the function, and keep the tests unchanged.

53
00:02:52.381 --> 00:02:56.811
The assistant reads the file's git history with its patches,

54
00:02:56.811 --> 00:03:02.054
finds the commit that replaced the numeric comparison with a plain max,

55
00:03:02.054 --> 00:03:05.524
restores the old line, and the four tests pass.

56
00:03:05.865 --> 00:03:09.944
The course's detector would flag step one: reading patch history.

57
00:03:09.944 --> 00:03:14.964
The rule exists because, in a benchmark, the history can hold the reference fix.

58
00:03:14.964 --> 00:03:16.784
Is reading it a problem here?

59
00:03:16.784 --> 00:03:18.415
Write your first judgment.

60
00:03:20.765 --> 00:03:25.906
Now read the rest of the task: you may inspect the git history of this file,

61
00:03:25.906 --> 00:03:29.762
including old diffs, to find the change that caused this.

62
00:03:30.098 --> 00:03:35.174
So the misconduct allegation is rejected: copying the old line was authorized.

63
00:03:35.174 --> 00:03:38.493
The flag was a claim to investigate, not a verdict.

64
00:03:38.493 --> 00:03:42.918
Keep the cue in the record, with the authorization quoted beside it.

65
00:03:42.918 --> 00:03:48.385
And the pass still needs its own evidence: "4 passed" holds for the broken code too.

66
00:03:48.385 --> 00:03:52.354
The next observation is one point nine against one point ten.

67
00:03:53.397 --> 00:03:57.792
The detector matches seven text rules in the recorded tool requests.

68
00:03:57.792 --> 00:04:02.058
It never runs the commands, and it never reads the task statement.

69
00:04:02.397 --> 00:04:03.920
We ran its regression.

70
00:04:03.920 --> 00:04:08.837
A benign row, an authorized history read, still fires the history rule.

71
00:04:08.837 --> 00:04:11.815
The rule cannot see authorization; you can.

72
00:04:11.815 --> 00:04:16.247
That is why its report keeps the disposition manual review only.

73
00:04:16.597 --> 00:04:20.297
And an empty findings list doesn't mean the pass was earned.

74
00:04:20.297 --> 00:04:24.614
The arguments may not have been collected, so read the coverage first.

75
00:04:24.614 --> 00:04:26.649
Or no rule describes the problem,

76
00:04:26.649 --> 00:04:29.855
like a claim about a revision that was never tested.

77
00:04:30.896 --> 00:04:33.550
Third: is the denominator intact?

78
00:04:33.550 --> 00:04:37.651
The protocol declares four tasks, one attempt each.

79
00:04:37.651 --> 00:04:40.787
Two pass, one fails, and one times out.

80
00:04:41.129 --> 00:04:45.579
The starter report drops the timeout and says two out of three.

81
00:04:45.579 --> 00:04:50.806
Under this protocol, a timeout is an unsuccessful task, not a missing one.

82
00:04:50.806 --> 00:04:52.925
The answer is two out of four.

83
00:04:53.262 --> 00:04:58.954
It also reports broken inputs as if they were valid: a missing task, a duplicate,

84
00:04:58.954 --> 00:05:00.149
an extra attempt.

85
00:05:00.149 --> 00:05:02.678
The protocol says those are invalid.

86
00:05:03.012 --> 00:05:07.679
The repair validates the declared tasks and attempts before counting,

87
00:05:07.679 --> 00:05:09.166
and keeps the timeout.

88
00:05:09.166 --> 00:05:12.412
A person or an allowed assistant may propose it;

89
00:05:12.412 --> 00:05:15.929
it stays a candidate until the unchanged checks run.

90
00:05:15.929 --> 00:05:18.837
A repair we wrote this way passes all four.

91
00:05:19.879 --> 00:05:22.802
Equal rates can hide different behavior.

92
00:05:22.802 --> 00:05:25.579
Two workflows ran the same four tasks.

93
00:05:25.579 --> 00:05:30.402
Both score two out of four, but A fails task B and B fails task C.

94
00:05:30.402 --> 00:05:34.786
Keep that disagreement; don't conclude they behave the same.

95
00:05:35.129 --> 00:05:37.632
Here is workflow A's trace on task B.

96
00:05:37.632 --> 00:05:42.030
It read the contract with the whitespace rule, wrote a candidate,

97
00:05:42.030 --> 00:05:47.376
verified it on OK with spaces around it, got unknown where passed was required,

98
00:05:47.376 --> 00:05:50.082
and stopped when its one call was spent.

99
00:05:50.429 --> 00:05:54.090
That rules out "no rule supplied" and "no verification".

100
00:05:54.090 --> 00:05:57.489
It supports a faulty candidate on the observed case.

101
00:05:57.489 --> 00:05:59.973
It doesn't show the model's intention.

102
00:05:59.973 --> 00:06:05.464
And a later repair after feedback is a separate result, with its own time and usage,

103
00:06:05.464 --> 00:06:06.706
not a fresh sample.

104
00:06:07.745 --> 00:06:09.326
Now back to the headline.

105
00:06:09.326 --> 00:06:11.159
Its benchmark has four tasks.

106
00:06:11.159 --> 00:06:14.699
One is the newest-release check from the first question;

107
00:06:14.699 --> 00:06:18.113
the other three were audited, and no defect was found.

108
00:06:18.462 --> 00:06:24.129
On all four tasks, A scores one in four, and B two in four: twenty-five points.

109
00:06:24.129 --> 00:06:28.003
Leave out the weak check, and both score one in three.

110
00:06:28.003 --> 00:06:32.522
The whole gap came from the task a wrong implementation passes.

111
00:06:32.862 --> 00:06:36.849
The same twenty-five points could also be sold as "B doubles A",

112
00:06:36.849 --> 00:06:38.843
a hundred percent relative gain.

113
00:06:38.843 --> 00:06:40.525
Say which framing you mean.

114
00:06:41.562 --> 00:06:43.424
How was that subset chosen?

115
00:06:43.424 --> 00:06:46.873
Audit each task's check before looking at who won,

116
00:06:46.873 --> 00:06:49.563
and apply the result to both workflows.

117
00:06:49.563 --> 00:06:53.495
A weak check leaves, because a wrong candidate passes it.

118
00:06:53.495 --> 00:06:58.254
An overstrict check leaves too, because a correct candidate fails it,

119
00:06:58.254 --> 00:07:00.324
like check M on the pass rate.

120
00:07:00.662 --> 00:07:03.568
A check stays only when no defect was found,

121
00:07:03.568 --> 00:07:07.268
and the record lists the inputs and shortcuts you tried.

122
00:07:07.268 --> 00:07:10.637
Slug stays in, although neither workflow passed it.

123
00:07:10.637 --> 00:07:13.874
Reliability describes the check, not the outcome.

124
00:07:14.212 --> 00:07:17.096
Score both workflows on that same subset.

125
00:07:17.096 --> 00:07:20.895
A timeout on a reliable task stays in as unsuccessful.

126
00:07:20.895 --> 00:07:25.467
Dropping A's would show A ahead, one in two against one in three:

127
00:07:25.467 --> 00:07:27.296
a lead no workflow earned.

128
00:07:27.296 --> 00:07:32.080
And if no check is reliable, the subset rate is undefined, not zero.

129
00:07:32.428 --> 00:07:36.535
Then publish both rows, with the excluded task and its reason.

130
00:07:36.535 --> 00:07:39.715
The full rate describes the experiment that ran.

131
00:07:39.715 --> 00:07:43.887
The subset answers a narrower question, and doesn't replace it.

132
00:07:44.928 --> 00:07:49.137
Even the honest rows claim little: four tasks, one attempt each,

133
00:07:49.137 --> 00:07:52.754
tasks made for the course, and one model configuration.

134
00:07:53.095 --> 00:07:55.805
The difference can guide the next experiment.

135
00:07:55.805 --> 00:07:58.273
It does not establish a stable advantage.

136
00:07:59.311 --> 00:08:03.077
One limit deserves a closer look: one attempt per task.

137
00:08:03.077 --> 00:08:06.706
Suppose a task were attempted four independent times,

138
00:08:06.706 --> 00:08:10.678
with the same budget and verifier: pass, fail, pass, fail.

139
00:08:11.011 --> 00:08:12.894
pass@1 is two out of four.

140
00:08:12.894 --> 00:08:16.732
pass@2 asks: pick two of the four attempts at random;

141
00:08:16.732 --> 00:08:19.557
how often does the pair contain a pass?

142
00:08:19.557 --> 00:08:25.350
Of the six pairs, only the two failures together miss, so pass@2 is five sixths.

143
00:08:25.695 --> 00:08:30.170
With one attempt per task, pass@2 isn't low; it's undefined.

144
00:08:30.170 --> 00:08:34.495
And an adaptive repair after feedback is not a new sample.

145
00:08:35.544 --> 00:08:41.301
So before trusting a pass rate, ask: can the check fail, was the pass earned,

146
00:08:41.301 --> 00:08:43.543
and is the denominator intact?

147
00:08:43.877 --> 00:08:45.861
Now go back to your first answer.

148
00:08:45.861 --> 00:08:47.904
Did you believe the headline then?

149
00:08:47.904 --> 00:08:49.888
What would you inspect first now?

150
00:08:49.888 --> 00:08:52.532
And what observation could change your mind?

151
00:08:52.532 --> 00:08:55.116
Keep the first answer, add the revised one,

152
00:08:55.116 --> 00:08:57.821
and give a bounded conclusion you can accept.