The round accidentally ran every level twice, which turned into the most useful
thing it produced: 45 retest pairs on this exact set. 84% of them replayed the
identical action sequence, so the agent is near-deterministic and a second run
of a level buys almost nothing — the mechanical metrics retest at r = 1.00 for
that reason, not because they survived a hard test.
The ratings do not. Difficulty retests at r = 0.55 here, against the 0.93 this
project had on record from a smaller sample, and every pair that disagreed had
played the same trajectory: same moves, same outcome, different number. The
report now prints the reliability it measured rather than quoting the older
figure, and says which one it replaces.
Averaged by band over ~20 runs, though, the ratings finally track the curve —
2.00 / 2.05 / 2.40 with stuck at 1.21 / 1.15 / 1.74, where round 1 was a flat
line. So the rule is not "ratings are useless", it is "ratings are a band
statistic, never a per-level rank", and that is what the rules section says.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
