Regenerating levels/prototypes/d5.json renumbers every level, so the previous
build — which resolved runs through the current set's ordinals — silently
dropped every run played against an earlier set. The report I committed one
commit ago was built that way: 130 runs instead of 229, and the test-retest
table down to 7 paired levels from 18.
Levels are now keyed by id throughout. Runs carry their own level, so ids the
current set no longer contains are read back out of the archive and given an
ordinal after the live ones; the cross-model table intersects on ids rather
than assuming two sets number the same level alike.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every model that played the D5 set complained about the same moment: two
pieces left, interlocked, and nothing on the board saying which way to turn.
codex-spark took it to the end — it cycled all six orientations on L18 and
gave up on a level that was solvable, with clarity rated 5. Replaying the
position shows exactly one of the six orientations releases anything.
deadTurnAudit now reports endgameMinOpen: across reachable states with three
pieces or fewer left, the fewest orientations that release anything. It is
computed inside the walk that was already happening, so it costs nothing. The
generator requires at least two.
The gate picks out precisely the four levels every model stalled on, with no
false positives — the other fourteen already had two or three. Regenerating
drops the spiral family entirely: its structure produces single-exit endgames
by construction, not by unlucky seed.
Measured on the regenerated set, rule bot first and then grok-4.6 xhigh over
all 18:
blind rotations 1.17 -> 0.29 per level
extra rotations 1.22 -> 0.35
dead taps 2 -> 0
run time 55.5s -> 34.7s
stuck rating 1.89 -> 1.41 (this metric repeats at r = 0.85)
"which way do I turn" complaints 22% -> 12% of levels
The gate is a floor, not a difficulty dial: difficulty also fell, 2.39 -> 2.12,
because the levels it removes are the hard ones. And it treats a symptom — the
t0/t3 interlock is what the four-piece chain in createDeepLock builds by
construction, so the complaint is halved rather than gone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One batch of 18 runs was recorded as codex-spark at xhigh but had actually run
as gpt-6-astra at low effort: the CLI config was rewritten between the check
and the batch, and codex resolved the new model silently. The archives are
relabelled from the CLI's own startup header rather than from what the agent
was told to call itself, and the report now says to trust that header over the
self-reported name. The re-run verifies the resolved model after every level
and aborts the batch on a mismatch.
With the real codex-spark batch added, five independent runs of four models
from three vendors now cover the same 18 levels on the fixed board, and 16 of
17 comparable levels agree exactly on the rotation triple.
codex-spark gave up on L18 after cycling all six orientations, reporting no
exit anywhere — replaying the position shows orientation 1 releases t0, and
that it is the only one of the six that does. The level is solvable and the
routes are legible (it rated clarity 5), but nothing on the board says which
way to turn: a 1-in-6 guess with a dead end behind every other door. That is
the terminal-interlock complaint the other models kept raising, at its limit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
design-review.html is the gameplay and UI read on the shipped campaign, with
boards rendered from the repo's own geometry rather than redrawn.
bench-reliability.html asks which of the bench's numbers are worth acting on,
and answers it by running the same 18 levels twice under identical conditions.
The mechanical rotation counters reproduce (r = 0.82-0.92) and three different
models agree on them level for level, so they measure the level rather than
the player — a 12-second rule-bot sweep gets the same answer as hours of
model time. Difficulty and stuck ratings also hold up (r = 0.93 / 0.85), but
the ratings for rotation, clarity and fun do not (0.32 / 0.19 / 0.56): asking
a model whether rotation matters is noise, while counting blind rotations is
not.
Every figure is computed from bench/runs by build-bench-report.mjs; nothing in
the page is hand-written, so it can be regenerated as runs accumulate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
