One batch of 18 runs was recorded as codex-spark at xhigh but had actually run
as gpt-6-astra at low effort: the CLI config was rewritten between the check
and the batch, and codex resolved the new model silently. The archives are
relabelled from the CLI's own startup header rather than from what the agent
was told to call itself, and the report now says to trust that header over the
self-reported name. The re-run verifies the resolved model after every level
and aborts the batch on a mismatch.
With the real codex-spark batch added, five independent runs of four models
from three vendors now cover the same 18 levels on the fixed board, and 16 of
17 comparable levels agree exactly on the rotation triple.
codex-spark gave up on L18 after cycling all six orientations, reporting no
exit anywhere — replaying the position shows orientation 1 releases t0, and
that it is the only one of the six that does. The level is solvable and the
routes are legible (it rated clarity 5), but nothing on the board says which
way to turn: a 1-in-6 guess with a dead end behind every other door. That is
the terminal-interlock complaint the other models kept raising, at its limit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
design-review.html is the gameplay and UI read on the shipped campaign, with
boards rendered from the repo's own geometry rather than redrawn.
bench-reliability.html asks which of the bench's numbers are worth acting on,
and answers it by running the same 18 levels twice under identical conditions.
The mechanical rotation counters reproduce (r = 0.82-0.92) and three different
models agree on them level for level, so they measure the level rather than
the player — a 12-second rule-bot sweep gets the same answer as hours of
model time. Difficulty and stuck ratings also hold up (r = 0.93 / 0.85), but
the ratings for rotation, clarity and fun do not (0.32 / 0.19 / 0.56): asking
a model whether rotation matters is noise, while counting blind rotations is
not.
Every figure is computed from bench/runs by build-bench-report.mjs; nothing in
the page is hand-written, so it can be regenerated as runs accumulate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
