Round 3 tested one hypothesis from round 2's data: levels the agent rated 3
carried 59% more 反差 than the rest, so select for it. With both rounds now at
~100 runs, the answer is in and it is no. The back band went from 2.39 ± 0.10 to
2.19 ± 0.08, roughly two standard errors apart, and the levels averaging 2.5 or
better fell from 11 to 6. Contrast is real and it does correlate with being
rated harder; selecting on it does not produce harder levels.
The low retest reliability does not rescue the change either — r = 0.55 limits
ranking one level against another, not the mean of eighteen levels measured a
hundred times.
So round 2's levels are the ones handed over, rebuilt from its runs via
round-set --emitCampaign. select-50 cannot reproduce them: the pool it drew
from has been rebuilt since, so the same flags now return different levels. The
runs are the record.
Rule bot on the shipped set: 50/50 cleared, dead taps 0.00, blind rotations
0.12, worst blind streak 1. 198 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rule-bot table carries the third set now, which is the point of having a
rule bot at all: a full 50-level acceptance pass takes one second, so every
selection change can be checked against the previous one before any agent time
is spent. Blind rotations 0.24 → 0.12 → 0.14, worst streak 4 → 1 → 1, dead taps
zero throughout.
The band table shows all three rounds side by side, each against the levels it
actually played. Round 3 is reported as what it is: an unconfirmed hypothesis.
It raised contrast and held rotation depth, and the back band's rating did not
follow. On a rating that retests at 0.55 that settles nothing either way, so
the page says the two are indistinguishable and names the design reasons the
third set is the one being handed over — rather than quoting whichever number
makes the last change look good.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The round comparison was about to be wrong. levels/campaign-50-r2.json was
copied from levels/campaign-50.json to preserve "the set round 2 played", but
select-50 writes that path in place and several selection experiments ran
between the round and the copy. The file kept 28 of the 50 boards round 2 had
actually played and 46 of round 3's — so the comparison was scoring one round
against another round's levels, and it read as round 3 regressing.
Runs store the whole board and its campaign index, so a round is its own record
and cannot drift. round-set.mjs rebuilds it, the report uses that, and the
snapshot file is gone rather than left lying around looking authoritative.
With the rounds correctly matched, round 3 is not a regression and not an
improvement: back band 2.17 against round 2's 2.39, on a rating that retests at
0.55, which is not enough to separate seventeen levels. The report says that
rather than picking whichever reading flatters the change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The round accidentally ran every level twice, which turned into the most useful
thing it produced: 45 retest pairs on this exact set. 84% of them replayed the
identical action sequence, so the agent is near-deterministic and a second run
of a level buys almost nothing — the mechanical metrics retest at r = 1.00 for
that reason, not because they survived a hard test.
The ratings do not. Difficulty retests at r = 0.55 here, against the 0.93 this
project had on record from a smaller sample, and every pair that disagreed had
played the same trajectory: same moves, same outcome, different number. The
report now prints the reliability it measured rather than quoting the older
figure, and says which one it replaces.
Averaged by band over ~20 runs, though, the ratings finally track the curve —
2.00 / 2.05 / 2.40 with stuck at 1.21 / 1.15 / 1.74, where round 1 was a flat
line. So the rule is not "ratings are useless", it is "ratings are a band
statistic, never a per-level rank", and that is what the rules section says.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 1 played all fifty levels and came back with the mechanics clean and one
complaint repeated across the reports: 短暂以为转错了方向 — a turn that changed
nothing. That one is mechanical rather than perceptual, so it can be gated.
The first attempt gated `doubleDead`, and it was wrong. That counts dead states,
which scales with the state space: three-turn levels have ~187 reachable stuck
states and five-turn ones ~739, so a raw count made deep levels look eight times
worse when the rate is 2% against 5%. It would have erased every level past
three turns — the opposite of the brief, where turning is the signature and is
not to be rationed. The gate is `maxDeadTurns`, the longest run of turns before
something opens, at 1: one beat of suspense, then a release. Eleven of the fifty
shipped at 2.
Selection could not express the curve either. Spreading picks evenly over the
ranked pool returns whatever the pool is made of, and the pool is 88%
three-turn, so no weighting moved the distribution off {3: 46}. A quota per
depth states the curve outright, deepest first so the scarce end picks before a
family cap is spent on levels the shallow end has hundreds of.
Same rule bot, both sets: blind rotations 0.24 → 0.12, worst blind streak 4 → 1,
decisions unchanged at 10.3. The same amount to do, half the wasted motion.
Rotation now runs 1.8 / 3.0 / 4.1 across the bands against 1.8 / 3.0 / 3.5.
Verdicts are keyed by the board rather than by its id, because a reselect
re-issues ids from the ordinal and `c50-01-v05` is not the same board before and
after — the review sheet was one reselect away from printing one level's verdict
under another's thumbnail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
refit-weights regresses the weights select-50 scores with against what agents
actually experienced, so a round of play is a flag on the next selection rather
than an edit. Ridge, because seven collinear features on fifty levels otherwise
hand back large coefficients that cancel.
Most of the work here is in refusing to produce a number. The response averages
the difficulty rating with two continuous measures of the same thing, and a
component with no spread is dropped by name instead of being silently averaged
in as zeros — which is what the first version did, reporting R² = 0.75 for a
fit that never saw a rating. When the rating is flat the script now says so and
says what the remaining fit is really measuring, which is board size.
legibility() asks what it costs to know what you may tap: how far the eye must
follow the cheapest free piece, and how many pieces are held by somebody their
line never touches. It was built to explain why ratings came back flat. It does
not — it correlates -0.02 with them — so it stays a measurement, printed on the
review sheet beside the longest route on the board, and gates nothing.
prototype-d5 takes --seedFrom so a wide sweep shards across cores.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The chain locks all four of its pieces at the opening by construction, and the
pieces grown around it are mostly locked too, so a level built from it hands a
player two rim pieces and a third of a second of animation before any thinking
starts. Measured across the 235-level pool: median two tappable pieces at the
opening, and the best departure available plays for 0.57s.
Growth now reserves a couple of slots for showcase pieces — tappable at the
opening, with a route long enough to read as a journey and short enough to stay
under the animation ceiling. Median opening choices go 2 to 3 and the best
opening show 0.72s to 1.12s, without shortcutting the lock: each candidate is
still put through the solver.
Rotation quota is gone from the board. Turning is free and it is what the game
is about, so depth is grown rather than capped; what is gated is whether each
turn pays out, whether pieces swing between hopeless and trivial, and whether
there is a show up front.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The chain built its own worst moment: hero and tail deadlock at orientations 0
and +2 and open only at +1, with +3/+4/+5 left entirely to chance. That is why
some levels released at exactly one of six orientations, and why every model
kept writing "I don't know which way to turn" — at +2 one direction was right
and the other five were dead ends.
createDeepLock now traces candidates at all six orientations and requires the
tail to also release at +3, certified with the real swept tracer. A player
stuck at +2 turns either way and something moves. Measured by the rule bot
over the regenerated set: blind rotations, wrong-way turns and extra rotations
all go to zero, from 1.17 / 0.44 / 1.22 per level. Endgame orientation counts
shift from {1:4, 2:10, 3:4} to {2:3, 3:13, 4:1}.
It costs yield: switchback r3 and crossing r3 now produce nothing, and the set
comes out at 17 levels rather than 18. The test witnesses moved with it — a
witness is whichever seed still satisfies the construction, so they are
re-derived from the sweep rather than pinned.
docs/level-board.html is the task board for getting to 50 levels. Task state is
in bench/board.json; every number on the page — candidate count, rotation
quota, blind turns, monkey actions-per-piece, the agent batches — is measured
from the level set and the run archive at build time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
design-review.html is the gameplay and UI read on the shipped campaign, with
boards rendered from the repo's own geometry rather than redrawn.
bench-reliability.html asks which of the bench's numbers are worth acting on,
and answers it by running the same 18 levels twice under identical conditions.
The mechanical rotation counters reproduce (r = 0.82-0.92) and three different
models agree on them level for level, so they measure the level rather than
the player — a 12-second rule-bot sweep gets the same answer as hours of
model time. Difficulty and stuck ratings also hold up (r = 0.93 / 0.85), but
the ratings for rotation, clarity and fun do not (0.32 / 0.19 / 0.56): asking
a model whether rotation matters is noise, while counting blind rotations is
not.
Every figure is computed from bench/runs by build-bench-report.mjs; nothing in
the page is hand-written, so it can be regenerated as runs accumulate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
