docs: design review and bench reliability analysis

design-review.html is the gameplay and UI read on the shipped campaign, with
boards rendered from the repo's own geometry rather than redrawn.

bench-reliability.html asks which of the bench's numbers are worth acting on,
and answers it by running the same 18 levels twice under identical conditions.
The mechanical rotation counters reproduce (r = 0.82-0.92) and three different
models agree on them level for level, so they measure the level rather than
the player — a 12-second rule-bot sweep gets the same answer as hours of
model time. Difficulty and stuck ratings also hold up (r = 0.93 / 0.85), but
the ratings for rotation, clarity and fun do not (0.32 / 0.19 / 0.56): asking
a model whether rotation matters is noise, while counting blind rotations is
not.

Every figure is computed from bench/runs by build-bench-report.mjs; nothing in
the page is hand-written, so it can be regenerated as runs accumulate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
cabinw
2026-09-14 14:00:33 +08:00
co-authored by Claude Opus 5
parent 643cba0ed9
commit 752b437078
4 changed files with 2056 additions and 0 deletions
+414
View File
@@ -0,0 +1,414 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>关卡实验台:信度检验与 1000 关流程</title>
<style>
:root{
--ground:#EEF1F4; --surface:#FFFFFF; --inset:#E3E8ED; --panel:#F7F9FA;
--ink:#141E28; --ink2:#48596A; --ink3:#6B7A88;
--rule:#D3DAE1; --rule2:#BFC9D2;
--rail:#435F78; --rail-soft:#8FA6BA;
--signal:#B23A26; --signal-soft:#F2DDD8;
--warn:#8A5C0F; --warn-soft:#F4E9D4;
--good:#236E64; --good-soft:#D9EDEA;
--shadow:0 1px 2px rgba(20,30,40,.06),0 6px 18px rgba(20,30,40,.05);
color-scheme:light;
}
@media (prefers-color-scheme:dark){:root:not([data-theme=light]){
--ground:#0F161C; --surface:#141D25; --inset:#0A1015; --panel:#141D25;
--ink:#E4EAEF; --ink2:#AEBDC9; --ink3:#8494A2;
--rule:#28343F; --rule2:#35434F;
--rail:#8CB0CC; --rail-soft:#4E6577;
--signal:#E4836C; --signal-soft:#3A211B;
--warn:#E0AE5E; --warn-soft:#33280F;
--good:#4FC7B7; --good-soft:#11302C;
--shadow:0 1px 2px rgba(0,0,0,.3),0 6px 18px rgba(0,0,0,.28);
color-scheme:dark;
}}
*{box-sizing:border-box}
body{margin:0;background:var(--ground);color:var(--ink);
font-family:"PingFang SC","Hiragino Sans GB","Microsoft YaHei",ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;
line-height:1.72;-webkit-font-smoothing:antialiased}
.wrap{max-width:1000px;margin:0 auto;padding:0 24px 88px}
code,.mono{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-variant-numeric:tabular-nums}
header.head{padding:60px 0 30px;border-bottom:1px solid var(--rule);margin-bottom:40px}
.eyebrow{font-size:11.5px;letter-spacing:.18em;text-transform:uppercase;color:var(--ink3);margin-bottom:14px}
h1{font-size:clamp(28px,4.4vw,42px);line-height:1.18;margin:0 0 16px;letter-spacing:-.02em;text-wrap:balance}
.lede{font-size:17px;color:var(--ink2);max-width:66ch;margin:0}
.meta{margin-top:20px;font-size:12.5px;color:var(--ink3)}
h2{font-size:23px;margin:56px 0 8px;letter-spacing:-.01em;text-wrap:balance}
h2 .num{color:var(--rail-soft);font-family:ui-monospace,monospace;font-size:15px;margin-right:10px}
h3{font-size:16px;margin:32px 0 10px;color:var(--ink)}
p{margin:0 0 14px;max-width:72ch}
.sub{color:var(--ink2);font-size:14.5px}
.card{background:var(--surface);border:1px solid var(--rule);border-radius:12px;padding:20px 22px;box-shadow:var(--shadow);margin:20px 0}
.callout{border-left:3px solid var(--rail);background:var(--panel);border-radius:0 10px 10px 0;padding:16px 20px;margin:20px 0}
.callout.bad{border-left-color:var(--signal);background:var(--signal-soft)}
.callout.good{border-left-color:var(--good);background:var(--good-soft)}
.callout p:last-child{margin-bottom:0}
.callout b{font-weight:650}
table{width:100%;border-collapse:collapse;font-size:13.5px;margin:16px 0}
th,td{text-align:left;padding:9px 10px;border-bottom:1px solid var(--rule)}
th{font-size:11.5px;letter-spacing:.06em;color:var(--ink3);text-transform:uppercase;font-weight:600;white-space:nowrap}
td.n,th.n{text-align:right;font-family:ui-monospace,monospace;font-variant-numeric:tabular-nums;white-space:nowrap}
tbody tr:hover{background:var(--panel)}
.scroll{overflow-x:auto;-webkit-overflow-scrolling:touch}
.tag{display:inline-block;font-size:11px;padding:2px 8px;border-radius:20px;font-weight:600;white-space:nowrap}
.tag.good{background:var(--good-soft);color:var(--good)}
.tag.signal{background:var(--signal-soft);color:var(--signal)}
.tag.warn{background:var(--warn-soft);color:var(--warn)}
.grid{display:grid;gap:16px;grid-template-columns:repeat(auto-fit,minmax(210px,1fr));margin:20px 0}
.tile{background:var(--surface);border:1px solid var(--rule);border-radius:12px;padding:16px 18px}
.tile .k{font-size:11.5px;letter-spacing:.06em;color:var(--ink3);text-transform:uppercase}
.tile .v{font-family:ui-monospace,monospace;font-size:27px;font-weight:600;line-height:1.25;margin-top:4px;font-variant-numeric:tabular-nums}
.tile .n{font-size:12.5px;color:var(--ink2);margin-top:2px}
.scatters{display:grid;gap:14px;grid-template-columns:repeat(auto-fit,minmax(190px,1fr));margin:18px 0}
.sc{background:var(--surface);border:1px solid var(--rule);border-radius:12px;padding:12px 12px 6px;text-align:center}
.sc h4{margin:0 0 2px;font-size:13.5px;font-weight:600}
.sc .r{font-family:ui-monospace,monospace;font-size:12.5px;color:var(--ink2)}
.sc svg{width:100%;height:auto;display:block;margin-top:4px}
.ax{stroke:var(--rule2);stroke-width:1}
.diag{stroke:var(--rail-soft);stroke-width:1;stroke-dasharray:3 3}
.axl{font-size:9px;fill:var(--ink3);font-family:ui-monospace,monospace}
.dot circle{fill:var(--rail);fill-opacity:.72}
.dot.solid circle{fill:var(--good);fill-opacity:.8}
.dot.noise circle{fill:var(--signal);fill-opacity:.7}
.dot.flat circle{fill:var(--warn);fill-opacity:.7}
.bars{margin:14px 0}
.bar-row{display:grid;grid-template-columns:140px 1fr 48px;align-items:center;gap:12px;padding:5px 0}
.bar-k{font-size:13px}
.bar-k small{color:var(--ink3);margin-left:6px;font-size:11px}
.bar-track{background:var(--inset);border-radius:4px;height:9px;overflow:hidden}
.bar-track i{display:block;height:100%;background:var(--signal);border-radius:4px}
.bar-track i.zero{background:var(--good)}
.bar-v{font-family:ui-monospace,monospace;font-size:13px;text-align:right}
.note{border-left:2px solid var(--rule2);padding:2px 0 2px 14px;margin:10px 0;font-size:13.5px;color:var(--ink2)}
.note b{color:var(--ink);font-weight:600}
.steps{counter-reset:s;list-style:none;padding:0;margin:22px 0}
.steps>li{counter-increment:s;position:relative;padding:0 0 22px 46px;border-left:1px solid var(--rule);margin-left:14px}
.steps>li:last-child{border-left-color:transparent;padding-bottom:0}
.steps>li::before{content:"T" counter(s);position:absolute;left:-15px;top:0;width:30px;height:30px;border-radius:50%;
background:var(--rail);color:var(--surface);display:grid;place-items:center;font-family:ui-monospace,monospace;font-size:11px;font-weight:700}
.steps h3{margin:2px 0 6px}
footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:var(--ink3);font-size:12.5px}
@media(max-width:640px){.bar-row{grid-template-columns:100px 1fr 42px}.wrap{padding:0 16px 60px}}
</style>
</head>
<body>
<div class="wrap">
<header class="head">
<div class="eyebrow">HEX TRAILS · 关卡实验台</div>
<h1>Agent 评级里,哪些数字真能拿来做关卡</h1>
<p class="lede">同一批关卡、同一个模型、同一套盘面,跑两遍。能重现的指标才配进流程——这份报告把每个指标逐个检验了一遍,结论和直觉相反。</p>
<p class="meta mono">生成于 2026-09-14 13:55 GMT+8 · 数据来自 bench/runs 的 139 局 · 关卡集 levels/prototypes/d5.json(18 关)</p>
</header>
<h2><span class="num">01</span>参与检验的批次</h2>
<p class="sub">每一批用 <code>agent</code> 字段标识,不靠文件名猜来源。「旧盘面」指擦碰只报第一个阻挡者的版本,「新盘面」指修复后全部列出的版本。</p>
<div class="scroll"><table>
<thead><tr><th>批次</th><th class="n">关数</th><th class="n">通关</th><th class="n">空点</th><th class="n">擦碰空点</th><th class="n">盲转</th><th class="n">转错</th><th class="n">多转</th><th class="n">单局均</th><th class="n">每决策</th></tr></thead>
<tbody>
<tr><td>grok-4.6 · 旧盘面</td><td class="n">18</td><td class="n">18/18</td><td class="n">4</td><td class="n">4</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">83 s</td><td class="n">6.8 s</td></tr>
<tr><td>grok-4.6 · 新盘面 · 第 1 次</td><td class="n">18</td><td class="n">18/18</td><td class="n">2</td><td class="n">1</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">55 s</td><td class="n">3.9 s</td></tr>
<tr><td>grok-4.6 · 新盘面 · 第 2 次</td><td class="n">18</td><td class="n">18/18</td><td class="n">0</td><td class="n">0</td><td class="n">25</td><td class="n">10</td><td class="n">28</td><td class="n">49 s</td><td class="n">3.9 s</td></tr>
<tr><td>codex-spark · 旧盘面</td><td class="n">6</td><td class="n">6/6</td><td class="n">0</td><td class="n">0</td><td class="n">2</td><td class="n">2</td><td class="n">2</td><td class="n">15 s</td><td class="n">1.4 s</td></tr>
<tr><td>kimi-k3 · 新盘面</td><td class="n">18</td><td class="n">18/18</td><td class="n">0</td><td class="n">0</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">48 s</td><td class="n">5.2 s</td></tr>
</tbody></table></div>
<p class="sub">所有批次通关率都是 100%。<b>这本身就是一条结论:通关率在这批内容上毫无区分度,任何流程都不该建立在通过/失败上。</b></p>
<h2><span class="num">02</span>信度检验:跑两遍还一样吗</h2>
<p>同模型(grok-4.6 xhigh)、同 harness(pi)、同盘面(新)、同 18 关,只是又跑了一遍。两次之间唯一的变量就是运行本身,所以下面的相关系数<b>就是这个指标的复测信度</b>。</p>
<p class="sub">判定规则:<b>关间标准差 &lt; 0.35</b> 说明这个指标在不同关卡上几乎一样,再稳也没用(无法排序);<b>复测 r &lt; 0.5</b> 说明换一次运行结果就变,不能作为依据。两条都过才算可用。</p>
<div class="scroll"><table>
<thead><tr><th>指标</th><th class="n">关间标准差</th><th class="n">复测相关 r</th><th class="n">逐关完全一致</th><th>判定</th></tr></thead>
<tbody>
<tr><td>多转<br><small class="sub">清盘旋转数 − 求解器最少旋转数</small></td><td class="n">1.58</td><td class="n">0.87</td><td class="n">17/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>盲转<br><small class="sub">转了一次,什么出口都没打开</small></td><td class="n">1.57</td><td class="n">0.92</td><td class="n">17/18</td><td><span class="tag good">可用 · 高</span></td></tr>
<tr><td>转错方向<br><small class="sub">另一边本来能打开</small></td><td class="n">0.50</td><td class="n">0.82</td><td class="n">17/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>步数</td><td class="n">2.24</td><td class="n">0.76</td><td class="n">16/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>决策数<br><small class="sub">看了几次盘面</small></td><td class="n">1.89</td><td class="n">0.77</td><td class="n">10/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>空点<br><small class="sub">点了没动</small></td><td class="n">0.46</td><td class="n">—</td><td class="n">17/18</td><td><span class="tag signal">噪声</span></td></tr>
<tr><td>动作÷棋子</td><td class="n">0.15</td><td class="n">0.69</td><td class="n">16/18</td><td><span class="tag warn">无区分度</span></td></tr>
<tr><td>评分:无解感</td><td class="n">0.87</td><td class="n">0.85</td><td class="n">14/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>评分:难度</td><td class="n">0.59</td><td class="n">0.93</td><td class="n">17/18</td><td><span class="tag good">可用 · 高</span></td></tr>
<tr><td>评分:旋转意义</td><td class="n">0.37</td><td class="n">0.32</td><td class="n">15/18</td><td><span class="tag signal">噪声</span></td></tr>
<tr><td>评分:清晰度</td><td class="n">0.47</td><td class="n">0.19</td><td class="n">12/18</td><td><span class="tag signal">噪声</span></td></tr>
<tr><td>评分:好玩</td><td class="n">0.42</td><td class="n">0.56</td><td class="n">15/18</td><td><span class="tag good">可用</span></td></tr>
</tbody></table></div>
<p class="sub">「关间标准差」衡量这个指标能不能把不同关卡区分开——标准差接近 0 的指标就算完全可复现也没用,因为每关都一样。<b>两个条件要同时满足</b>:既能区分关卡,又能跑两遍不变。</p>
<div class="scatters">
<div class="sc"><h4>多转</h4><div class="r">r = 0.87</div><svg viewBox="0 0 168 168" role="img" aria-label="多转 两次运行散点图,相关系数 0.87">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="39.2" cy="128.8" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="39.2" cy="128.8" r="3.6"/><circle cx="39.2" cy="128.8" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="52.4" cy="115.6" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="39.2" cy="128.8" r="3.6"/><circle cx="78.8" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="78.8" cy="89.2" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="78.8" cy="89.2" r="3.6"/><circle cx="78.8" cy="89.2" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>盲转</h4><div class="r">r = 0.92</div><svg viewBox="0 0 168 168" role="img" aria-label="盲转 两次运行散点图,相关系数 0.92">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot solid"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="92.0" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>转错方向</h4><div class="r">r = 0.82</div><svg viewBox="0 0 168 168" role="img" aria-label="转错方向 两次运行散点图,相关系数 0.82">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>步数</h4><div class="r">r = 0.76</div><svg viewBox="0 0 168 168" role="img" aria-label="步数 两次运行散点图,相关系数 0.76">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="67.7" cy="114.2" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="53.8" cy="114.2" r="3.6"/><circle cx="32.9" cy="135.1" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="60.7" cy="107.3" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="53.8" cy="114.2" r="3.6"/><circle cx="74.6" cy="10.0" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="53.8" cy="114.2" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="74.6" cy="93.4" r="3.6"/><circle cx="74.6" cy="93.4" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>决策数</h4><div class="r">r = 0.77</div><svg viewBox="0 0 168 168" role="img" aria-label="决策数 两次运行散点图,相关系数 0.77">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="35.4" cy="142.0" r="3.6"/><circle cx="63.7" cy="113.7" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="35.4" cy="132.6" r="3.6"/><circle cx="35.4" cy="132.6" r="3.6"/><circle cx="26.0" cy="123.1" r="3.6"/><circle cx="26.0" cy="132.6" r="3.6"/><circle cx="26.0" cy="132.6" r="3.6"/><circle cx="54.3" cy="113.7" r="3.6"/><circle cx="26.0" cy="132.6" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="35.4" cy="123.1" r="3.6"/><circle cx="73.1" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="63.7" cy="104.3" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="63.7" cy="104.3" r="3.6"/><circle cx="73.1" cy="94.9" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>动作÷棋子</h4><div class="r">r = 0.69</div><svg viewBox="0 0 168 168" role="img" aria-label="动作÷棋子 两次运行散点图,相关系数 0.69">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot flat"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="50.8" cy="133.8" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="45.2" cy="122.8" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="59.0" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="78.2" cy="89.8" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="59.0" cy="109.0" r="3.6"/><circle cx="59.0" cy="109.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>评分:无解感</h4><div class="r">r = 0.85</div><svg viewBox="0 0 168 168" role="img" aria-label="评分:无解感 两次运行散点图,相关系数 0.85">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="158.0" cy="54.0" r="3.6"/><circle cx="26.0" cy="98.0" r="3.6"/><circle cx="114.0" cy="54.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="114.0" cy="54.0" r="3.6"/><circle cx="114.0" cy="54.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>评分:难度</h4><div class="r">r = 0.93</div><svg viewBox="0 0 168 168" role="img" aria-label="评分:难度 两次运行散点图,相关系数 0.93">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot solid"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="158.0" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
</div>
<p class="sub">虚线是「两次完全相同」。点贴着虚线 = 可复现;点散开 = 这个数字换一次运行就变。</p>
<h3>对照:换了盘面格式的那一对</h3>
<p class="sub">上一节控制了所有变量。作为对照,这里是<b>旧盘面 vs 新盘面</b>的同 18 关配对——它多了一个变量(盘面变了),所以相关系数是信度的<b>下限</b>而非真值。两张表一起看,能分清「指标本身不稳」和「盘面变化导致的差异」。</p>
<div class="scroll"><table>
<thead><tr><th>指标</th><th class="n">控制变量后 r</th><th class="n">换盘面后 r</th><th>差异说明</th></tr></thead>
<tbody>
<tr><td>多转</td><td class="n">0.87</td><td class="n">1.00</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>盲转</td><td class="n">0.92</td><td class="n">1.00</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>转错方向</td><td class="n">0.82</td><td class="n">1.00</td><td class="sub">控制变量后反而更低——值得留意</td></tr>
<tr><td>步数</td><td class="n">0.76</td><td class="n">0.95</td><td class="sub">控制变量后反而更低——值得留意</td></tr>
<tr><td>决策数</td><td class="n">0.77</td><td class="n">0.87</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>空点</td><td class="n">—</td><td class="n">-0.10</td><td class="sub">—</td></tr>
<tr><td>动作÷棋子</td><td class="n">0.69</td><td class="n">0.92</td><td class="sub">控制变量后反而更低——值得留意</td></tr>
<tr><td>评分:无解感</td><td class="n">0.85</td><td class="n">0.80</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>评分:难度</td><td class="n">0.93</td><td class="n">0.63</td><td class="sub">控制变量后明显更高——之前被盘面变化压低了</td></tr>
<tr><td>评分:旋转意义</td><td class="n">0.32</td><td class="n">0.06</td><td class="sub">控制变量后明显更高——之前被盘面变化压低了</td></tr>
<tr><td>评分:清晰度</td><td class="n">0.19</td><td class="n">0.09</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>评分:好玩</td><td class="n">0.56</td><td class="n">0.21</td><td class="sub">控制变量后明显更高——之前被盘面变化压低了</td></tr>
</tbody></table></div>
<h2><span class="num">03</span>跨模型交叉验证:这些数字属于关卡,不属于玩家</h2>
<p>下表是同一套盘面下所有批次的 <code>多转 / 盲转 / 转错方向</code>。不同厂商的模型、不同 harness、各自独立跑——如果这些数字真的由关卡决定,它们就该一样。</p>
<div class="grid">
<div class="tile"><div class="k">逐关完全一致</div><div class="v">17 / 18</div><div class="n">三项数字一个不差</div></div>
<div class="tile"><div class="k">参与批次</div><div class="v">3</div><div class="n">grok新① · grok新② · kimi</div></div>
</div>
<div class="scroll"><table>
<thead><tr><th>关</th><th>家族</th><th class="n">grok新①</th><th class="n">grok新②</th><th class="n">kimi</th><th>一致</th></tr></thead>
<tbody>
<tr><td>L1</td><td>switchback</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L2</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L3</td><td>weave</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L4</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L5</td><td>weave</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L6</td><td>weave</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L7</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L8</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L9</td><td>weave</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L10</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L11</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L12</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L13</td><td>orbits</td><td class="n">4/4/1</td><td class="n">10/8/3</td><td class="n">4/4/1</td><td><span class="tag signal">✘</span></td></tr>
<tr><td>L14</td><td>crossing</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L15</td><td>orbits</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L16</td><td>orbits</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L17</td><td>spiral</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L18</td><td>spiral</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
</tbody></table></div>
<div class="callout good">
<p><b>这是全份报告里最硬的一条结论。</b>17/18 关上,独立运行的不同模型给出了完全相同的旋转三元组。这意味着<b>规则机器人 12 秒扫出来的这三个数,等价于大模型跑几小时的结果</b>——这个维度完全不需要 agent。</p>
</div>
<h3>不一致的关:不是难,是不稳定</h3>
<p class="sub">下面这些关在不同运行之间给出了不同的数字。排除模型差异后,剩下的解释是<b>同一关的体验取决于起手方向</b>——两个玩家玩同一关,一个觉得还行,一个觉得这关有毛病。这比"稳定地难"更值得处理。</p>
<div class="note"><b>L13 orbits</b>:4/4/1 / 10/8/3 / 4/4/1</div>
<h2><span class="num">04</span>家族结构:机械指标指向同一个地方</h2>
<p>把「多转」按路线家族平均。这是关卡结构问题,不是玩家问题——两批 grok、以及 spark,在这一点上给出一致的结果。</p>
<div class="card"><h3 style="margin-top:0">grok-4.6 · 旧盘面 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">4.0</b></div>
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:66.7%" class=""></i></span>
<b class="bar-v">2.7</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="card"><h3 style="margin-top:0">grok-4.6 · 新盘面 · 第 1 次 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">4.0</b></div>
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:66.7%" class=""></i></span>
<b class="bar-v">2.7</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="card"><h3 style="margin-top:0">grok-4.6 · 新盘面 · 第 2 次 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">4.7</b></div>
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:85.7%" class=""></i></span>
<b class="bar-v">4.0</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:16.1%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:16.1%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="card"><h3 style="margin-top:0">codex-spark · 旧盘面 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">switchback<small>3 关</small></span>
<span class="bar-track"><i style="width:33.3%" class=""></i></span>
<b class="bar-v">0.3</b></div>
<div class="bar-row"><span class="bar-k">weave<small>3 关</small></span>
<span class="bar-track"><i style="width:33.3%" class=""></i></span>
<b class="bar-v">0.3</b></div></div></div>
<div class="card"><h3 style="margin-top:0">kimi-k3 · 新盘面 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">4.0</b></div>
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:66.7%" class=""></i></span>
<b class="bar-v">2.7</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="callout bad">
<p><b>orbits 和 spiral 是问题家族。</b>它们稳定地需要 7 次旋转(最少 3 次)外加 4 次盲转——等于把 6 个朝向挨个试一遍。而 petals 和 crossing 是干净的 0。锁住率解释不了这个差异(L16 也是 orbits、锁 78%,却是 3/3)。</p>
</div>
<h2><span class="num">05</span>Agent 说了什么</h2>
<p class="sub">评分是 1–5 的数字,<code>worst</code> 是它自己写的一句话。下面按反复出现的主题归类——<b>价值在重复模式,不在单条内容</b>。</p>
<h3>擦碰看不出来 · 22 次提及</h3>
<div class="note"><b>L1 switchback</b>(grok旧):开局 t2/t11 路上看着没人其实会擦到,差点空点;只剩 t0 和 t3 对堵时短暂以为卡住了</div>
<div class="note"><b>L3 weave</b>(grok旧):t5走后(第9步)t2和t3都像能出界,其实互相擦碰,只能再转</div>
<div class="note"><b>L3 weave</b>(grok新①):朝向1时 t2 和 t3 互相擦碰、t0 被 t2 压着,短暂不确定是再转还是还有能点的</div>
<div class="note"><b>L3 weave</b>(grok新②):第9步清完后t2和t3互相擦挡,谁先走都不行,只能转</div>
<h3>不知道该转哪边 · 16 次提及</h3>
<div class="note"><b>L2 switchback</b>(grok旧):第 13 步右转后朝向 3 没新出口,t0 和 t3 仍互相挡,不确定该不该继续同向转</div>
<div class="note"><b>L2 switchback</b>(grok新①):第15步右转后朝向3没有新出口,最后两枚还互压,一度担心转错了方向</div>
<div class="note"><b>L2 switchback</b>(spark旧):第 8-9 步清掉 t5,t4,t2 后只剩 t0/t3 互压,朝向2不再有出口,开始不确定该是继续走还是转哪边</div>
<div class="note"><b>L3 weave</b>(spark旧):第 5 步:第一轮转盘前所有可见线路都被压住/擦边,必须停下来判断该往哪个方向转</div>
<h3>长路线读不动 · 20 次提及</h3>
<div class="note"><b>L1 switchback</b>(kimi):第 10 步后 t0/t2/t3/t10 四枚互相压成环,盘面上看不出解法,只能靠旋转试出新朝向</div>
<div class="note"><b>L6 weave</b>(grok新①):开局 t0/t1 路线三十多格绕完全盘,只能靠路上有谁的摘要,看不出线实际怎么走</div>
<div class="note"><b>L7 petals</b>(grok旧):开局 t0/t3 各 19 格 15 弯还叠着好几枚,文字路线太长,分不清该先解哪一层</div>
<div class="note"><b>L7 petals</b>(grok新①):第 10 步再右转只开了 t7,t0 和 t3 仍 19 格对压,不确定是不是转错方向</div>
<h2><span class="num">06</span>1000 关该怎么跑</h2>
<p>核心判断:<b>机械计数器当主力,agent 做抽检</b>。免费的规则机器人能扫全量并稳定给出盲转/多转/转错;agent 贵但能提供两样机器给不了的东西——难度的相对排序,和用自然语言点出的系统性缺陷。<b>关键是只收集经过信度检验的那几项</b>。</p>
<ol class="steps">
<li><h3>生成期硬门槛 · 零成本</h3><p class="sub">用已有求解器,在生成时就卡掉:最少旋转 ≥ 2、开局锁住率 ≤ 90%、首转两个方向都能推进、无死局状态。不合格的根本不进候选池。</p></li>
<li><h3>规则机器人全量扫 · 1000 关 12 秒</h3><p class="sub">产出盲转 / 多转 / 转错方向。按 <code>盲转 ≥ 2 或 多转 ≥ 2</code> 打标。这条规则在 D5 上正好命中 orbits+spiral 那几关,和两个大模型跑两小时得到的结论一致。</p></li>
<li><h3>Agent 只跑被标记的 + 5% 随机抽样</h3><p class="sub">不是 1000 关全跑。收集<b>三样东西</b>:<code>difficulty</code>(r=0.93,可用来给 1000 关排序)、<code>stuck</code>(r=0.85,体感无解的直接信号)、以及 <code>worst</code> 文字(重复模式能挖出系统性缺陷)。<b>丢掉 rotation / clarity / fun</b>——复测 r 只有 0.32 / 0.19 / 0.56,是模型噪声。按实测 51 s/关、4 路并行,150 关约半小时。</p></li>
<li><h3>真人标定锚点 · 5–10 关</h3><p class="sub">Agent 的难度<b>排序</b>可信(复测 r=0.93),但绝对值可能整体偏移——它觉得「刚好」的关,真人可能觉得太简单。所以真人只需要玩少数几关当<b>锚点</b>,把 agent 的相对排序映射到真实难度刻度上,不必逐关评。这比原本设想的省很多人力。</p></li>
</ol>
<div class="callout good">
<p><b>最该做的一件事:把它变成回归测试,而不是评审流程。</b>改完生成器,12 秒跑一遍规则机器人,比较盲转分布。从 22% 降到 5% 就是客观改善,升上去就是刚引入了退化。12 秒一轮可以进 CI;几小时一轮且分数不可复现的人工评审,不可能成为迭代循环。</p>
</div>
<h2><span class="num">07</span>这份结论的边界</h2>
<p class="sub">写清楚哪里不够硬,比多列几个数字重要。</p>
<ul class="sub">
<li>跨模型一致性只验证了 <b>6 关</b>(spark 与 grok 的交集),其中 1 关不一致。机械指标"与玩家无关"这个说法,样本还太小。</li>
<li>复测只做了 <b>2 次</b>。2 次能算相关系数,但对评分这种低方差的离散量,2 次的估计本身就有不小的不确定性。</li>
<li>Agent 读的是文字不是像素,<b>测不出"看不清"</b>。clarity 这一项从原理上就不该指望它。</li>
<li>所有结论都基于 <b>D5 原型这 18 关</b>。这批关卡通关率 100%、难度评分方差极小,换一批更难的内容,结论可能不同。</li>
</ul>
<footer>
<p class="mono">数据:bench/runs · 生成脚本:scripts/build-bench-report.mjs · 重新生成:<code>node scripts/build-bench-report.mjs</code></p>
<p>本页所有数字由脚本从存档直接计算,未经手工填写。相关系数为 Pearson r,标准差为总体标准差。</p>
</footer>
</div>
</body>
</html>
File diff suppressed because one or more lines are too long