Files
HexTrails/docs/bench-reliability.html
T
cabinwandClaude Opus 5 cefb7a5a56 docs: five-way cross-check, and correct a mislabelled batch
One batch of 18 runs was recorded as codex-spark at xhigh but had actually run
as gpt-6-astra at low effort: the CLI config was rewritten between the check
and the batch, and codex resolved the new model silently. The archives are
relabelled from the CLI's own startup header rather than from what the agent
was told to call itself, and the report now says to trust that header over the
self-reported name. The re-run verifies the resolved model after every level
and aborts the batch on a mismatch.

With the real codex-spark batch added, five independent runs of four models
from three vendors now cover the same 18 levels on the fixed board, and 16 of
17 comparable levels agree exactly on the rotation triple.

codex-spark gave up on L18 after cycling all six orientations, reporting no
exit anywhere — replaying the position shows orientation 1 releases t0, and
that it is the only one of the six that does. The level is solvable and the
routes are legible (it rated clarity 5), but nothing on the board says which
way to turn: a 1-in-6 guess with a dead end behind every other door. That is
the terminal-interlock complaint the other models kept raising, at its limit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 15:37:38 +08:00

457 lines
44 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>关卡实验台:信度检验与 1000 关流程</title>
<style>
:root{
--ground:#EEF1F4; --surface:#FFFFFF; --inset:#E3E8ED; --panel:#F7F9FA;
--ink:#141E28; --ink2:#48596A; --ink3:#6B7A88;
--rule:#D3DAE1; --rule2:#BFC9D2;
--rail:#435F78; --rail-soft:#8FA6BA;
--signal:#B23A26; --signal-soft:#F2DDD8;
--warn:#8A5C0F; --warn-soft:#F4E9D4;
--good:#236E64; --good-soft:#D9EDEA;
--shadow:0 1px 2px rgba(20,30,40,.06),0 6px 18px rgba(20,30,40,.05);
color-scheme:light;
}
@media (prefers-color-scheme:dark){:root:not([data-theme=light]){
--ground:#0F161C; --surface:#141D25; --inset:#0A1015; --panel:#141D25;
--ink:#E4EAEF; --ink2:#AEBDC9; --ink3:#8494A2;
--rule:#28343F; --rule2:#35434F;
--rail:#8CB0CC; --rail-soft:#4E6577;
--signal:#E4836C; --signal-soft:#3A211B;
--warn:#E0AE5E; --warn-soft:#33280F;
--good:#4FC7B7; --good-soft:#11302C;
--shadow:0 1px 2px rgba(0,0,0,.3),0 6px 18px rgba(0,0,0,.28);
color-scheme:dark;
}}
*{box-sizing:border-box}
body{margin:0;background:var(--ground);color:var(--ink);
font-family:"PingFang SC","Hiragino Sans GB","Microsoft YaHei",ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;
line-height:1.72;-webkit-font-smoothing:antialiased}
.wrap{max-width:1000px;margin:0 auto;padding:0 24px 88px}
code,.mono{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-variant-numeric:tabular-nums}
header.head{padding:60px 0 30px;border-bottom:1px solid var(--rule);margin-bottom:40px}
.eyebrow{font-size:11.5px;letter-spacing:.18em;text-transform:uppercase;color:var(--ink3);margin-bottom:14px}
h1{font-size:clamp(28px,4.4vw,42px);line-height:1.18;margin:0 0 16px;letter-spacing:-.02em;text-wrap:balance}
.lede{font-size:17px;color:var(--ink2);max-width:66ch;margin:0}
.meta{margin-top:20px;font-size:12.5px;color:var(--ink3)}
h2{font-size:23px;margin:56px 0 8px;letter-spacing:-.01em;text-wrap:balance}
h2 .num{color:var(--rail-soft);font-family:ui-monospace,monospace;font-size:15px;margin-right:10px}
h3{font-size:16px;margin:32px 0 10px;color:var(--ink)}
p{margin:0 0 14px;max-width:72ch}
.sub{color:var(--ink2);font-size:14.5px}
.card{background:var(--surface);border:1px solid var(--rule);border-radius:12px;padding:20px 22px;box-shadow:var(--shadow);margin:20px 0}
.callout{border-left:3px solid var(--rail);background:var(--panel);border-radius:0 10px 10px 0;padding:16px 20px;margin:20px 0}
.callout.bad{border-left-color:var(--signal);background:var(--signal-soft)}
.callout.good{border-left-color:var(--good);background:var(--good-soft)}
.callout p:last-child{margin-bottom:0}
.callout b{font-weight:650}
table{width:100%;border-collapse:collapse;font-size:13.5px;margin:16px 0}
th,td{text-align:left;padding:9px 10px;border-bottom:1px solid var(--rule)}
th{font-size:11.5px;letter-spacing:.06em;color:var(--ink3);text-transform:uppercase;font-weight:600;white-space:nowrap}
td.n,th.n{text-align:right;font-family:ui-monospace,monospace;font-variant-numeric:tabular-nums;white-space:nowrap}
tbody tr:hover{background:var(--panel)}
.scroll{overflow-x:auto;-webkit-overflow-scrolling:touch}
.tag{display:inline-block;font-size:11px;padding:2px 8px;border-radius:20px;font-weight:600;white-space:nowrap}
.tag.good{background:var(--good-soft);color:var(--good)}
.tag.signal{background:var(--signal-soft);color:var(--signal)}
.tag.warn{background:var(--warn-soft);color:var(--warn)}
.grid{display:grid;gap:16px;grid-template-columns:repeat(auto-fit,minmax(210px,1fr));margin:20px 0}
.tile{background:var(--surface);border:1px solid var(--rule);border-radius:12px;padding:16px 18px}
.tile .k{font-size:11.5px;letter-spacing:.06em;color:var(--ink3);text-transform:uppercase}
.tile .v{font-family:ui-monospace,monospace;font-size:27px;font-weight:600;line-height:1.25;margin-top:4px;font-variant-numeric:tabular-nums}
.tile .n{font-size:12.5px;color:var(--ink2);margin-top:2px}
.scatters{display:grid;gap:14px;grid-template-columns:repeat(auto-fit,minmax(190px,1fr));margin:18px 0}
.sc{background:var(--surface);border:1px solid var(--rule);border-radius:12px;padding:12px 12px 6px;text-align:center}
.sc h4{margin:0 0 2px;font-size:13.5px;font-weight:600}
.sc .r{font-family:ui-monospace,monospace;font-size:12.5px;color:var(--ink2)}
.sc svg{width:100%;height:auto;display:block;margin-top:4px}
.ax{stroke:var(--rule2);stroke-width:1}
.diag{stroke:var(--rail-soft);stroke-width:1;stroke-dasharray:3 3}
.axl{font-size:9px;fill:var(--ink3);font-family:ui-monospace,monospace}
.dot circle{fill:var(--rail);fill-opacity:.72}
.dot.solid circle{fill:var(--good);fill-opacity:.8}
.dot.noise circle{fill:var(--signal);fill-opacity:.7}
.dot.flat circle{fill:var(--warn);fill-opacity:.7}
.bars{margin:14px 0}
.bar-row{display:grid;grid-template-columns:140px 1fr 48px;align-items:center;gap:12px;padding:5px 0}
.bar-k{font-size:13px}
.bar-k small{color:var(--ink3);margin-left:6px;font-size:11px}
.bar-track{background:var(--inset);border-radius:4px;height:9px;overflow:hidden}
.bar-track i{display:block;height:100%;background:var(--signal);border-radius:4px}
.bar-track i.zero{background:var(--good)}
.bar-v{font-family:ui-monospace,monospace;font-size:13px;text-align:right}
.note{border-left:2px solid var(--rule2);padding:2px 0 2px 14px;margin:10px 0;font-size:13.5px;color:var(--ink2)}
.note b{color:var(--ink);font-weight:600}
.steps{counter-reset:s;list-style:none;padding:0;margin:22px 0}
.steps>li{counter-increment:s;position:relative;padding:0 0 22px 46px;border-left:1px solid var(--rule);margin-left:14px}
.steps>li:last-child{border-left-color:transparent;padding-bottom:0}
.steps>li::before{content:"T" counter(s);position:absolute;left:-15px;top:0;width:30px;height:30px;border-radius:50%;
background:var(--rail);color:var(--surface);display:grid;place-items:center;font-family:ui-monospace,monospace;font-size:11px;font-weight:700}
.steps h3{margin:2px 0 6px}
footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:var(--ink3);font-size:12.5px}
@media(max-width:640px){.bar-row{grid-template-columns:100px 1fr 42px}.wrap{padding:0 16px 60px}}
</style>
</head>
<body>
<div class="wrap">
<header class="head">
<div class="eyebrow">HEX TRAILS · 关卡实验台</div>
<h1>Agent 评级里,哪些数字真能拿来做关卡</h1>
<p class="lede">同一批关卡、同一个模型、同一套盘面,跑两遍。能重现的指标才配进流程——这份报告把每个指标逐个检验了一遍,结论和直觉相反。</p>
<p class="meta mono">生成于 2026-09-14 15:37 GMT+8 · 数据来自 bench/runs 的 175 局 · 关卡集 levels/prototypes/d5.json(18 关)</p>
</header>
<h2><span class="num">01</span>参与检验的批次</h2>
<p class="sub">每一批用 <code>agent</code> 字段标识,不靠文件名猜来源。「旧盘面」指擦碰只报第一个阻挡者的版本,「新盘面」指修复后全部列出的版本。</p>
<p class="sub">模型名以 <b>CLI 启动时打印的解析结果</b>为准,而不是提示词里让 agent 自报的名字——有一批 18 局曾被标成 codex-spark,实际是配置在运行前被改掉、跑成了 gpt-6-astra/low,存档已按日志头更正。</p>
<div class="scroll"><table>
<thead><tr><th>批次</th><th class="n">关数</th><th class="n">通关</th><th class="n">空点</th><th class="n">擦碰空点</th><th class="n">盲转</th><th class="n">转错</th><th class="n">多转</th><th class="n">单局均</th><th class="n">每决策</th></tr></thead>
<tbody>
<tr><td>grok-4.6 · 旧盘面</td><td class="n">18</td><td class="n">18/18</td><td class="n">4</td><td class="n">4</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">83 s</td><td class="n">6.8 s</td></tr>
<tr><td>grok-4.6 · 新盘面 · 第 1 次</td><td class="n">18</td><td class="n">18/18</td><td class="n">2</td><td class="n">1</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">55 s</td><td class="n">3.9 s</td></tr>
<tr><td>grok-4.6 · 新盘面 · 第 2 次</td><td class="n">18</td><td class="n">18/18</td><td class="n">0</td><td class="n">0</td><td class="n">25</td><td class="n">10</td><td class="n">28</td><td class="n">49 s</td><td class="n">3.9 s</td></tr>
<tr><td>codex-spark · 旧盘面</td><td class="n">6</td><td class="n">6/6</td><td class="n">0</td><td class="n">0</td><td class="n">2</td><td class="n">2</td><td class="n">2</td><td class="n">15 s</td><td class="n">1.4 s</td></tr>
<tr><td>gpt-6-astra · low · 新盘面</td><td class="n">18</td><td class="n">18/18</td><td class="n">1</td><td class="n">0</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">41 s</td><td class="n">7.0 s</td></tr>
<tr><td>codex-spark · xhigh · 新盘面</td><td class="n">18</td><td class="n">17/18</td><td class="n">0</td><td class="n">0</td><td class="n">21</td><td class="n">7</td><td class="n">21</td><td class="n">21 s</td><td class="n">1.8 s</td></tr>
<tr><td>kimi-k3 · high · 新盘面</td><td class="n">18</td><td class="n">18/18</td><td class="n">0</td><td class="n">0</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">48 s</td><td class="n">5.2 s</td></tr>
</tbody></table></div>
<p class="sub">已完成的对局通关率全部是 100%(个别批次仍有对局在进行中,未计入)。<b>通关率在这批内容上毫无区分度,任何流程都不该建立在通过/失败上。</b></p>
<h2><span class="num">02</span>信度检验:跑两遍还一样吗</h2>
<p>同模型(grok-4.6 xhigh)、同 harness(pi)、同盘面(新)、同 18 关,只是又跑了一遍。两次之间唯一的变量就是运行本身,所以下面的相关系数<b>就是这个指标的复测信度</b>。</p>
<p class="sub">判定规则:<b>关间标准差 &lt; 0.35</b> 说明这个指标在不同关卡上几乎一样,再稳也没用(无法排序);<b>复测 r &lt; 0.5</b> 说明换一次运行结果就变,不能作为依据。两条都过才算可用。</p>
<div class="scroll"><table>
<thead><tr><th>指标</th><th class="n">关间标准差</th><th class="n">复测相关 r</th><th class="n">逐关完全一致</th><th>判定</th></tr></thead>
<tbody>
<tr><td>多转<br><small class="sub">清盘旋转数 − 求解器最少旋转数</small></td><td class="n">1.58</td><td class="n">0.87</td><td class="n">17/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>盲转<br><small class="sub">转了一次,什么出口都没打开</small></td><td class="n">1.57</td><td class="n">0.92</td><td class="n">17/18</td><td><span class="tag good">可用 · 高</span></td></tr>
<tr><td>转错方向<br><small class="sub">另一边本来能打开</small></td><td class="n">0.50</td><td class="n">0.82</td><td class="n">17/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>步数</td><td class="n">2.24</td><td class="n">0.76</td><td class="n">16/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>决策数<br><small class="sub">看了几次盘面</small></td><td class="n">1.89</td><td class="n">0.77</td><td class="n">10/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>空点<br><small class="sub">点了没动</small></td><td class="n">0.46</td><td class="n">—</td><td class="n">17/18</td><td><span class="tag signal">噪声</span></td></tr>
<tr><td>动作÷棋子</td><td class="n">0.15</td><td class="n">0.69</td><td class="n">16/18</td><td><span class="tag warn">无区分度</span></td></tr>
<tr><td>评分:无解感</td><td class="n">0.87</td><td class="n">0.85</td><td class="n">14/18</td><td><span class="tag good">可用</span></td></tr>
<tr><td>评分:难度</td><td class="n">0.59</td><td class="n">0.93</td><td class="n">17/18</td><td><span class="tag good">可用 · 高</span></td></tr>
<tr><td>评分:旋转意义</td><td class="n">0.37</td><td class="n">0.32</td><td class="n">15/18</td><td><span class="tag signal">噪声</span></td></tr>
<tr><td>评分:清晰度</td><td class="n">0.47</td><td class="n">0.19</td><td class="n">12/18</td><td><span class="tag signal">噪声</span></td></tr>
<tr><td>评分:好玩</td><td class="n">0.42</td><td class="n">0.56</td><td class="n">15/18</td><td><span class="tag good">可用</span></td></tr>
</tbody></table></div>
<p class="sub">「关间标准差」衡量这个指标能不能把不同关卡区分开——标准差接近 0 的指标就算完全可复现也没用,因为每关都一样。<b>两个条件要同时满足</b>:既能区分关卡,又能跑两遍不变。</p>
<div class="scatters">
<div class="sc"><h4>多转</h4><div class="r">r = 0.87</div><svg viewBox="0 0 168 168" role="img" aria-label="多转 两次运行散点图,相关系数 0.87">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="39.2" cy="128.8" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="39.2" cy="128.8" r="3.6"/><circle cx="39.2" cy="128.8" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="52.4" cy="115.6" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="39.2" cy="128.8" r="3.6"/><circle cx="78.8" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="78.8" cy="89.2" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="78.8" cy="89.2" r="3.6"/><circle cx="78.8" cy="89.2" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>盲转</h4><div class="r">r = 0.92</div><svg viewBox="0 0 168 168" role="img" aria-label="盲转 两次运行散点图,相关系数 0.92">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot solid"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="92.0" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>转错方向</h4><div class="r">r = 0.82</div><svg viewBox="0 0 168 168" role="img" aria-label="转错方向 两次运行散点图,相关系数 0.82">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>步数</h4><div class="r">r = 0.76</div><svg viewBox="0 0 168 168" role="img" aria-label="步数 两次运行散点图,相关系数 0.76">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="67.7" cy="114.2" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="53.8" cy="114.2" r="3.6"/><circle cx="32.9" cy="135.1" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="60.7" cy="107.3" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="53.8" cy="114.2" r="3.6"/><circle cx="74.6" cy="10.0" r="3.6"/><circle cx="46.8" cy="121.2" r="3.6"/><circle cx="53.8" cy="114.2" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="74.6" cy="93.4" r="3.6"/><circle cx="74.6" cy="93.4" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>决策数</h4><div class="r">r = 0.77</div><svg viewBox="0 0 168 168" role="img" aria-label="决策数 两次运行散点图,相关系数 0.77">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="35.4" cy="142.0" r="3.6"/><circle cx="63.7" cy="113.7" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="35.4" cy="132.6" r="3.6"/><circle cx="35.4" cy="132.6" r="3.6"/><circle cx="26.0" cy="123.1" r="3.6"/><circle cx="26.0" cy="132.6" r="3.6"/><circle cx="26.0" cy="132.6" r="3.6"/><circle cx="54.3" cy="113.7" r="3.6"/><circle cx="26.0" cy="132.6" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="35.4" cy="123.1" r="3.6"/><circle cx="73.1" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="63.7" cy="104.3" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="63.7" cy="104.3" r="3.6"/><circle cx="73.1" cy="94.9" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>动作÷棋子</h4><div class="r">r = 0.69</div><svg viewBox="0 0 168 168" role="img" aria-label="动作÷棋子 两次运行散点图,相关系数 0.69">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot flat"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="50.8" cy="133.8" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="45.2" cy="122.8" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="42.5" cy="125.5" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="59.0" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="78.2" cy="89.8" r="3.6"/><circle cx="34.2" cy="133.8" r="3.6"/><circle cx="59.0" cy="109.0" r="3.6"/><circle cx="59.0" cy="109.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>评分:无解感</h4><div class="r">r = 0.85</div><svg viewBox="0 0 168 168" role="img" aria-label="评分:无解感 两次运行散点图,相关系数 0.85">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot ok"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="70.0" cy="98.0" r="3.6"/><circle cx="158.0" cy="54.0" r="3.6"/><circle cx="26.0" cy="98.0" r="3.6"/><circle cx="114.0" cy="54.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="114.0" cy="54.0" r="3.6"/><circle cx="114.0" cy="54.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
<div class="sc"><h4>评分:难度</h4><div class="r">r = 0.93</div><svg viewBox="0 0 168 168" role="img" aria-label="评分:难度 两次运行散点图,相关系数 0.93">
<line class="ax" x1="26" y1="142" x2="160" y2="142"/>
<line class="ax" x1="26" y1="8" x2="26" y2="142"/>
<line class="diag" x1="26" y1="142" x2="158" y2="10"/>
<g class="dot solid"><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="158.0" cy="10.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="26.0" cy="142.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/><circle cx="92.0" cy="76.0" r="3.6"/></g>
<text class="axl" x="84" y="162" text-anchor="middle">第 1 次</text>
<text class="axl" x="10" y="84" text-anchor="middle" transform="rotate(-90 10 84)">第 2 次</text>
</svg></div>
</div>
<p class="sub">虚线是「两次完全相同」。点贴着虚线 = 可复现;点散开 = 这个数字换一次运行就变。</p>
<h3>对照:换了盘面格式的那一对</h3>
<p class="sub">上一节控制了所有变量。作为对照,这里是<b>旧盘面 vs 新盘面</b>的同 18 关配对——它多了一个变量(盘面变了),所以相关系数是信度的<b>下限</b>而非真值。两张表一起看,能分清「指标本身不稳」和「盘面变化导致的差异」。</p>
<div class="scroll"><table>
<thead><tr><th>指标</th><th class="n">控制变量后 r</th><th class="n">换盘面后 r</th><th>差异说明</th></tr></thead>
<tbody>
<tr><td>多转</td><td class="n">0.87</td><td class="n">1.00</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>盲转</td><td class="n">0.92</td><td class="n">1.00</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>转错方向</td><td class="n">0.82</td><td class="n">1.00</td><td class="sub">控制变量后反而更低——值得留意</td></tr>
<tr><td>步数</td><td class="n">0.76</td><td class="n">0.95</td><td class="sub">控制变量后反而更低——值得留意</td></tr>
<tr><td>决策数</td><td class="n">0.77</td><td class="n">0.87</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>空点</td><td class="n">—</td><td class="n">-0.10</td><td class="sub">—</td></tr>
<tr><td>动作÷棋子</td><td class="n">0.69</td><td class="n">0.92</td><td class="sub">控制变量后反而更低——值得留意</td></tr>
<tr><td>评分:无解感</td><td class="n">0.85</td><td class="n">0.80</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>评分:难度</td><td class="n">0.93</td><td class="n">0.63</td><td class="sub">控制变量后明显更高——之前被盘面变化压低了</td></tr>
<tr><td>评分:旋转意义</td><td class="n">0.32</td><td class="n">0.06</td><td class="sub">控制变量后明显更高——之前被盘面变化压低了</td></tr>
<tr><td>评分:清晰度</td><td class="n">0.19</td><td class="n">0.09</td><td class="sub">两者接近,盘面变化影响很小</td></tr>
<tr><td>评分:好玩</td><td class="n">0.56</td><td class="n">0.21</td><td class="sub">控制变量后明显更高——之前被盘面变化压低了</td></tr>
</tbody></table></div>
<h2><span class="num">03</span>跨模型交叉验证:这些数字属于关卡,不属于玩家</h2>
<p>下表是同一套盘面下所有批次的 <code>多转 / 盲转 / 转错方向</code>。不同厂商的模型、不同 harness、各自独立跑——如果这些数字真的由关卡决定,它们就该一样。</p>
<div class="grid">
<div class="tile"><div class="k">逐关完全一致</div><div class="v">16 / 18</div><div class="n">三项数字一个不差</div></div>
<div class="tile"><div class="k">参与批次</div><div class="v">5</div><div class="n">grok新① · grok新② · kimi · astra · spark新</div></div>
</div>
<div class="scroll"><table>
<thead><tr><th>关</th><th>家族</th><th class="n">grok新①</th><th class="n">grok新②</th><th class="n">kimi</th><th class="n">astra</th><th class="n">spark新</th><th>一致</th></tr></thead>
<tbody>
<tr><td>L1</td><td>switchback</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L2</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L3</td><td>weave</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L4</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L5</td><td>weave</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L6</td><td>weave</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L7</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L8</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L9</td><td>weave</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L10</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L11</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L12</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L13</td><td>orbits</td><td class="n">4/4/1</td><td class="n">10/8/3</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag signal">✘</span></td></tr>
<tr><td>L14</td><td>crossing</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L15</td><td>orbits</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L16</td><td>orbits</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L17</td><td>spiral</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
<tr><td>L18</td><td>spiral</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">3/4/0</td><td><span class="tag signal">✘</span></td></tr>
</tbody></table></div>
<div class="callout good">
<p><b>这是全份报告里最硬的一条结论。</b>16/18 关上,独立运行的不同模型给出了完全相同的旋转三元组。这意味着<b>规则机器人 12 秒扫出来的这三个数,等价于大模型跑几小时的结果</b>——这个维度完全不需要 agent。</p>
</div>
<h3>不一致的关:不是难,是不稳定</h3>
<p class="sub">下面这些关在不同运行之间给出了不同的数字。排除模型差异后,剩下的解释是<b>同一关的体验取决于起手方向</b>——两个玩家玩同一关,一个觉得还行,一个觉得这关有毛病。这比"稳定地难"更值得处理。</p>
<div class="note"><b>L13 orbits</b>:4/4/1 / 10/8/3 / 4/4/1 / 4/4/1 / 4/4/1</div><div class="note"><b>L18 spiral</b>:4/4/1 / 4/4/1 / 4/4/1 / 4/4/1 / 3/4/0</div>
<h2><span class="num">04</span>家族结构:机械指标指向同一个地方</h2>
<p>把「多转」按路线家族平均。这是关卡结构问题,不是玩家问题——两批 grok、以及 spark,在这一点上给出一致的结果。</p>
<div class="card"><h3 style="margin-top:0">grok-4.6 · 旧盘面 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">4.0</b></div>
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:66.7%" class=""></i></span>
<b class="bar-v">2.7</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="card"><h3 style="margin-top:0">grok-4.6 · 新盘面 · 第 1 次 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">4.0</b></div>
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:66.7%" class=""></i></span>
<b class="bar-v">2.7</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="card"><h3 style="margin-top:0">grok-4.6 · 新盘面 · 第 2 次 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">4.7</b></div>
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:85.7%" class=""></i></span>
<b class="bar-v">4.0</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:16.1%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:16.1%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="card"><h3 style="margin-top:0">codex-spark · 旧盘面 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">switchback<small>3 关</small></span>
<span class="bar-track"><i style="width:33.3%" class=""></i></span>
<b class="bar-v">0.3</b></div>
<div class="bar-row"><span class="bar-k">weave<small>3 关</small></span>
<span class="bar-track"><i style="width:33.3%" class=""></i></span>
<b class="bar-v">0.3</b></div></div></div>
<div class="card"><h3 style="margin-top:0">gpt-6-astra · low · 新盘面 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">4.0</b></div>
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:66.7%" class=""></i></span>
<b class="bar-v">2.7</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="card"><h3 style="margin-top:0">codex-spark · xhigh · 新盘面 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">3.5</b></div>
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:76.2%" class=""></i></span>
<b class="bar-v">2.7</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:21.4%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:21.4%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="card"><h3 style="margin-top:0">kimi-k3 · high · 新盘面 · 各家族平均多转</h3><div class="bars">
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
<b class="bar-v">4.0</b></div>
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
<span class="bar-track"><i style="width:66.7%" class=""></i></span>
<b class="bar-v">2.7</b></div>
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
<span class="bar-track"><i style="width:18.8%" class=""></i></span>
<b class="bar-v">0.8</b></div>
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div>
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
<b class="bar-v">0.0</b></div></div></div>
<div class="callout bad">
<p><b>orbits 和 spiral 是问题家族。</b>它们稳定地需要 7 次旋转(最少 3 次)外加 4 次盲转——等于把 6 个朝向挨个试一遍。而 petals 和 crossing 是干净的 0。锁住率解释不了这个差异(L16 也是 orbits、锁 78%,却是 3/3)。</p>
</div>
<h2><span class="num">05</span>Agent 说了什么</h2>
<p class="sub">评分是 1–5 的数字,<code>worst</code> 是它自己写的一句话。下面按反复出现的主题归类——<b>价值在重复模式,不在单条内容</b>。</p>
<h3>擦碰看不出来 · 28 次提及</h3>
<div class="note"><b>L1 switchback</b>(grok旧):开局 t2/t11 路上看着没人其实会擦到,差点空点;只剩 t0 和 t3 对堵时短暂以为卡住了</div>
<div class="note"><b>L3 weave</b>(grok旧):t5走后(第9步)t2和t3都像能出界,其实互相擦碰,只能再转</div>
<div class="note"><b>L3 weave</b>(grok新①):朝向1时 t2 和 t3 互相擦碰、t0 被 t2 压着,短暂不确定是再转还是还有能点的</div>
<div class="note"><b>L3 weave</b>(grok新②):第9步清完后t2和t3互相擦挡,谁先走都不行,只能转</div>
<h3>不知道该转哪边 · 17 次提及</h3>
<div class="note"><b>L2 switchback</b>(grok旧):第 13 步右转后朝向 3 没新出口,t0 和 t3 仍互相挡,不确定该不该继续同向转</div>
<div class="note"><b>L2 switchback</b>(grok新①):第15步右转后朝向3没有新出口,最后两枚还互压,一度担心转错了方向</div>
<div class="note"><b>L2 switchback</b>(spark旧):第 8-9 步清掉 t5,t4,t2 后只剩 t0/t3 互压,朝向2不再有出口,开始不确定该是继续走还是转哪边</div>
<div class="note"><b>L3 weave</b>(spark旧):第 5 步:第一轮转盘前所有可见线路都被压住/擦边,必须停下来判断该往哪个方向转</div>
<h3>长路线读不动 · 23 次提及</h3>
<div class="note"><b>L1 switchback</b>(kimi):第 10 步后 t0/t2/t3/t10 四枚互相压成环,盘面上看不出解法,只能靠旋转试出新朝向</div>
<div class="note"><b>L6 weave</b>(grok新①):开局 t0/t1 路线三十多格绕完全盘,只能靠路上有谁的摘要,看不出线实际怎么走</div>
<div class="note"><b>L7 petals</b>(grok旧):开局 t0/t3 各 19 格 15 弯还叠着好几枚,文字路线太长,分不清该先解哪一层</div>
<div class="note"><b>L7 petals</b>(grok新①):第 10 步再右转只开了 t7,t0 和 t3 仍 19 格对压,不确定是不是转错方向</div>
<h2><span class="num">06</span>1000 关该怎么跑</h2>
<p>核心判断:<b>机械计数器当主力,agent 做抽检</b>。免费的规则机器人能扫全量并稳定给出盲转/多转/转错;agent 贵但能提供两样机器给不了的东西——难度的相对排序,和用自然语言点出的系统性缺陷。<b>关键是只收集经过信度检验的那几项</b>。</p>
<ol class="steps">
<li><h3>生成期硬门槛 · 零成本</h3><p class="sub">用已有求解器,在生成时就卡掉:最少旋转 ≥ 2、开局锁住率 ≤ 90%、首转两个方向都能推进、无死局状态。不合格的根本不进候选池。</p></li>
<li><h3>规则机器人全量扫 · 1000 关 12 秒</h3><p class="sub">产出盲转 / 多转 / 转错方向。按 <code>盲转 ≥ 2 或 多转 ≥ 2</code> 打标。这条规则在 D5 上正好命中 orbits+spiral 那几关,和两个大模型跑两小时得到的结论一致。</p></li>
<li><h3>Agent 只跑被标记的 + 5% 随机抽样</h3><p class="sub">不是 1000 关全跑。收集<b>三样东西</b>:<code>difficulty</code>(r=0.93,可用来给 1000 关排序)、<code>stuck</code>(r=0.85,体感无解的直接信号)、以及 <code>worst</code> 文字(重复模式能挖出系统性缺陷)。<b>丢掉 rotation / clarity / fun</b>——复测 r 只有 0.32 / 0.19 / 0.56,是模型噪声。按实测 43 s/关、4 路并行,150 关约半小时。</p></li>
<li><h3>真人标定锚点 · 5–10 关</h3><p class="sub">Agent 的难度<b>排序</b>可信(复测 r=0.93),但绝对值可能整体偏移——它觉得「刚好」的关,真人可能觉得太简单。所以真人只需要玩少数几关当<b>锚点</b>,把 agent 的相对排序映射到真实难度刻度上,不必逐关评。这比原本设想的省很多人力。</p></li>
</ol>
<div class="callout good">
<p><b>最该做的一件事:把它变成回归测试,而不是评审流程。</b>改完生成器,12 秒跑一遍规则机器人,比较盲转分布。从 22% 降到 5% 就是客观改善,升上去就是刚引入了退化。12 秒一轮可以进 CI;几小时一轮且分数不可复现的人工评审,不可能成为迭代循环。</p>
</div>
<h2><span class="num">07</span>这份结论的边界</h2>
<p class="sub">写清楚哪里不够硬,比多列几个数字重要。</p>
<ul class="sub">
<li>跨模型一致性只验证了 <b>6 关</b>(spark 与 grok 的交集),其中 1 关不一致。机械指标"与玩家无关"这个说法,样本还太小。</li>
<li>复测只做了 <b>2 次</b>。2 次能算相关系数,但对评分这种低方差的离散量,2 次的估计本身就有不小的不确定性。</li>
<li>Agent 读的是文字不是像素,<b>测不出"看不清"</b>。clarity 这一项从原理上就不该指望它。</li>
<li>所有结论都基于 <b>D5 原型这 18 关</b>。这批关卡通关率 100%、难度评分方差极小,换一批更难的内容,结论可能不同。</li>
</ul>
<footer>
<p class="mono">数据:bench/runs · 生成脚本:scripts/build-bench-report.mjs · 重新生成:<code>node scripts/build-bench-report.mjs</code></p>
<p>本页所有数字由脚本从存档直接计算,未经手工填写。相关系数为 Pearson r,标准差为总体标准差。</p>
</footer>
</div>
</body>
</html>