docs: five-way cross-check, and correct a mislabelled batch
One batch of 18 runs was recorded as codex-spark at xhigh but had actually run as gpt-6-astra at low effort: the CLI config was rewritten between the check and the batch, and codex resolved the new model silently. The archives are relabelled from the CLI's own startup header rather than from what the agent was told to call itself, and the report now says to trust that header over the self-reported name. The re-run verifies the resolved model after every level and aborts the batch on a mismatch. With the real codex-spark batch added, five independent runs of four models from three vendors now cover the same 18 levels on the fixed board, and 16 of 17 comparable levels agree exactly on the rotation triple. codex-spark gave up on L18 after cycling all six orientations, reporting no exit anywhere — replaying the position shows orientation 1 releases t0, and that it is the only one of the six that does. The level is solvable and the routes are legible (it rated clarity 5), but nothing on the board says which way to turn: a 1-in-6 guess with a dead end behind every other door. That is the terminal-interlock complaint the other models kept raising, at its limit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+54
-32
@@ -103,11 +103,12 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
|
||||
<div class="eyebrow">HEX TRAILS · 关卡实验台</div>
|
||||
<h1>Agent 评级里,哪些数字真能拿来做关卡</h1>
|
||||
<p class="lede">同一批关卡、同一个模型、同一套盘面,跑两遍。能重现的指标才配进流程——这份报告把每个指标逐个检验了一遍,结论和直觉相反。</p>
|
||||
<p class="meta mono">生成于 2026-09-14 14:57 GMT+8 · 数据来自 bench/runs 的 157 局 · 关卡集 levels/prototypes/d5.json(18 关)</p>
|
||||
<p class="meta mono">生成于 2026-09-14 15:37 GMT+8 · 数据来自 bench/runs 的 175 局 · 关卡集 levels/prototypes/d5.json(18 关)</p>
|
||||
</header>
|
||||
|
||||
<h2><span class="num">01</span>参与检验的批次</h2>
|
||||
<p class="sub">每一批用 <code>agent</code> 字段标识,不靠文件名猜来源。「旧盘面」指擦碰只报第一个阻挡者的版本,「新盘面」指修复后全部列出的版本。</p>
|
||||
<p class="sub">模型名以 <b>CLI 启动时打印的解析结果</b>为准,而不是提示词里让 agent 自报的名字——有一批 18 局曾被标成 codex-spark,实际是配置在运行前被改掉、跑成了 gpt-6-astra/low,存档已按日志头更正。</p>
|
||||
<div class="scroll"><table>
|
||||
<thead><tr><th>批次</th><th class="n">关数</th><th class="n">通关</th><th class="n">空点</th><th class="n">擦碰空点</th><th class="n">盲转</th><th class="n">转错</th><th class="n">多转</th><th class="n">单局均</th><th class="n">每决策</th></tr></thead>
|
||||
<tbody>
|
||||
@@ -115,10 +116,11 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
|
||||
<tr><td>grok-4.6 · 新盘面 · 第 1 次</td><td class="n">18</td><td class="n">18/18</td><td class="n">2</td><td class="n">1</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">55 s</td><td class="n">3.9 s</td></tr>
|
||||
<tr><td>grok-4.6 · 新盘面 · 第 2 次</td><td class="n">18</td><td class="n">18/18</td><td class="n">0</td><td class="n">0</td><td class="n">25</td><td class="n">10</td><td class="n">28</td><td class="n">49 s</td><td class="n">3.9 s</td></tr>
|
||||
<tr><td>codex-spark · 旧盘面</td><td class="n">6</td><td class="n">6/6</td><td class="n">0</td><td class="n">0</td><td class="n">2</td><td class="n">2</td><td class="n">2</td><td class="n">15 s</td><td class="n">1.4 s</td></tr>
|
||||
<tr><td>codex-spark · 新盘面</td><td class="n">18</td><td class="n">18/18</td><td class="n">1</td><td class="n">0</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">41 s</td><td class="n">7.0 s</td></tr>
|
||||
<tr><td>kimi-k3 · 新盘面</td><td class="n">18</td><td class="n">18/18</td><td class="n">0</td><td class="n">0</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">48 s</td><td class="n">5.2 s</td></tr>
|
||||
<tr><td>gpt-6-astra · low · 新盘面</td><td class="n">18</td><td class="n">18/18</td><td class="n">1</td><td class="n">0</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">41 s</td><td class="n">7.0 s</td></tr>
|
||||
<tr><td>codex-spark · xhigh · 新盘面</td><td class="n">18</td><td class="n">17/18</td><td class="n">0</td><td class="n">0</td><td class="n">21</td><td class="n">7</td><td class="n">21</td><td class="n">21 s</td><td class="n">1.8 s</td></tr>
|
||||
<tr><td>kimi-k3 · high · 新盘面</td><td class="n">18</td><td class="n">18/18</td><td class="n">0</td><td class="n">0</td><td class="n">21</td><td class="n">8</td><td class="n">22</td><td class="n">48 s</td><td class="n">5.2 s</td></tr>
|
||||
</tbody></table></div>
|
||||
<p class="sub">所有批次通关率都是 100%。<b>这本身就是一条结论:通关率在这批内容上毫无区分度,任何流程都不该建立在通过/失败上。</b></p>
|
||||
<p class="sub">已完成的对局通关率全部是 100%(个别批次仍有对局在进行中,未计入)。<b>通关率在这批内容上毫无区分度,任何流程都不该建立在通过/失败上。</b></p>
|
||||
|
||||
|
||||
<h2><span class="num">02</span>信度检验:跑两遍还一样吗</h2>
|
||||
@@ -236,38 +238,38 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
|
||||
|
||||
<p>下表是同一套盘面下所有批次的 <code>多转 / 盲转 / 转错方向</code>。不同厂商的模型、不同 harness、各自独立跑——如果这些数字真的由关卡决定,它们就该一样。</p>
|
||||
<div class="grid">
|
||||
<div class="tile"><div class="k">逐关完全一致</div><div class="v">17 / 18</div><div class="n">三项数字一个不差</div></div>
|
||||
<div class="tile"><div class="k">参与批次</div><div class="v">4</div><div class="n">grok新① · grok新② · kimi · spark新</div></div>
|
||||
<div class="tile"><div class="k">逐关完全一致</div><div class="v">16 / 18</div><div class="n">三项数字一个不差</div></div>
|
||||
<div class="tile"><div class="k">参与批次</div><div class="v">5</div><div class="n">grok新① · grok新② · kimi · astra · spark新</div></div>
|
||||
</div>
|
||||
<div class="scroll"><table>
|
||||
<thead><tr><th>关</th><th>家族</th><th class="n">grok新①</th><th class="n">grok新②</th><th class="n">kimi</th><th class="n">spark新</th><th>一致</th></tr></thead>
|
||||
<thead><tr><th>关</th><th>家族</th><th class="n">grok新①</th><th class="n">grok新②</th><th class="n">kimi</th><th class="n">astra</th><th class="n">spark新</th><th>一致</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>L1</td><td>switchback</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L2</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L3</td><td>weave</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L4</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L5</td><td>weave</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L6</td><td>weave</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L7</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L8</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L9</td><td>weave</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L10</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L11</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L12</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L13</td><td>orbits</td><td class="n">4/4/1</td><td class="n">10/8/3</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag signal">✘</span></td></tr>
|
||||
<tr><td>L14</td><td>crossing</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L15</td><td>orbits</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L16</td><td>orbits</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L17</td><td>spiral</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L18</td><td>spiral</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L1</td><td>switchback</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L2</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L3</td><td>weave</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L4</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L5</td><td>weave</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L6</td><td>weave</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L7</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L8</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L9</td><td>weave</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td class="n">2/1/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L10</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L11</td><td>petals</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L12</td><td>switchback</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td class="n">1/1/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L13</td><td>orbits</td><td class="n">4/4/1</td><td class="n">10/8/3</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag signal">✘</span></td></tr>
|
||||
<tr><td>L14</td><td>crossing</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L15</td><td>orbits</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L16</td><td>orbits</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td class="n">0/0/0</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L17</td><td>spiral</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td><span class="tag good">✔</span></td></tr>
|
||||
<tr><td>L18</td><td>spiral</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">4/4/1</td><td class="n">3/4/0</td><td><span class="tag signal">✘</span></td></tr>
|
||||
</tbody></table></div>
|
||||
<div class="callout good">
|
||||
<p><b>这是全份报告里最硬的一条结论。</b>17/18 关上,独立运行的不同模型给出了完全相同的旋转三元组。这意味着<b>规则机器人 12 秒扫出来的这三个数,等价于大模型跑几小时的结果</b>——这个维度完全不需要 agent。</p>
|
||||
<p><b>这是全份报告里最硬的一条结论。</b>16/18 关上,独立运行的不同模型给出了完全相同的旋转三元组。这意味着<b>规则机器人 12 秒扫出来的这三个数,等价于大模型跑几小时的结果</b>——这个维度完全不需要 agent。</p>
|
||||
</div>
|
||||
|
||||
<h3>不一致的关:不是难,是不稳定</h3>
|
||||
<p class="sub">下面这些关在不同运行之间给出了不同的数字。排除模型差异后,剩下的解释是<b>同一关的体验取决于起手方向</b>——两个玩家玩同一关,一个觉得还行,一个觉得这关有毛病。这比"稳定地难"更值得处理。</p>
|
||||
<div class="note"><b>L13 orbits</b>:4/4/1 / 10/8/3 / 4/4/1 / 4/4/1</div>
|
||||
<div class="note"><b>L13 orbits</b>:4/4/1 / 10/8/3 / 4/4/1 / 4/4/1 / 4/4/1</div><div class="note"><b>L18 spiral</b>:4/4/1 / 4/4/1 / 4/4/1 / 4/4/1 / 3/4/0</div>
|
||||
|
||||
<h2><span class="num">04</span>家族结构:机械指标指向同一个地方</h2>
|
||||
<p>把「多转」按路线家族平均。这是关卡结构问题,不是玩家问题——两批 grok、以及 spark,在这一点上给出一致的结果。</p>
|
||||
@@ -340,7 +342,7 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
|
||||
<span class="bar-track"><i style="width:33.3%" class=""></i></span>
|
||||
<b class="bar-v">0.3</b></div></div></div>
|
||||
|
||||
<div class="card"><h3 style="margin-top:0">codex-spark · 新盘面 · 各家族平均多转</h3><div class="bars">
|
||||
<div class="card"><h3 style="margin-top:0">gpt-6-astra · low · 新盘面 · 各家族平均多转</h3><div class="bars">
|
||||
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
|
||||
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
|
||||
<b class="bar-v">4.0</b></div>
|
||||
@@ -360,7 +362,27 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
|
||||
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
|
||||
<b class="bar-v">0.0</b></div></div></div>
|
||||
|
||||
<div class="card"><h3 style="margin-top:0">kimi-k3 · 新盘面 · 各家族平均多转</h3><div class="bars">
|
||||
<div class="card"><h3 style="margin-top:0">codex-spark · xhigh · 新盘面 · 各家族平均多转</h3><div class="bars">
|
||||
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
|
||||
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
|
||||
<b class="bar-v">3.5</b></div>
|
||||
<div class="bar-row"><span class="bar-k">orbits<small>3 关</small></span>
|
||||
<span class="bar-track"><i style="width:76.2%" class=""></i></span>
|
||||
<b class="bar-v">2.7</b></div>
|
||||
<div class="bar-row"><span class="bar-k">switchback<small>4 关</small></span>
|
||||
<span class="bar-track"><i style="width:21.4%" class=""></i></span>
|
||||
<b class="bar-v">0.8</b></div>
|
||||
<div class="bar-row"><span class="bar-k">weave<small>4 关</small></span>
|
||||
<span class="bar-track"><i style="width:21.4%" class=""></i></span>
|
||||
<b class="bar-v">0.8</b></div>
|
||||
<div class="bar-row"><span class="bar-k">petals<small>4 关</small></span>
|
||||
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
|
||||
<b class="bar-v">0.0</b></div>
|
||||
<div class="bar-row"><span class="bar-k">crossing<small>1 关</small></span>
|
||||
<span class="bar-track"><i style="width:0.0%" class="zero"></i></span>
|
||||
<b class="bar-v">0.0</b></div></div></div>
|
||||
|
||||
<div class="card"><h3 style="margin-top:0">kimi-k3 · high · 新盘面 · 各家族平均多转</h3><div class="bars">
|
||||
<div class="bar-row"><span class="bar-k">spiral<small>2 关</small></span>
|
||||
<span class="bar-track"><i style="width:100.0%" class=""></i></span>
|
||||
<b class="bar-v">4.0</b></div>
|
||||
@@ -386,7 +408,7 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
|
||||
<h2><span class="num">05</span>Agent 说了什么</h2>
|
||||
<p class="sub">评分是 1–5 的数字,<code>worst</code> 是它自己写的一句话。下面按反复出现的主题归类——<b>价值在重复模式,不在单条内容</b>。</p>
|
||||
|
||||
<h3>擦碰看不出来 · 27 次提及</h3>
|
||||
<h3>擦碰看不出来 · 28 次提及</h3>
|
||||
<div class="note"><b>L1 switchback</b>(grok旧):开局 t2/t11 路上看着没人其实会擦到,差点空点;只剩 t0 和 t3 对堵时短暂以为卡住了</div>
|
||||
<div class="note"><b>L3 weave</b>(grok旧):t5走后(第9步)t2和t3都像能出界,其实互相擦碰,只能再转</div>
|
||||
<div class="note"><b>L3 weave</b>(grok新①):朝向1时 t2 和 t3 互相擦碰、t0 被 t2 压着,短暂不确定是再转还是还有能点的</div>
|
||||
@@ -398,7 +420,7 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
|
||||
<div class="note"><b>L2 switchback</b>(spark旧):第 8-9 步清掉 t5,t4,t2 后只剩 t0/t3 互压,朝向2不再有出口,开始不确定该是继续走还是转哪边</div>
|
||||
<div class="note"><b>L3 weave</b>(spark旧):第 5 步:第一轮转盘前所有可见线路都被压住/擦边,必须停下来判断该往哪个方向转</div>
|
||||
|
||||
<h3>长路线读不动 · 22 次提及</h3>
|
||||
<h3>长路线读不动 · 23 次提及</h3>
|
||||
<div class="note"><b>L1 switchback</b>(kimi):第 10 步后 t0/t2/t3/t10 四枚互相压成环,盘面上看不出解法,只能靠旋转试出新朝向</div>
|
||||
<div class="note"><b>L6 weave</b>(grok新①):开局 t0/t1 路线三十多格绕完全盘,只能靠路上有谁的摘要,看不出线实际怎么走</div>
|
||||
<div class="note"><b>L7 petals</b>(grok旧):开局 t0/t3 各 19 格 15 弯还叠着好几枚,文字路线太长,分不清该先解哪一层</div>
|
||||
@@ -409,7 +431,7 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
|
||||
<ol class="steps">
|
||||
<li><h3>生成期硬门槛 · 零成本</h3><p class="sub">用已有求解器,在生成时就卡掉:最少旋转 ≥ 2、开局锁住率 ≤ 90%、首转两个方向都能推进、无死局状态。不合格的根本不进候选池。</p></li>
|
||||
<li><h3>规则机器人全量扫 · 1000 关 12 秒</h3><p class="sub">产出盲转 / 多转 / 转错方向。按 <code>盲转 ≥ 2 或 多转 ≥ 2</code> 打标。这条规则在 D5 上正好命中 orbits+spiral 那几关,和两个大模型跑两小时得到的结论一致。</p></li>
|
||||
<li><h3>Agent 只跑被标记的 + 5% 随机抽样</h3><p class="sub">不是 1000 关全跑。收集<b>三样东西</b>:<code>difficulty</code>(r=0.93,可用来给 1000 关排序)、<code>stuck</code>(r=0.85,体感无解的直接信号)、以及 <code>worst</code> 文字(重复模式能挖出系统性缺陷)。<b>丢掉 rotation / clarity / fun</b>——复测 r 只有 0.32 / 0.19 / 0.56,是模型噪声。按实测 48 s/关、4 路并行,150 关约半小时。</p></li>
|
||||
<li><h3>Agent 只跑被标记的 + 5% 随机抽样</h3><p class="sub">不是 1000 关全跑。收集<b>三样东西</b>:<code>difficulty</code>(r=0.93,可用来给 1000 关排序)、<code>stuck</code>(r=0.85,体感无解的直接信号)、以及 <code>worst</code> 文字(重复模式能挖出系统性缺陷)。<b>丢掉 rotation / clarity / fun</b>——复测 r 只有 0.32 / 0.19 / 0.56,是模型噪声。按实测 43 s/关、4 路并行,150 关约半小时。</p></li>
|
||||
<li><h3>真人标定锚点 · 5–10 关</h3><p class="sub">Agent 的难度<b>排序</b>可信(复测 r=0.93),但绝对值可能整体偏移——它觉得「刚好」的关,真人可能觉得太简单。所以真人只需要玩少数几关当<b>锚点</b>,把 agent 的相对排序映射到真实难度刻度上,不必逐关评。这比原本设想的省很多人力。</p></li>
|
||||
</ol>
|
||||
<div class="callout good">
|
||||
|
||||
@@ -31,13 +31,15 @@ const BATCHES = [
|
||||
{ tag: 'pi · Grok 4.6 xhigh', label: 'grok-4.6 · 新盘面 · 第 1 次', short: 'grok新①', format: 'new' },
|
||||
{ tag: 'pi · Grok 4.6 xhigh · 复测', label: 'grok-4.6 · 新盘面 · 第 2 次', short: 'grok新②', format: 'new' },
|
||||
{ tag: 'Codex CLI · xhigh', label: 'codex-spark · 旧盘面', short: 'spark旧', format: 'old' },
|
||||
{ tag: 'Codex CLI · xhigh · 新盘面', label: 'codex-spark · 新盘面', short: 'spark新', format: 'new' },
|
||||
{ tag: 'Kimi CLI · k3 high', label: 'kimi-k3 · 新盘面', short: 'kimi', format: 'new' },
|
||||
{ tag: 'Codex CLI · gpt-6-astra low', label: 'gpt-6-astra · low · 新盘面', short: 'astra', format: 'new' },
|
||||
{ tag: 'Codex CLI · spark xhigh', label: 'codex-spark · xhigh · 新盘面', short: 'spark新', format: 'new' },
|
||||
{ tag: 'Kimi CLI · k3 high', label: 'kimi-k3 · high · 新盘面', short: 'kimi', format: 'new' },
|
||||
];
|
||||
|
||||
// Batches run under the fixed board format, i.e. everything that can be
|
||||
// compared across models without the format change confounding it.
|
||||
const NEW_TAGS = ['pi · Grok 4.6 xhigh', 'pi · Grok 4.6 xhigh · 复测', 'Kimi CLI · k3 high', 'Codex CLI · xhigh · 新盘面'];
|
||||
const NEW_TAGS = ['pi · Grok 4.6 xhigh', 'pi · Grok 4.6 xhigh · 复测', 'Kimi CLI · k3 high',
|
||||
'Codex CLI · gpt-6-astra low', 'Codex CLI · spark xhigh'];
|
||||
|
||||
const runs = readdirSync(RUNS).filter(f => f.endsWith('.json') && f !== 'index.json')
|
||||
.map(f => { try { return JSON.parse(readFileSync(resolve(RUNS, f), 'utf8')); } catch { return null; } })
|
||||
@@ -78,7 +80,7 @@ const METRICS = [
|
||||
];
|
||||
|
||||
function pair(tagA, tagB) {
|
||||
const A = byTag[tagA], B = byTag[tagB];
|
||||
const A = byTag[tagA] || {}, B = byTag[tagB] || {};
|
||||
const shared = Object.keys(A).filter(k => B[k]).map(Number).sort((a, b) => a - b);
|
||||
return { shared, A, B };
|
||||
}
|
||||
@@ -172,7 +174,6 @@ function batchSummary(b) {
|
||||
const summaries = present.map(batchSummary).filter(Boolean);
|
||||
const clean = reliability('pi · Grok 4.6 xhigh', 'pi · Grok 4.6 xhigh · 复测');
|
||||
const confounded = reliability('Grok Build CLI · xhigh', 'pi · Grok 4.6 xhigh');
|
||||
const sparkPair = reliability('Codex CLI · xhigh · 新盘面', 'pi · Grok 4.6 xhigh');
|
||||
|
||||
// worst-step notes, grouped by the theme they name
|
||||
const THEMES = [
|
||||
@@ -340,6 +341,7 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
|
||||
|
||||
<h2><span class="num">01</span>参与检验的批次</h2>
|
||||
<p class="sub">每一批用 <code>agent</code> 字段标识,不靠文件名猜来源。「旧盘面」指擦碰只报第一个阻挡者的版本,「新盘面」指修复后全部列出的版本。</p>
|
||||
<p class="sub">模型名以 <b>CLI 启动时打印的解析结果</b>为准,而不是提示词里让 agent 自报的名字——有一批 18 局曾被标成 codex-spark,实际是配置在运行前被改掉、跑成了 gpt-6-astra/low,存档已按日志头更正。</p>
|
||||
<div class="scroll"><table>
|
||||
<thead><tr><th>批次</th><th class="n">关数</th><th class="n">通关</th><th class="n">空点</th><th class="n">擦碰空点</th><th class="n">盲转</th><th class="n">转错</th><th class="n">多转</th><th class="n">单局均</th><th class="n">每决策</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
Reference in New Issue
Block a user