docs: five-way cross-check, and correct a mislabelled batch

One batch of 18 runs was recorded as codex-spark at xhigh but had actually run
as gpt-6-astra at low effort: the CLI config was rewritten between the check
and the batch, and codex resolved the new model silently. The archives are
relabelled from the CLI's own startup header rather than from what the agent
was told to call itself, and the report now says to trust that header over the
self-reported name. The re-run verifies the resolved model after every level
and aborts the batch on a mismatch.

With the real codex-spark batch added, five independent runs of four models
from three vendors now cover the same 18 levels on the fixed board, and 16 of
17 comparable levels agree exactly on the rotation triple.

codex-spark gave up on L18 after cycling all six orientations, reporting no
exit anywhere — replaying the position shows orientation 1 releases t0, and
that it is the only one of the six that does. The level is solvable and the
routes are legible (it rated clarity 5), but nothing on the board says which
way to turn: a 1-in-6 guess with a dead end behind every other door. That is
the terminal-interlock complaint the other models kept raising, at its limit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
cabinw
2026-09-14 15:37:38 +08:00
co-authored by Claude Opus 5
parent d123b17a54
commit cefb7a5a56
2 changed files with 61 additions and 37 deletions
+7 -5
View File
@@ -31,13 +31,15 @@ const BATCHES = [
{ tag: 'pi · Grok 4.6 xhigh', label: 'grok-4.6 · 新盘面 · 第 1 次', short: 'grok新①', format: 'new' },
{ tag: 'pi · Grok 4.6 xhigh · 复测', label: 'grok-4.6 · 新盘面 · 第 2 次', short: 'grok新②', format: 'new' },
{ tag: 'Codex CLI · xhigh', label: 'codex-spark · 旧盘面', short: 'spark旧', format: 'old' },
{ tag: 'Codex CLI · xhigh · 新盘面', label: 'codex-spark · 新盘面', short: 'spark新', format: 'new' },
{ tag: 'Kimi CLI · k3 high', label: 'kimi-k3 · 新盘面', short: 'kimi', format: 'new' },
{ tag: 'Codex CLI · gpt-6-astra low', label: 'gpt-6-astra · low · 新盘面', short: 'astra', format: 'new' },
{ tag: 'Codex CLI · spark xhigh', label: 'codex-spark · xhigh · 新盘面', short: 'spark新', format: 'new' },
{ tag: 'Kimi CLI · k3 high', label: 'kimi-k3 · high · 新盘面', short: 'kimi', format: 'new' },
];
// Batches run under the fixed board format, i.e. everything that can be
// compared across models without the format change confounding it.
const NEW_TAGS = ['pi · Grok 4.6 xhigh', 'pi · Grok 4.6 xhigh · 复测', 'Kimi CLI · k3 high', 'Codex CLI · xhigh · 新盘面'];
const NEW_TAGS = ['pi · Grok 4.6 xhigh', 'pi · Grok 4.6 xhigh · 复测', 'Kimi CLI · k3 high',
'Codex CLI · gpt-6-astra low', 'Codex CLI · spark xhigh'];
const runs = readdirSync(RUNS).filter(f => f.endsWith('.json') && f !== 'index.json')
.map(f => { try { return JSON.parse(readFileSync(resolve(RUNS, f), 'utf8')); } catch { return null; } })
@@ -78,7 +80,7 @@ const METRICS = [
];
function pair(tagA, tagB) {
const A = byTag[tagA], B = byTag[tagB];
const A = byTag[tagA] || {}, B = byTag[tagB] || {};
const shared = Object.keys(A).filter(k => B[k]).map(Number).sort((a, b) => a - b);
return { shared, A, B };
}
@@ -172,7 +174,6 @@ function batchSummary(b) {
const summaries = present.map(batchSummary).filter(Boolean);
const clean = reliability('pi · Grok 4.6 xhigh', 'pi · Grok 4.6 xhigh · 复测');
const confounded = reliability('Grok Build CLI · xhigh', 'pi · Grok 4.6 xhigh');
const sparkPair = reliability('Codex CLI · xhigh · 新盘面', 'pi · Grok 4.6 xhigh');
// worst-step notes, grouped by the theme they name
const THEMES = [
@@ -340,6 +341,7 @@ footer{margin-top:64px;padding-top:22px;border-top:1px solid var(--rule);color:v
<h2><span class="num">01</span>参与检验的批次</h2>
<p class="sub">每一批用 <code>agent</code> 字段标识,不靠文件名猜来源。「旧盘面」指擦碰只报第一个阻挡者的版本,「新盘面」指修复后全部列出的版本。</p>
<p class="sub">模型名以 <b>CLI 启动时打印的解析结果</b>为准,而不是提示词里让 agent 自报的名字——有一批 18 局曾被标成 codex-spark,实际是配置在运行前被改掉、跑成了 gpt-6-astra/low,存档已按日志头更正。</p>
<div class="scroll"><table>
<thead><tr><th>批次</th><th class="n">关数</th><th class="n">通关</th><th class="n">空点</th><th class="n">擦碰空点</th><th class="n">盲转</th><th class="n">转错</th><th class="n">多转</th><th class="n">单局均</th><th class="n">每决策</th></tr></thead>
<tbody>