feat(levels): gate the dead turn, and state the curve instead of hoping for it
Round 1 played all fifty levels and came back with the mechanics clean and one
complaint repeated across the reports: 短暂以为转错了方向 — a turn that changed
nothing. That one is mechanical rather than perceptual, so it can be gated.
The first attempt gated `doubleDead`, and it was wrong. That counts dead states,
which scales with the state space: three-turn levels have ~187 reachable stuck
states and five-turn ones ~739, so a raw count made deep levels look eight times
worse when the rate is 2% against 5%. It would have erased every level past
three turns — the opposite of the brief, where turning is the signature and is
not to be rationed. The gate is `maxDeadTurns`, the longest run of turns before
something opens, at 1: one beat of suspense, then a release. Eleven of the fifty
shipped at 2.
Selection could not express the curve either. Spreading picks evenly over the
ranked pool returns whatever the pool is made of, and the pool is 88%
three-turn, so no weighting moved the distribution off {3: 46}. A quota per
depth states the curve outright, deepest first so the scarce end picks before a
family cap is spent on levels the shallow end has hundreds of.
Same rule bot, both sets: blind rotations 0.24 → 0.12, worst blind streak 4 → 1,
decisions unchanged at 10.3. The same amount to do, half the wasted motion.
Rotation now runs 1.8 / 3.0 / 4.1 across the bands against 1.8 / 3.0 / 3.5.
Verdicts are keyed by the board rather than by its id, because a reselect
re-issues ids from the ordinal and `c50-01-v05` is not the same board before and
after — the review sheet was one reselect away from printing one level's verdict
under another's thumbnail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,205 @@
|
||||
#!/usr/bin/env node
|
||||
// Build docs/level-process.html — what this round changed about how levels
|
||||
// get made. Every figure is measured here, not typed in.
|
||||
//
|
||||
// node scripts/build-process-report.mjs
|
||||
import { existsSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
|
||||
import { resolve } from 'node:path';
|
||||
import { trace } from '../src/core.js';
|
||||
import { metrics } from '../src/bench.js';
|
||||
import { legibility } from '../src/deep-lock.js';
|
||||
|
||||
const esc = s => String(s ?? '').replace(/[&<>"]/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"' }[c]));
|
||||
const mean = a => a.length ? a.reduce((x, y) => x + y, 0) / a.length : null;
|
||||
const f1 = x => x == null || Number.isNaN(x) ? '—' : (Math.round(x * 10) / 10).toFixed(1);
|
||||
const f2 = x => x == null || Number.isNaN(x) ? '—' : (Math.round(x * 100) / 100).toFixed(2);
|
||||
const pct = (a, b) => b ? Math.round(100 * a / b) + '%' : '—';
|
||||
|
||||
const levels = JSON.parse(readFileSync(resolve('levels/campaign-50.json'), 'utf8')).levels;
|
||||
const byId = new Map(levels.map(l => [l.id, l]));
|
||||
|
||||
// Every round on record, so the claim "agent rounds catch defects" is shown
|
||||
// against the rounds rather than asserted.
|
||||
const runs = [];
|
||||
for (const f of existsSync(resolve('bench/runs')) ? readdirSync(resolve('bench/runs')) : []) {
|
||||
if (!f.endsWith('.json') || f === 'index.json') continue;
|
||||
try {
|
||||
const d = JSON.parse(readFileSync(resolve('bench/runs', f), 'utf8'));
|
||||
if (d.status === 'playing') continue;
|
||||
runs.push({ raw: d, m: metrics(d) });
|
||||
} catch { /* torn read */ }
|
||||
}
|
||||
const r1 = runs.filter(r => r.raw.agent === 'R1 · grok xhigh' && byId.has(r.raw.level?.id));
|
||||
const rated = r1.filter(r => r.m.rating && Number.isFinite(Number(r.m.rating.difficulty)));
|
||||
const byDiff = rated.reduce((m, r) => (m[r.m.rating.difficulty] = (m[r.m.rating.difficulty] || 0) + 1, m), {});
|
||||
const topDiff = Object.entries(byDiff).sort((a, b) => b[1] - a[1])[0] ?? ['—', 0];
|
||||
|
||||
const longest = l => Math.max(...l.tiles.map(t => trace(l, t, [], true).steps.length));
|
||||
const c50Longest = Math.max(...levels.map(longest));
|
||||
const loadPool = p => { try { const r = JSON.parse(readFileSync(resolve(p), 'utf8')); return Array.isArray(r) ? r : r.levels; } catch { return []; } };
|
||||
const v05 = loadPool('bench/pool-easy.json');
|
||||
const v05Longest = v05.length ? Math.max(...v05.map(longest)) : null;
|
||||
const traps = levels.map(l => legibility(l).grazeTraps);
|
||||
|
||||
const decisions = rated.map(r => r.m.decisions ?? 0);
|
||||
const R1 = {
|
||||
n: r1.length, rated: rated.length,
|
||||
cleared: r1.filter(r => r.m.cleared).length,
|
||||
dead: mean(r1.map(r => r.m.deadTaps ?? 0)),
|
||||
blind: mean(r1.map(r => r.m.blindRotations ?? 0)),
|
||||
wrong: mean(r1.map(r => r.m.wrongWayRotations ?? 0)),
|
||||
flat: topDiff, decMin: Math.min(...decisions), decMax: Math.max(...decisions),
|
||||
};
|
||||
|
||||
// The rules, each with the observation that forced it. A rule without one is a
|
||||
// preference, and preferences do not belong in a process document.
|
||||
const RULES = [
|
||||
{ n: 1, rule: 'agent 轮次用来抓缺陷,不用来排难度',
|
||||
ev: `R1 的 ${R1.rated} 关里有 ${R1.flat[1]} 关难度评分是 ${R1.flat[0]} 分,而这些关卡的决策数从 ${R1.decMin} 到 ${R1.decMax}。评分不是不准,是不区分。`,
|
||||
why: '文字盘面把每颗棋子的路线、挡路者、紧贴者列成清单,agent 不必用眼睛顺线走一遍——而那正是这游戏的难点。agent 自己写过:「挡路名单已经写全,不用顺着格子数」。',
|
||||
now: '难度排序改为机械假设(旋转深度、棋子数、互锁),由人工审核裁定;agent 轮次只看死局、盲转、转错、空点、演出。' },
|
||||
{ n: 2, rule: '先用规则机器人扫,再花 agent 时间',
|
||||
ev: '机械旋转指标已被证明测的是关卡而不是玩家:4 次运行、3 家厂商,18 关里 17 关结论一致。',
|
||||
why: '既然同一关在不同模型下得出同样的机械读数,那这个读数不需要模型来产生。',
|
||||
now: '12 秒的规则机器人扫描替代一整轮 agent 跑批;agent 只用于抽样确认和读自由文本抱怨。' },
|
||||
{ n: 3, rule: '门槛写在生成期,不写在筛选期',
|
||||
ev: '把开局门槛提到「可点≥3 枚、演出≥0.8s」后筛选写出了 0 关——池子的中位数是 2 枚 / 0.57s。',
|
||||
why: '筛选只能从池子里挑,挑不出池子里没有的东西。分布不对时,要改的是构造。',
|
||||
now: '开局问题由构造里的展示位插入解决,不是靠筛。' },
|
||||
{ n: 4, rule: '每加一个门槛,先证明它挡掉了东西',
|
||||
ev: '两次结构性猜测被实测推翻:「插入端多朝向出口」过滤器对漏斗零效果;为解释难度评分而建的 legibility 指标与评分相关 -0.02。',
|
||||
why: '一个不改变漏斗的门槛,只会让人以为问题已经解决了。',
|
||||
now: '两者都已撤回/降级为纯测量。漏斗每一级的剩余数量都打印出来。' },
|
||||
{ n: 5, rule: '抱怨要对照基线,不要直接当缺陷',
|
||||
ev: `agent 33 条抱怨里最集中的是「路线太长读不完」。但已上线的 v0.5 最长路线 ${v05Longest ?? '—'} 格,比这批新关卡的 ${c50Longest} 格还长。`,
|
||||
why: '那是这个游戏本来的性格。把它当缺陷修掉,改的是游戏而不是关卡。',
|
||||
now: '只在审核页标出,交给人裁定。' },
|
||||
];
|
||||
|
||||
const html = `<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>关卡制作流程复盘</title>
|
||||
<style>
|
||||
:root{
|
||||
--ground:#EFF1F3; --surface:#FFF; --panel:#F6F8F9; --ink:#131C24; --ink2:#465766; --ink3:#6D7C89;
|
||||
--rule:#D4DAE0; --rail:#3F5E79; --good:#226D63; --bad:#AF3A26; --warn:#895B0E;
|
||||
color-scheme:light;
|
||||
}
|
||||
@media (prefers-color-scheme:dark){:root:not([data-theme=light]){
|
||||
--ground:#0E151B; --surface:#131C24; --panel:#131C24; --ink:#E3E9EE; --ink2:#ADBCC8; --ink3:#8393A1;
|
||||
--rule:#27333E; --rail:#8BAFCB; --good:#4EC6B6; --bad:#E3826B; --warn:#DFAD5D;
|
||||
color-scheme:dark;
|
||||
}}
|
||||
*{box-sizing:border-box}
|
||||
body{margin:0;background:var(--ground);color:var(--ink);
|
||||
font-family:"PingFang SC","Hiragino Sans GB",ui-sans-serif,system-ui,sans-serif;line-height:1.7}
|
||||
.wrap{max-width:920px;margin:0 auto;padding:44px 22px 96px}
|
||||
h1{font-size:31px;margin:0 0 12px;letter-spacing:-.02em}
|
||||
.lede{color:var(--ink2);font-size:15.5px;max-width:68ch;margin:0 0 6px}
|
||||
.meta{font-size:12.5px;color:var(--ink3);font-family:ui-monospace,monospace;margin:0 0 34px}
|
||||
h2{font-size:20px;margin:44px 0 12px;padding-bottom:7px;border-bottom:1px solid var(--rule)}
|
||||
p{max-width:70ch}
|
||||
.tiles{display:grid;gap:12px;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));margin:18px 0 8px}
|
||||
.tile{background:var(--surface);border:1px solid var(--rule);border-radius:10px;padding:13px 15px}
|
||||
.tile .k{font-size:11px;letter-spacing:.06em;text-transform:uppercase;color:var(--ink3);font-weight:600}
|
||||
.tile .v{font-size:25px;font-family:ui-monospace,monospace;font-variant-numeric:tabular-nums;margin:3px 0 1px}
|
||||
.tile .n{font-size:11.5px;color:var(--ink3)}
|
||||
.tile.ok .v{color:var(--good)} .tile.miss .v{color:var(--bad)}
|
||||
.rule{background:var(--surface);border:1px solid var(--rule);border-radius:12px;padding:17px 20px;margin:14px 0}
|
||||
.rule h3{margin:0 0 9px;font-size:16.5px;display:flex;gap:10px;align-items:baseline}
|
||||
.rule h3 .no{font-family:ui-monospace,monospace;color:var(--rail);font-weight:700}
|
||||
.rule dl{margin:0;display:grid;grid-template-columns:auto 1fr;gap:5px 13px;font-size:13.5px}
|
||||
.rule dt{color:var(--ink3);white-space:nowrap;font-size:11.5px;letter-spacing:.04em;padding-top:3px}
|
||||
.rule dd{margin:0;color:var(--ink2)}
|
||||
.rule dd.ev{color:var(--ink)}
|
||||
table{width:100%;border-collapse:collapse;font-size:13.5px;margin:14px 0 8px}
|
||||
th,td{text-align:left;padding:8px 10px;border-bottom:1px solid var(--rule);vertical-align:top}
|
||||
th{font-size:11px;letter-spacing:.06em;color:var(--ink3);text-transform:uppercase;font-weight:600}
|
||||
td.n{text-align:right;font-family:ui-monospace,monospace;white-space:nowrap}
|
||||
.no{color:var(--bad);font-weight:600}.yes{color:var(--good);font-weight:600}
|
||||
ol.recipe{counter-reset:s;list-style:none;padding:0;margin:14px 0}
|
||||
ol.recipe li{counter-increment:s;position:relative;padding:0 0 13px 40px;font-size:14px;color:var(--ink2);max-width:70ch}
|
||||
ol.recipe li::before{content:counter(s);position:absolute;left:0;top:0;width:25px;height:25px;border-radius:50%;
|
||||
background:var(--rail);color:#fff;font-family:ui-monospace,monospace;font-size:12.5px;font-weight:700;
|
||||
display:grid;place-items:center}
|
||||
ol.recipe b{color:var(--ink)}
|
||||
code{font-family:ui-monospace,monospace;font-size:.92em;background:var(--panel);padding:1px 5px;border-radius:4px;border:1px solid var(--rule)}
|
||||
footer{margin-top:54px;padding-top:18px;border-top:1px solid var(--rule);color:var(--ink3);font-size:12px}
|
||||
a{color:var(--rail)}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
<h1>关卡制作流程复盘</h1>
|
||||
<p class="lede">这一轮做 50 关的过程里,最有用的结论是一个<b>否定结论</b>:原计划的「agent 打分 → 拟合权重 → 重选关卡」这个闭环<b>合不上</b>,而且不是样本量的问题。下面是它为什么合不上,以及流程该改成什么样。</p>
|
||||
<p class="meta">实测批次 R1 · grok-4.6 xhigh · ${R1.n} 关 · 生成于 ${new Date().toLocaleString('sv-SE').slice(0, 16)}</p>
|
||||
|
||||
<h2>R1 实测到了什么</h2>
|
||||
<div class="tiles">
|
||||
<div class="tile ${R1.cleared === R1.n ? 'ok' : 'miss'}"><div class="k">通关</div><div class="v">${R1.cleared}/${R1.n}</div><div class="n">无死局</div></div>
|
||||
<div class="tile ${R1.dead === 0 ? 'ok' : 'miss'}"><div class="k">空点 / 关</div><div class="v">${f2(R1.dead)}</div><div class="n">目标 0</div></div>
|
||||
<div class="tile ${R1.blind < .5 ? 'ok' : 'miss'}"><div class="k">盲转 / 关</div><div class="v">${f2(R1.blind)}</div><div class="n">目标 0</div></div>
|
||||
<div class="tile ${R1.wrong < .5 ? 'ok' : 'miss'}"><div class="k">转错 / 关</div><div class="v">${f2(R1.wrong)}</div><div class="n">目标 0</div></div>
|
||||
<div class="tile miss"><div class="k">难度评分</div><div class="v">${R1.flat[1]}/${R1.rated}</div><div class="n">都是 ${R1.flat[0]} 分 — 不区分</div></div>
|
||||
</div>
|
||||
<p>机械面干净,说明生成期的闸门是有效的:「动了很多次没结果」这类结构缺陷已经被挡在关卡集之外。<b>难度评分那一格是这一轮真正的发现</b>——它不是不准(复测 r = 0.93,相当稳),它是<b>不区分</b>:机械上差着三倍的关卡,它一律给 ${R1.flat[0]} 分。</p>
|
||||
|
||||
<h2>五条规则,各自的证据</h2>
|
||||
${RULES.map(r => `<div class="rule">
|
||||
<h3><span class="no">${r.n}</span>${esc(r.rule)}</h3>
|
||||
<dl>
|
||||
<dt>证据</dt><dd class="ev">${esc(r.ev)}</dd>
|
||||
<dt>为什么</dt><dd>${esc(r.why)}</dd>
|
||||
<dt>现在怎么做</dt><dd>${esc(r.now)}</dd>
|
||||
</dl>
|
||||
</div>`).join('\n')}
|
||||
|
||||
<h2>agent 能测什么,不能测什么</h2>
|
||||
<table>
|
||||
<thead><tr><th>指标</th><th class="n">可用</th><th>依据</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>盲转 / 多余转 / 转错方向</td><td class="n yes">可用</td><td>复测 r ≈ 0.82–0.92;驱动了真实修复(1.17 / 1.22 / 0.44 → 0)</td></tr>
|
||||
<tr><td>是否通关、空点次数</td><td class="n yes">可用</td><td>直接观察,无主观成分</td></tr>
|
||||
<tr><td>自由文本抱怨</td><td class="n yes">可用</td><td>33 条里两个主题高度重复,指标抓不到;但须对照基线(规则 5)</td></tr>
|
||||
<tr><td>难度评分</td><td class="n no">只能看总体</td><td>稳定但不区分;单关排序不可用,整批读数(偏易一档)可用</td></tr>
|
||||
<tr><td>无解感评分</td><td class="n">谨慎</td><td>复测 r = 0.85,与无效动作相关 0.66,可当挫败感的代理</td></tr>
|
||||
<tr><td>好玩 / 清晰度 / 旋转评分</td><td class="n no">不可用</td><td>复测 r ≤ 0.56,是模型噪声</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>要做 1000 关的话</h2>
|
||||
<ol class="recipe">
|
||||
<li><b>先把判据写成可执行的闸门</b>,再开始生成。这一轮的闸门链是:长成 → 首转双向 → 锁住 ≤90% → 无不可解态 → 终局朝向 ≥2 → 每转释放 ≥2 → 反差 ≥12 → 开局可点 ≥3 → 开局演出 ≥0.9s → 长巡游 ≥3。每一级剩多少必须打印出来。</li>
|
||||
<li><b>用规则机器人做全量验收</b>(12 秒 / 全池),不用 agent。机械旋转指标已证明测的是关卡不是玩家。</li>
|
||||
<li><b>agent 只抽样跑</b>,而且只看两样东西:机械缺陷指标,和自由文本抱怨。不要为难度标定花 agent 时间——那条路这一轮已经证明走不通。</li>
|
||||
<li><b>难度曲线靠机械假设铺,靠人眼裁定。</b>用 <code>select-50.mjs</code> 按特征打分并均匀铺开,家族上限防止一套模板占满(上一版 1000 关实际只有 30 套模板)。</li>
|
||||
<li><b>每一轮只动一个变量</b>,并且必须能在漏斗里看到它挡掉了什么,否则撤回。</li>
|
||||
<li><b>人工审核要准备好材料</b>:开局盘面从求解器取(不是示意图)、标明哪些数字可信、每关一个试玩直达链接。</li>
|
||||
</ol>
|
||||
|
||||
<h2>被实测否掉的做法</h2>
|
||||
<table>
|
||||
<thead><tr><th>试过什么</th><th>结果</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>拟合 difficulty ~ 机械特征,用来重选关卡</td><td>响应几乎无方差(${R1.flat[1]}/${R1.rated} 同分),回归的是常数</td></tr>
|
||||
<tr><td>legibility(一眼看穿的成本)解释难度评分</td><td>相关 -0.02,无效;降级为纯测量</td></tr>
|
||||
<tr><td>路线交叠度当难度轴</td><td>中位已达 89%,没有上升空间</td></tr>
|
||||
<tr><td>换开局朝向来提高判断成本</td><td>127 个候选里只有 7 个变好</td></tr>
|
||||
<tr><td>「插入端多朝向出口」过滤器</td><td>对漏斗零效果,已删除</td></tr>
|
||||
<tr><td>把锁住比例当难度主轴排序</td><td>方向反了:它奖励「开局什么都做不了」</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<footer>
|
||||
<p class="mono">生成:node scripts/build-process-report.mjs · 页面所有数字由脚本从 levels/campaign-50.json 与 bench/runs 直接测得</p>
|
||||
<p>相关:<a href="review-50.html">五十关人工审核页</a> · <a href="level-board.html">任务看板</a> · <a href="bench-reliability.html">bench 复测可信度</a></p>
|
||||
</footer>
|
||||
</div>
|
||||
</body>
|
||||
</html>`;
|
||||
|
||||
writeFileSync(resolve('docs/level-process.html'), html);
|
||||
console.log(`流程复盘已生成:docs/level-process.html(R1 ${R1.n} 关实测,${RULES.length} 条规则)`);
|
||||
Reference in New Issue
Block a user