feat(bench): batch runners, aggregation and a standalone kit

Four ways in, depending on what you have: a shell agent driving bench.mjs, a
direct-API loop for models with only an endpoint (plain text in, one JSON line
out, so weak and local models can play), a rule-following bot that sweeps 1000
levels in 12 seconds, and a folder you can hand to someone else.

bench-kit.mjs builds that folder: CLI, engine, level pack, observer page and
tile atlas, about 1.5 MB, needing only Node 20 — no repo, no install, no
network. The engine files are discovered by following imports so a new module
can't be left behind, every path rewrite is asserted so a rename fails the
build loudly, and the builder plays a real game inside the result before
declaring it done.

bench-report.mjs aggregates two ways: per level (is this level any good) and
per model (how does this agent play). bench-import.mjs brings runs played
elsewhere back in under a tag.

bench-pack.mjs is the earlier, smaller export with no observer page; superseded
by bench-kit but kept in history rather than dropped silently.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
cabinw
2026-09-14 14:00:33 +08:00
co-authored by Claude Opus 5
parent 47be0ad9c4
commit f3e91e3597
7 changed files with 699 additions and 0 deletions
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env node
// Export a self-contained bench any agent can run in its own sandbox: four JS
// files, a level pack and a protocol. No repo, no npm, no network.
// node scripts/bench-pack.mjs [--set=levels/prototypes/d5.json] [--levels=1-18|all|1,4] [--out=bench/pack] [--peek]
// Hand the folder (or a zip of it) to Grok / DeepSeek / any harness; they run
// `node play.mjs …` and send back runs/*.json, which bench-import.mjs reads.
import { cpSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { resolve } from 'node:path';
const flags = Object.fromEntries(process.argv.slice(2).map(a => { const [k, ...v] = a.replace(/^--/, '').split('='); return [k, v.length ? v.join('=') : true]; }));
const setPath = flags.set || 'levels/prototypes/d5.json';
const out = resolve(flags.out || 'bench/pack');
const raw = JSON.parse(readFileSync(resolve(setPath), 'utf8'));
const all = Array.isArray(raw) ? raw : Array.isArray(raw.levels) ? raw.levels : Object.values(raw).filter(l => l && l.cells);
const spec = String(flags.levels || 'all');
const picked = spec === 'all' ? all
: spec.split(',').flatMap(part => { const m = part.match(/^(\d+)-(\d+)$/); return m ? Array.from({ length: Number(m[2]) - Number(m[1]) + 1 }, (_, i) => Number(m[1]) + i) : [Number(part)]; })
.filter(n => n >= 1 && n <= all.length).map(n => all[n - 1]);
if (!picked.length) { console.error('没有选中任何关卡'); process.exit(1); }
// Rewrite the repo CLI into a standalone one. Every replacement is asserted, so
// a rename in bench.mjs breaks the pack loudly instead of shipping it broken.
let cli = readFileSync(resolve('scripts/bench.mjs'), 'utf8');
const swap = (from, to) => { if (!cli.includes(from)) { console.error(`bench-pack 过时了:scripts/bench.mjs 里找不到 ${JSON.stringify(from)}`); process.exit(1); } cli = cli.split(from).join(to); };
swap("from '../src/bench.js'", "from './bench.js'");
swap("resolve('bench/runs')", "resolve('runs')");
swap("flags.set || 'levels/prototypes/d5.json'", "flags.set || 'levels.json'");
swap('node scripts/bench.mjs', 'node play.mjs');
swap('labs/bench/AGENT.md', 'AGENT.md');
rmSync(out, { recursive: true, force: true });
mkdirSync(resolve(out, 'runs'), { recursive: true });
for (const f of ['path-math.js', 'core.js', 'bench.js']) cpSync(resolve('src', f), resolve(out, f));
writeFileSync(resolve(out, 'play.mjs'), cli);
writeFileSync(resolve(out, 'levels.json'), JSON.stringify({ version: 1, source: setPath, levels: picked }));
// The pack ships the same single file agents read in the repo, with the
// command name and paths rewritten for a standalone folder.
const guide = readFileSync(resolve('labs/bench/AGENT.md'), 'utf8')
.split('node scripts/bench.mjs').join('node play.mjs')
.split('labs/bench/AGENT.md').join('AGENT.md')
.split('`src/`').join('`core.js` / `bench.js`')
.split('--set=levels/v0.5.json').join('--set=levels.json');
writeFileSync(resolve(out, 'AGENT.md'), guide);
writeFileSync(resolve(out, 'README.md'), `# HEX TRAILS 关卡实验台 · 便携包
给任意 Agent 用的独立实验台:**只要 Node 20+,不需要联网、不需要 npm install、不需要原仓库。**
包含 ${picked.length} 关(来自 \`${setPath}\`)。
## 给 Agent 的任务提示(整段复制给它)
> 读这个文件夹里的 \`AGENT.md\`,然后按它说的玩第 1 关。你的模型标识写 \`<换成它的名字>\`。
\`AGENT.md\` 里已经包含规则、读盘方法、全部命令和示例,Agent 读完就能直接开始,不需要别的说明。
## 命令
| 命令 | 作用 |
|---|---|
| \`node play.mjs list\` | 列出本包里的关卡 |
| \`node play.mjs start --model=X --level=N --persona=… --no-rules\` | 开一局(局名自动生成)|
| \`node play.mjs observe --run=X\` | 再看一次盘面 |
| \`node play.mjs act --run=X tap t4, tap t5 --note="…"\` | 提交一个决定(可连做几步) |
| \`node play.mjs rate --run=X --json='{…}'\` | 评分(必填) |
| \`node play.mjs status --run=X\` / \`metrics --run=X\` | 回看步骤 / 指标 |
人设:\`reader\`(读盘型)\`tapper\`(试错型)\`turner\`(转盘型)\`novice\`(新手)。
## 交回结果
每局写在 \`runs/<局名>.json\`,含每一步的动作、理由、结果和时间戳。把整个 \`runs/\` 交回即可,原仓库用
\`node scripts/bench-import.mjs --from=<路径>/runs --tag=<模型名>\` 导入,然后在观察页回放、在汇总表里比较。
生成时间:${new Date().toISOString()}
`);
console.log(`便携包已生成:${out}`);
console.log(` ${picked.length} 关 · 文件:play.mjs bench.js core.js path-math.js levels.json AGENT.md README.md runs/`);
console.log(` 打包发给别的 agent:cd ${resolve(out, '..')} && zip -r bench-pack.zip ${out.split('/').pop()}`);
console.log(` 自测:cd ${out} && node play.mjs list && node play.mjs start --run=t --level=1 --model=me`);