feat(bench): batch runners, aggregation and a standalone kit

Four ways in, depending on what you have: a shell agent driving bench.mjs, a
direct-API loop for models with only an endpoint (plain text in, one JSON line
out, so weak and local models can play), a rule-following bot that sweeps 1000
levels in 12 seconds, and a folder you can hand to someone else.

bench-kit.mjs builds that folder: CLI, engine, level pack, observer page and
tile atlas, about 1.5 MB, needing only Node 20 — no repo, no install, no
network. The engine files are discovered by following imports so a new module
can't be left behind, every path rewrite is asserted so a rename fails the
build loudly, and the builder plays a real game inside the result before
declaring it done.

bench-report.mjs aggregates two ways: per level (is this level any good) and
per model (how does this agent play). bench-import.mjs brings runs played
elsewhere back in under a tag.

bench-pack.mjs is the earlier, smaller export with no observer page; superseded
by bench-kit but kept in history rather than dropped silently.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
cabinw
2026-09-14 14:00:33 +08:00
co-authored by Claude Opus 5
parent 47be0ad9c4
commit f3e91e3597
7 changed files with 699 additions and 0 deletions
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env node
// Rebuild bench/runs/index.json from the run files. Parallel agents must not
// read-modify-write a shared index; they write their own run file and the
// index is regenerated here (the batch runner calls this periodically).
import { existsSync, readdirSync, readFileSync, writeFileSync } from 'node:fs';
import { resolve } from 'node:path';
export function rebuildIndex(dir = resolve('bench/runs')) {
if (!existsSync(dir)) return [];
const index = [];
for (const f of readdirSync(dir)) {
if (!f.endsWith('.json') || f === 'index.json') continue;
try {
const r = JSON.parse(readFileSync(resolve(dir, f), 'utf8'));
index.push({ run: r.run, level: r.level.id, persona: r.persona, model: r.model, peek: r.peek, status: r.status, steps: r.steps.length, updatedAt: r.updatedAt });
} catch { /* a file being written right now; next rebuild picks it up */ }
}
index.sort((a, b) => (b.updatedAt || '').localeCompare(a.updatedAt || ''));
writeFileSync(resolve(dir, 'index.json'), JSON.stringify(index, null, 1));
return index;
}
if (process.argv[1] && process.argv[1].endsWith('bench-index.mjs')) {
const index = rebuildIndex();
console.log(`index.json · ${index.length} 局`);
}