feat(bench): agent bench session core and CLI
Let any agent with a shell play a level the way a person does: look at the board, decide what to tap or which way to turn, read the feedback, decide again. Every step carries the agent's own reason, so a human can watch and judge the level from how it was played. observe() returns what the eye sees — each piece's drawn route, which cells it crosses, who is sitting on it, and who it brushes past — never "which pieces can leave". act() goes through the real Game rules; metrics() scores the run afterwards with the solver. Pieces are wide, so a route with nobody standing on it can still be blocked by a neighbour the curve hugs. That is invisible in a cell listing, so every obstacle is reported, on-route and grazing alike: reporting only the leading one hid the rest, and in 113 of 581 such cases clearing the blocker sprang a collision the agent had never been shown. AGENT.md is the single file an agent reads — rules, board reading, commands, rating scale, worked example. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
cabinw
@@ -0,0 +1,165 @@
|
|||||||
|
# 给 Agent：来玩一关 HEX TRAILS
|
||||||
|
|
||||||
|
**你现在是一名玩家，不是程序员。** 读完这个文件就直接开始，不需要再读任何别的文档。
|
||||||
|
|
||||||
|
你要像人一样玩一关六角轨道解谜：看盘、决定点哪枚或转哪边、看反馈、再决定。有人会在网页上实时看着你操作，所以每一步都要写清楚你为什么这么做。
|
||||||
|
|
||||||
|
**禁止**：读 `src/` 下的任何文件、读 `levels.json`、读 `runs/` 里的存档、调用求解器、写脚本或循环批量执行、修改任何文件。
|
||||||
|
你和棋盘之间只有 `node scripts/bench.mjs` 这一个命令——你知道的必须全部来自它打印给你的东西。绕过它就等于作弊，这一局的数据也就废了。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 第一步：开一局
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node scripts/bench.mjs start --model=<你的模型> --agent="<你的运行环境 · 模式>" --level=1 --persona=tapper --no-rules
|
||||||
|
```
|
||||||
|
|
||||||
|
**请如实报告你自己是谁**——观察页会把这两项显示在棋盘正上方，人类靠它知道现在是谁在玩：
|
||||||
|
|
||||||
|
- `--model`：你的模型标识符，例如 `claude-opus-5`、`grok-4.6`、`deepseek-flash`、`gpt-5`。不确定就写最接近的。
|
||||||
|
- `--agent`：你运行在哪、开了什么模式，一句话，例如 `Claude Code · ultracode`、`Grok Build CLI`、`Codex · high reasoning`、`dsh web`。包含任何你认为影响自己表现的配置（推理强度、是否联网、是否有思考预算）。不知道就写 harness 名字。
|
||||||
|
- `--level`：关号，不指定就是第 1 关。`node scripts/bench.mjs list` 可以看有哪些关。
|
||||||
|
- `--persona`：`tapper`（试错型，默认）/ `reader`（读盘型）/ `turner`（转盘型）/ `novice`（新手）。按这个风格玩。
|
||||||
|
|
||||||
|
命令会打印两样东西，都要留意：
|
||||||
|
|
||||||
|
1. **局名**（形如 `grok-4.6-L1-tapper`）——后面每一条命令都要带 `--run=<那个局名>`，原样抄，别自己改。
|
||||||
|
2. **观察页地址**——把那一行原样转告人类，他们能实时看你玩。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 规则（30 秒）
|
||||||
|
|
||||||
|
- 棋盘上画着轨道。**点一枚棋子，它沿自己箭头所指的那条轨道一路走，直到走出棋盘边缘**——这就是离场。
|
||||||
|
- 路上撞到别的棋子就退回原位，什么都不扣。棋子有宽度，**擦到紧挨着的棋子也算撞**。
|
||||||
|
- **旋转底盘**：整块棋盘连轨道转 60°，棋子的位置和箭头不动——所以每枚棋子接上的是另一条轨道，所有路线全变。左右转免费、可撤销，转 6 次回到原样。
|
||||||
|
- 目标：让所有棋子离场。**没有失败**——移走一枚棋子只会让别的路更通，不会把谁困住，所以不存在"点错一步就死"。有步数预算（默认 80 步）。
|
||||||
|
|
||||||
|
## 你会看到什么
|
||||||
|
|
||||||
|
最重要的是这一段——每枚棋子当前的路线，就是人眼在屏幕上看到的那条线：
|
||||||
|
|
||||||
|
```
|
||||||
|
每枚棋子当前的路线（箭头出发 → 依次经过的格 → 出界/闭环；[tX] = 那格上压着棋子 tX）：
|
||||||
|
t5 @ -2,4 ↙：就在边上，直接出界（0 格）· 路上没有棋子
|
||||||
|
t0 @ 4,-4 ↖：3,-4 → 2,-4[t9] → 1,-3 → 0,-3 → 出界（4 格 · 1 弯）· 路上有 t9
|
||||||
|
t2 @ 0,-2 ↙：-1,-1 → -2,0 → -3,1 → -4,2 → 出界（4 格 · 0 弯）· 路上没有棋子，但紧贴 t7（会擦到）
|
||||||
|
t1 @ 3,-1 ↓：3,0[t6] → 2,1 → 1,2 → 出界（3 格 · 1 弯）· 路上有 t6，另外紧贴 t4（会擦到）
|
||||||
|
t3 @ 2,2 ↗：3,1 → 3,0 → 2,1 → 2,2 → 闭环（4 格 · 2 弯）· 路上没有棋子
|
||||||
|
```
|
||||||
|
|
||||||
|
| 你看到 | 意味着 |
|
||||||
|
|---|---|
|
||||||
|
| `→ 出界 · 路上没有棋子` | **点它就能走** |
|
||||||
|
| `2,-4[t9] … 路上有 t9` | 被 t9 压着，**先让 t9 走** |
|
||||||
|
| `但紧贴 t7（会擦到）` | 棋子有宽度，线从 t7 身边擦过去，**点了会被挡**；同样先让 t7 走 |
|
||||||
|
| `路上有 t6，另外紧贴 t4` | 两个都挡着，**t6 和 t4 都清掉**这枚才能走 |
|
||||||
|
| `→ 闭环` | 这条轨道绕回来了，当前朝向出不去，**只有旋转能改变它** |
|
||||||
|
|
||||||
|
坐标 `q,r` 只是格子的名字，不需要做坐标运算。
|
||||||
|
|
||||||
|
**这一行把挡住它的棋子全部列全了**——路上压着的和身边擦到的都在里面。要让这枚走，得把列出来的全部清掉；反过来，只要这一行说「路上没有棋子」且没有「紧贴」，点它就一定能走。
|
||||||
|
|
||||||
|
## 怎么玩得好
|
||||||
|
|
||||||
|
1. 先点**路上没有棋子、不紧贴谁、能出界**的。
|
||||||
|
2. 有 `[tX]` 或 `紧贴 tX` 的，先解决 tX；tX 也被挡就顺着往下找，先解最外层。
|
||||||
|
3. **一枚都走不了**（每条路都被压着/擦着、或都是闭环）→ 旋转。转完所有路线都变，**必须重新看盘**。
|
||||||
|
4. **转一次没打开出口，不代表这一关无解。** 有些关要转 2–3 次、甚至换个方向转，出口才出现。转到没见过的朝向去，别在同一个朝向反复点。
|
||||||
|
5. 每一步都算预算。有把握再点，不确定就先看清楚。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 第二步：反复出招
|
||||||
|
|
||||||
|
**像人一样连点**：看清一次盘面，就把有把握的几步用逗号一起提交，不要每点一枚重新读一遍。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node scripts/bench.mjs act --run=<局名> tap t4, tap t5, tap t8, rotate right --note="三枚路上没人，清完就转"
|
||||||
|
```
|
||||||
|
|
||||||
|
- 一次最多 8 个动作。链会在**第一个没成功的动作**和**任何一次旋转**之后自动停下，并打印新盘面——所以不会闭眼连点到错的地方。
|
||||||
|
- 动作：`tap tX` · `rotate right` / `rotate left` · `undo` · `giveup 原因`
|
||||||
|
- `--note` 必填，一句话说明你为什么这么做。这是人类唯一能看到的你的想法。
|
||||||
|
- 只想再看一次盘面：`node scripts/bench.mjs observe --run=<局名>`
|
||||||
|
|
||||||
|
**如果你觉得这一关无解，不要硬撑**，用 `act --run=<局名> giveup 我认为无解，因为……` 说明你卡在哪、试过什么。**诚实地放弃比勉强通关更有价值**——我们正是要找出让人觉得无解的关卡。预算用完局也会自动结束，同样要评分。
|
||||||
|
|
||||||
|
### 命令报错了怎么办
|
||||||
|
|
||||||
|
| 报错 | 原因 | 怎么办 |
|
||||||
|
|---|---|---|
|
||||||
|
| `没有这局：X（先 start）` | `--run` 写错了 | 用 start 打印的那个局名，原样抄 |
|
||||||
|
| `无效：没有棋子 tX` | 棋子 id 记错，或它已经离场 | 先 `observe` 重新看一遍当前有哪些棋子 |
|
||||||
|
| `act 需要：tap ID \| rotate …` | 动作写法不对 | 只有 `tap tX` / `rotate left\|right` / `undo` / `giveup 理由` 四种 |
|
||||||
|
| `X 已存在` | 局名重复 | 不要自己指定 `--run`，让 start 自动生成 |
|
||||||
|
|
||||||
|
报错不扣步数，改对了重发就行。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 第三步：评分（必做）
|
||||||
|
|
||||||
|
清盘、放弃或预算用完之后：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node scripts/bench.mjs rate --run=<局名> --json='{"clarity":4,"rotation":3,"stuck":2,"fun":3,"difficulty":3,"worst":"第 6 步点 t2 被 t7 擦碰挡住，路线上看不出来"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
| 键 | 问题 | 1 | 3 | 5 |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `clarity` | 路线看得清吗 | 完全看不清 | 大致能读 | 一眼看清 |
|
||||||
|
| `rotation` | 旋转有意义吗 | 随便转都行/没用 | 有点用 | 每次转都是关键决定 |
|
||||||
|
| `stuck` | 有没有觉得无解的时刻 | 从未 | 卡过一会儿 | 长时间以为无解 |
|
||||||
|
| `fun` | 好玩吗 | 无聊 | 还行 | 很好玩 |
|
||||||
|
| `difficulty` | 难度是否合适 | **太简单** | **刚刚好** | **太难** |
|
||||||
|
| `worst` | 最困惑的一步是第几步、为什么 | 一句话 | | |
|
||||||
|
|
||||||
|
注意 `difficulty` 是两头都不好：3 分是最佳，1 和 5 都是问题。
|
||||||
|
|
||||||
|
**请诚实评分**，尤其 `rotation` 和 `stuck`——我们正是要用这两项判断关卡好不好，一律给高分帮不上忙。
|
||||||
|
|
||||||
|
`rate` 会打印本局统计和回放链接。最后用两三句话向人类汇报：局名、状态（cleared / gave-up / budget）、步数、决策次数、最困惑的一步。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 完整示例
|
||||||
|
|
||||||
|
```
|
||||||
|
$ node scripts/bench.mjs start --model=grok-4.6 --agent="Grok Build CLI" --level=1 --persona=tapper --no-rules
|
||||||
|
开始 grok-4.6-L1-tapper · 关卡 proto-d5-switchback-r4-s6 · 人设 tapper · 模型 grok-4.6 · Grok Build CLI · 预算 80 步
|
||||||
|
观察页（人类实时观看）：http://localhost:4173/labs/bench/index.html?run=grok-4.6-L1-tapper&live=1
|
||||||
|
后续命令都用 --run=grok-4.6-L1-tapper
|
||||||
|
…盘面…
|
||||||
|
|
||||||
|
$ node scripts/bench.mjs act --run=grok-4.6-L1-tapper tap t4, tap t5, tap t8 --note="三枚路上没人也不紧贴谁"
|
||||||
|
点击 t4 → 离场（走了 1 格、0 个弯）
|
||||||
|
点击 t5 → 离场（走了 1 格、0 个弯）
|
||||||
|
点击 t8 → 离场（走了 3 格、1 个弯）
|
||||||
|
…新盘面…
|
||||||
|
|
||||||
|
$ node scripts/bench.mjs act --run=grok-4.6-L1-tapper rotate right --note="剩下的互相压着或闭环，一枚都走不了"
|
||||||
|
右转 → 底盘转到朝向 1，4 枚棋子的出口打开了
|
||||||
|
（旋转后盘面已变，后面的动作没有执行）
|
||||||
|
…新盘面…
|
||||||
|
|
||||||
|
$ node scripts/bench.mjs rate --run=grok-4.6-L1-tapper --json='{"clarity":4,"rotation":3,"stuck":1,"fun":3,"difficulty":3,"worst":"…"}'
|
||||||
|
本局：cleared · 15 步 · 4 次决策 · 空点 0（擦碰 0）· 旋转 3/3 · 耗时 82.3 s
|
||||||
|
回放：http://localhost:4173/labs/bench/index.html?run=grok-4.6-L1-tapper
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 其它命令（一般用不到）
|
||||||
|
|
||||||
|
| 命令 | 作用 |
|
||||||
|
|---|---|
|
||||||
|
| `node scripts/bench.mjs list` | 列出本组关卡 |
|
||||||
|
| `node scripts/bench.mjs status --run=X` | 回看自己走过的每一步 |
|
||||||
|
| `node scripts/bench.mjs metrics --run=X` | 本局指标 |
|
||||||
|
| `node scripts/bench.mjs open --run=X` | 起服务并打开观察页 |
|
||||||
|
| `--format=full` | 观察时额外打印每格的端口配对表（读长路线时才需要） |
|
||||||
|
| `--set=<关卡文件>` | 换一组关卡；`start` 和 `list` 都支持 |
|
||||||
|
|
||||||
|
**要玩多关就重复第一步到第三步**，每关一个新局，`--level` 换成下一关。不要写循环一次跑完——一次一关，一次一个决定。
|
||||||
@@ -0,0 +1,173 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
// Agent bench CLI. One command per step so any agent with a shell can play.
|
||||||
|
// Agents are given labs/bench/AGENT.md and need nothing else.
|
||||||
|
// node scripts/bench.mjs list [--set=levels/prototypes/d5.json]
|
||||||
|
// node scripts/bench.mjs start --model=NAME [--agent='运行环境 · 模式'] [--level=N] [--run=NAME] [--persona=…] [--set=…] [--peek] [--budget=80] [--no-rules]
|
||||||
|
// node scripts/bench.mjs observe --run=NAME [--format=json|full]
|
||||||
|
// node scripts/bench.mjs act --run=NAME tap t3, tap t5, rotate right [--note=…] [--quiet]
|
||||||
|
// node scripts/bench.mjs rate --run=NAME --json='{"clarity":3,…}'
|
||||||
|
// node scripts/bench.mjs status|metrics --run=NAME [--json]
|
||||||
|
// node scripts/bench.mjs open [--run=NAME] # 起服务、打印/打开观察页
|
||||||
|
// node scripts/bench.mjs rules [--peek]
|
||||||
|
// Runs live in bench/runs/NAME.json; the observer page polls that file.
|
||||||
|
import { spawn, spawnSync } from 'node:child_process';
|
||||||
|
import { existsSync, mkdirSync, readFileSync, writeFileSync, readdirSync } from 'node:fs';
|
||||||
|
import { createConnection } from 'node:net';
|
||||||
|
import { resolve } from 'node:path';
|
||||||
|
import { createRun, observe, actMany, rate, metrics, rulesText, RATING_QUESTIONS, describeAction, describeResult } from '../src/bench.js';
|
||||||
|
|
||||||
|
const RUNS = resolve('bench/runs');
|
||||||
|
const argv = process.argv.slice(2);
|
||||||
|
const command = argv[0];
|
||||||
|
const flags = {}, positional = [];
|
||||||
|
for (const a of argv.slice(1)) {
|
||||||
|
if (a.startsWith('--')) { const [k, ...v] = a.slice(2).split('='); flags[k] = v.length ? v.join('=') : true; }
|
||||||
|
else positional.push(a);
|
||||||
|
}
|
||||||
|
const fail = message => { console.error(message); process.exit(1); };
|
||||||
|
// text = routes only (default, what a person sees); full = also the per-cell rail table; json = machine.
|
||||||
|
const fmt = () => flags.format === 'json' ? 'json' : flags.format === 'full' || flags.full ? 'full' : 'text';
|
||||||
|
const loadSet = path => {
|
||||||
|
const raw = JSON.parse(readFileSync(resolve(path), 'utf8'));
|
||||||
|
return Array.isArray(raw) ? raw : Array.isArray(raw.levels) ? raw.levels : Object.values(raw).filter(l => l && l.cells);
|
||||||
|
};
|
||||||
|
const runPath = name => resolve(RUNS, `${name}.json`);
|
||||||
|
const PORT = Number(process.env.PORT || 4173);
|
||||||
|
const slug = s => String(s).replace(/[^\w.-]+/g, '-').replace(/^-+|-+$/g, '') || 'agent';
|
||||||
|
// Name the run so the agent never has to invent one: model-L3-tapper, then -2, -3…
|
||||||
|
function autoRunName(model, level, persona) {
|
||||||
|
const base = `${slug(model)}-L${level}-${slug(persona)}`;
|
||||||
|
if (!existsSync(runPath(base))) return base;
|
||||||
|
for (let n = 2; n < 99; n++) if (!existsSync(runPath(`${base}-${n}`))) return `${base}-${n}`;
|
||||||
|
return `${base}-${Date.now()}`;
|
||||||
|
}
|
||||||
|
const portOpen = () => new Promise(done => {
|
||||||
|
const socket = createConnection({ port: PORT, host: '127.0.0.1' });
|
||||||
|
const finish = ok => { socket.destroy(); done(ok); };
|
||||||
|
socket.setTimeout(400);
|
||||||
|
socket.on('connect', () => finish(true));
|
||||||
|
socket.on('error', () => finish(false));
|
||||||
|
socket.on('timeout', () => finish(false));
|
||||||
|
});
|
||||||
|
// Bring the observer page within reach: start the static server if it is not
|
||||||
|
// already up, then print (and on macOS open) the URL for this run. Returns
|
||||||
|
// null in a portable pack, which ships no page and no server — a URL there
|
||||||
|
// would point at whatever else happens to hold the port.
|
||||||
|
const OBSERVER = resolve('labs/bench/index.html'), SERVER = resolve('scripts/serve.mjs');
|
||||||
|
async function observerUrl(run, { open = false } = {}) {
|
||||||
|
if (!existsSync(OBSERVER) || !existsSync(SERVER)) return null;
|
||||||
|
const url = `http://localhost:${PORT}/labs/bench/index.html${run ? `?run=${encodeURIComponent(run)}&live=1` : ''}`;
|
||||||
|
if (!(await portOpen())) {
|
||||||
|
const server = spawn(process.execPath, [SERVER], { detached: true, stdio: 'ignore' });
|
||||||
|
server.unref();
|
||||||
|
for (let i = 0; i < 20 && !(await portOpen()); i++) await new Promise(r => setTimeout(r, 150));
|
||||||
|
}
|
||||||
|
if (open && process.platform === 'darwin') spawnSync('open', [url], { stdio: 'ignore' });
|
||||||
|
return url;
|
||||||
|
}
|
||||||
|
const loadRun = () => {
|
||||||
|
if (!flags.run) fail('需要 --run=NAME');
|
||||||
|
if (!existsSync(runPath(flags.run))) fail(`没有这局：${flags.run}（先 start）`);
|
||||||
|
return JSON.parse(readFileSync(runPath(flags.run), 'utf8'));
|
||||||
|
};
|
||||||
|
const saveRun = runObj => {
|
||||||
|
mkdirSync(RUNS, { recursive: true });
|
||||||
|
runObj.updatedAt = new Date().toISOString();
|
||||||
|
writeFileSync(runPath(runObj.run), JSON.stringify(runObj));
|
||||||
|
const indexPath = resolve(RUNS, 'index.json');
|
||||||
|
const index = existsSync(indexPath) ? JSON.parse(readFileSync(indexPath, 'utf8')) : [];
|
||||||
|
const entry = { run: runObj.run, level: runObj.level.id, persona: runObj.persona, model: runObj.model, peek: runObj.peek, status: runObj.status, steps: runObj.steps.length, updatedAt: runObj.updatedAt };
|
||||||
|
const at = index.findIndex(e => e.run === runObj.run);
|
||||||
|
if (at >= 0) index[at] = entry; else index.push(entry);
|
||||||
|
writeFileSync(indexPath, JSON.stringify(index, null, 1));
|
||||||
|
};
|
||||||
|
|
||||||
|
if (command === 'list') {
|
||||||
|
const set = flags.set || 'levels/prototypes/d5.json';
|
||||||
|
const levels = loadSet(set);
|
||||||
|
console.log(`${set} · ${levels.length} 关`);
|
||||||
|
levels.forEach((l, i) => console.log(` ${String(i + 1).padStart(3)} ${l.id} ${l.cells.length} 格 · ${l.tiles.length} 枚${l.prototype ? ` · 锁住 ${Math.round(l.prototype.stats.lockedShare * 100)}% · 最少 ${l.prototype.stats.rotations} 转` : ''}`));
|
||||||
|
} else if (command === 'rules') {
|
||||||
|
console.log(rulesText(!!flags.peek));
|
||||||
|
} else if (command === 'start') {
|
||||||
|
const levels = loadSet(flags.set || 'levels/prototypes/d5.json');
|
||||||
|
const wanted = flags.level ?? 1;
|
||||||
|
const level = levels.find(l => l.id === wanted) || levels[Number(wanted) - 1];
|
||||||
|
if (!level) fail(`找不到关卡 ${wanted}（本组共 ${levels.length} 关，用 list 查看）`);
|
||||||
|
const persona = flags.persona || 'tapper';
|
||||||
|
const model = flags.model || process.env.BENCH_MODEL || 'unknown';
|
||||||
|
if (model === 'unknown') console.warn('提示：没有 --model，本局记为 unknown；填上你的模型标识才能按模型比较。');
|
||||||
|
const name = flags.run ? String(flags.run) : autoRunName(model, levels.indexOf(level) + 1, persona);
|
||||||
|
if (!/^[\w.-]+$/.test(name)) fail('run 名只能用字母、数字、_ - .');
|
||||||
|
if (existsSync(runPath(name)) && !flags.force) fail(`${name} 已存在；换个名字或加 --force`);
|
||||||
|
const agent = flags.agent ? String(flags.agent) : '';
|
||||||
|
const runObj = createRun({ level, run: name, persona, model, agent, peek: !!flags.peek, budget: Number(flags.budget || 80), startedAt: new Date().toISOString() });
|
||||||
|
saveRun(runObj);
|
||||||
|
const url = await observerUrl(name, { open: !!flags.open });
|
||||||
|
console.log(`开始 ${runObj.run} · 关卡 ${level.id} · 人设 ${runObj.persona} · 模型 ${runObj.model}${runObj.agent ? ` · ${runObj.agent}` : ''} · 预算 ${runObj.budget} 步 · peek ${runObj.peek ? '开' : '关'}`);
|
||||||
|
if (!runObj.agent) console.warn('提示：没有 --agent，观察页横幅只会显示模型名。填上运行环境和模式，例如 --agent="Claude Code · ultracode"。');
|
||||||
|
if (url) console.log(`观察页（人类实时观看）：${url}`);
|
||||||
|
console.log(`后续命令都用 --run=${runObj.run}`);
|
||||||
|
console.log('');
|
||||||
|
// --no-rules: an agent that already read AGENT.md skips the rules block.
|
||||||
|
if (!flags['no-rules']) { console.log(rulesText(runObj.peek)); console.log(''); }
|
||||||
|
console.log(observe(runObj, { format: fmt() }));
|
||||||
|
} else if (command === 'open') {
|
||||||
|
const url = await observerUrl(flags.run ? String(flags.run) : null, { open: flags.open !== 'false' });
|
||||||
|
console.log(url || '这个副本没有观察页（便携包）；把 runs/ 交回原仓库用 bench-import.mjs 导入后再看。');
|
||||||
|
} else if (command === 'observe') {
|
||||||
|
const runObj = loadRun();
|
||||||
|
const view = observe(runObj, { format: fmt() });
|
||||||
|
console.log(typeof view === 'string' ? view : JSON.stringify(view, null, 1));
|
||||||
|
} else if (command === 'act') {
|
||||||
|
const runObj = loadRun();
|
||||||
|
// One action, or several separated by commas: `tap t4, tap t5, rotate right`.
|
||||||
|
// A chain stops at the first tap that fails and after any turn.
|
||||||
|
const parse = words => {
|
||||||
|
const [kind, ...rest] = words;
|
||||||
|
if (kind === 'tap' || kind === 'peek') return { type: kind, id: rest[0] };
|
||||||
|
if (kind === 'rotate') return { type: 'rotate', direction: ['right', 'r', '1', '+1', '右'].includes(rest[0]) ? 1 : ['left', 'l', '-1', '左'].includes(rest[0]) ? -1 : 0 };
|
||||||
|
if (kind === 'undo') return { type: 'undo' };
|
||||||
|
if (kind === 'giveup') return { type: 'giveup', reason: rest.join(' ') };
|
||||||
|
fail('act 需要：tap ID | rotate left|right | undo | peek ID | giveup 原因；多个动作用逗号隔开');
|
||||||
|
};
|
||||||
|
const groups = positional.join(' ').split(',').map(s => s.trim()).filter(Boolean).map(s => parse(s.split(/\s+/)));
|
||||||
|
if (!groups.length) fail('act 需要一个动作');
|
||||||
|
if (groups.length > 8) fail('一次最多连做 8 个动作');
|
||||||
|
const { outcomes, stopped } = actMany(runObj, groups, { note: flags.note ? String(flags.note) : '' });
|
||||||
|
saveRun(runObj);
|
||||||
|
for (const out of outcomes) console.log(`${describeAction(out.step?.action || out.rejected?.action)} → ${out.feedback}`);
|
||||||
|
if (stopped) console.log(`（${stopped}）`);
|
||||||
|
if (!flags.quiet && runObj.status === 'playing') { console.log(''); console.log(observe(runObj, { format: fmt() === 'json' ? 'text' : fmt() })); }
|
||||||
|
if (runObj.status !== 'playing') console.log(`\n本局结束：${runObj.status}。请用 rate 提交评分。`);
|
||||||
|
if (flags.json) console.log(JSON.stringify({ outcomes, stopped }));
|
||||||
|
} else if (command === 'rate') {
|
||||||
|
const runObj = loadRun();
|
||||||
|
let answers;
|
||||||
|
try { answers = JSON.parse(flags.json || '{}'); } catch { fail('--json 需要合法 JSON'); }
|
||||||
|
const rating = rate(runObj, answers);
|
||||||
|
saveRun(runObj);
|
||||||
|
console.log('已记录评分：' + JSON.stringify(rating));
|
||||||
|
const missing = RATING_QUESTIONS.filter(q => rating[q.key] === null).map(q => q.key);
|
||||||
|
if (missing.length) console.log('未填：' + missing.join(', '));
|
||||||
|
// Close the loop for the agent: the numbers it should report, and the link.
|
||||||
|
const m = metrics(runObj);
|
||||||
|
const secs = ms => ms == null ? '—' : ms >= 1000 ? `${(ms / 1000).toFixed(1)} s` : `${ms} ms`;
|
||||||
|
console.log(`本局：${m.status} · ${m.steps} 步 · ${m.decisions} 次决策 · 空点 ${m.deadTaps}（擦碰 ${m.grazeDeadTaps}）· 旋转 ${m.rotations}/${m.minimumRotations ?? '?'} · 耗时 ${secs(m.timing.totalMs)}`);
|
||||||
|
const replayUrl = await observerUrl(runObj.run);
|
||||||
|
if (replayUrl) console.log(`回放：${replayUrl}`);
|
||||||
|
} else if (command === 'status') {
|
||||||
|
const runObj = loadRun();
|
||||||
|
console.log(`${runObj.run} · ${runObj.level.id} · ${runObj.status} · ${runObj.steps.length}/${runObj.budget} 步 · 人设 ${runObj.persona} · 模型 ${runObj.model}`);
|
||||||
|
for (const s of runObj.steps) console.log(` ${String(s.n).padStart(3)} ${describeAction(s.action).padEnd(10)} → ${describeResult(s.result)}${s.note ? ` 「${s.note}」` : ''}`);
|
||||||
|
} else if (command === 'metrics') {
|
||||||
|
const runObj = loadRun();
|
||||||
|
const m = metrics(runObj);
|
||||||
|
if (flags.json) console.log(JSON.stringify(m, null, 1));
|
||||||
|
else console.log(Object.entries(m).filter(([k]) => k !== 'rating').map(([k, v]) => `${k}: ${typeof v === 'number' ? Number(v.toFixed(2)) : v}`).join('\n') + (m.rating ? '\nrating: ' + JSON.stringify(m.rating) : ''));
|
||||||
|
} else if (command === 'runs') {
|
||||||
|
const dir = existsSync(RUNS) ? readdirSync(RUNS).filter(f => f.endsWith('.json') && f !== 'index.json') : [];
|
||||||
|
for (const f of dir) { const r = JSON.parse(readFileSync(resolve(RUNS, f), 'utf8')); console.log(`${r.run.padEnd(28)} ${r.level.id.padEnd(34)} ${r.status.padEnd(9)} ${String(r.steps.length).padStart(3)} 步 ${r.persona}/${r.model}`); }
|
||||||
|
} else {
|
||||||
|
fail('用法：node scripts/bench.mjs list|start|act|observe|rate|status|metrics|runs|open|rules …\n完整说明（给 Agent 读的那一份）：labs/bench/AGENT.md');
|
||||||
|
}
|
||||||
+292
@@ -0,0 +1,292 @@
|
|||||||
|
import { DIRS, Game, available, clone, opposite, rotateBoard, solveRotations, trace } from './core.js';
|
||||||
|
|
||||||
|
// Agent bench: a level session that any agent can drive through observe/act.
|
||||||
|
// The observation carries only what a person can read off the screen — cells,
|
||||||
|
// rails, pieces, arrows, orientation. Never which pieces can currently leave,
|
||||||
|
// and never the solver. Metrics use the solver afterwards, on the recorded run.
|
||||||
|
|
||||||
|
export const PORT_NAMES = ['↑', '↗', '↘', '↓', '↙', '↖'];
|
||||||
|
export const ACTIONS = ['tap', 'rotate', 'undo', 'peek', 'giveup'];
|
||||||
|
const countBends = route => route.steps.filter(s => s.enter !== null && opposite(s.enter) !== s.exit).length;
|
||||||
|
|
||||||
|
export function rulesText(peek) {
|
||||||
|
return [
|
||||||
|
'规则：棋盘由六角格组成，每格画着轨道。点击一枚棋子，它沿着自己箭头所指的那条轨道一路走，直到走出棋盘边缘（离场）。途中如果撞上另一枚棋子，就退回原位（棋子有宽度，擦到紧邻的棋子也算撞）。被挡不扣任何东西。',
|
||||||
|
'旋转底盘：整块棋盘连轨道转 60°，棋子的位置和箭头不动——于是每枚棋子接上的是另一条轨道，路线全部改变。左转右转免费、可反复、可撤销；转 6 次回到原样。',
|
||||||
|
'目标：让所有棋子离场。没有失败；有步数预算。',
|
||||||
|
'',
|
||||||
|
'怎么看盘面：观察里给出每枚棋子当前的路线——它会依次经过哪些格、在哪里出界（或绕成闭环）。路线上压着别的棋子，标成 [tX]；路线紧贴着别的棋子（棋子有宽度，会擦到），标成"紧贴 tX"。这就是人眼在屏幕上看到的那条线。',
|
||||||
|
'怎么玩：',
|
||||||
|
'1) 路上没有棋子、不紧贴谁、而且能出界的，点它就能离场。先点这些。',
|
||||||
|
'2) 路上有 [tX] 或紧贴 tX 的，得先让 tX 走：tX 自己能走就先点 tX；tX 也被挡就顺着往下找，先解开最外层的。',
|
||||||
|
'3) 每条路线都有人挡、或都是闭环——一枚都走不了，那就旋转。旋转后所有路线都变，重新看。转完没有出口，可以再转或转回去。',
|
||||||
|
'4) 每一步都算预算：先做有把握的，别乱试。' + (peek ? ' peek 与观察里的路线是同一信息，一般用不着。' : ''),
|
||||||
|
].join('\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Routes are shown in full: a folded "…(共 54 格)" hides exactly the part a
|
||||||
|
// careful player would re-check. Callers that must save tokens pass routeLimit.
|
||||||
|
const ROUTE_SHOW = Infinity;
|
||||||
|
// The drawn line, as a person sees it: the cells it passes, who is sitting on
|
||||||
|
// it, and — because pieces are wide — who it brushes past. The brush is taken
|
||||||
|
// from the real swept collider, since that is what the eye judges from the
|
||||||
|
// curve hugging a neighbour; a cell-adjacency guess would over-warn.
|
||||||
|
function pieceRoutes(game) {
|
||||||
|
const occupant = new Map(game.tiles.map(t => [`${t.q},${t.r}`, t.id]));
|
||||||
|
return game.tiles.map(t => {
|
||||||
|
const geo = trace(game.level, t, [], true);
|
||||||
|
const cells = geo.steps.map(s => { const who = occupant.get(`${s.q},${s.r}`); return { q: s.q, r: s.r, piece: who && who !== t.id ? who : null }; });
|
||||||
|
const onRoute = [...new Set(cells.filter(c => c.piece).map(c => c.piece))];
|
||||||
|
// Every piece that would stop this one, not just the first. Reporting only
|
||||||
|
// the leading blocker hid every graze behind it: in 28% of "blocked by a
|
||||||
|
// piece on the line" cases, clearing that piece sprang a collision the
|
||||||
|
// agent had never been shown. Test each candidate on its own so nothing
|
||||||
|
// masks anything else.
|
||||||
|
const grazes = [];
|
||||||
|
for (const other of game.tiles) {
|
||||||
|
if (other.id === t.id || onRoute.includes(other.id)) continue;
|
||||||
|
const hit = trace(game.level, t, [other]);
|
||||||
|
if (hit.status === 'blocked' && hit.blocker === other.id) grazes.push({ id: other.id, at: hit.steps.length });
|
||||||
|
}
|
||||||
|
grazes.sort((a, b) => a.at - b.at);
|
||||||
|
return { id: t.id, q: t.q, r: t.r, dir: t.dir, arrow: PORT_NAMES[t.dir], color: t.color,
|
||||||
|
route: { end: geo.status === 'exit' ? 'exit' : 'loop', cells: cells.slice(1), bends: countBends(geo), onRoute, grazes: grazes.map(g => g.id) } };
|
||||||
|
});
|
||||||
|
}
|
||||||
|
function routeLine(p) {
|
||||||
|
const shown = p.route.cells.slice(0, ROUTE_SHOW).map(c => `${c.q},${c.r}${c.piece ? `[${c.piece}]` : ''}`);
|
||||||
|
const tail = p.route.cells.length > ROUTE_SHOW ? ` … (共 ${p.route.cells.length} 格)` : '';
|
||||||
|
const end = p.route.end === 'exit' ? '出界' : '闭环';
|
||||||
|
const who = p.route.onRoute.length ? `路上有 ${p.route.onRoute.join(',')}` : '路上没有棋子';
|
||||||
|
const brush = p.route.grazes.length ? `${p.route.onRoute.length ? '，另外' : '，但'}紧贴 ${p.route.grazes.join(',')}（会擦到）` : '';
|
||||||
|
if (!p.route.cells.length) return `${p.id} @ ${p.q},${p.r} ${p.arrow}：就在边上，直接出界（0 格）· 路上没有棋子${brush}`;
|
||||||
|
return `${p.id} @ ${p.q},${p.r} ${p.arrow}：${shown.join(' → ')}${tail} → ${end}（${p.route.cells.length} 格 · ${p.route.bends} 弯）· ${who}${brush}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// `model` is the machine-readable id used for grouping; `agent` is whatever the
|
||||||
|
// agent says about itself — harness, mode, reasoning effort — shown as the
|
||||||
|
// banner over the board so a watcher knows who is playing.
|
||||||
|
export function createRun({ level, run, persona = 'default', model = 'unknown', agent = '', peek = false, budget = 80, startedAt = null }) {
|
||||||
|
if (!level?.cells || !level?.tiles) throw new Error('createRun needs a level with cells and tiles');
|
||||||
|
return { version: 1, run, persona, model, agent: String(agent || '').slice(0, 120), peek, budget, startedAt, updatedAt: startedAt, status: 'playing', level: clone(level), steps: [], rating: null, reason: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rebuild the live game by replaying the recorded actions.
|
||||||
|
export function replay(runObj) {
|
||||||
|
const game = new Game(runObj.level);
|
||||||
|
for (const step of runObj.steps) {
|
||||||
|
if (step.action.type === 'tap') game.tap(step.action.id);
|
||||||
|
else if (step.action.type === 'rotate') game.rotate(step.action.direction);
|
||||||
|
else if (step.action.type === 'undo') game.undo();
|
||||||
|
}
|
||||||
|
return game;
|
||||||
|
}
|
||||||
|
|
||||||
|
function pairs(tracks) {
|
||||||
|
const out = [];
|
||||||
|
for (let a = 0; a < 6; a++) if (a < tracks[a]) out.push(`${a}-${tracks[a]}`);
|
||||||
|
return out.join(' ');
|
||||||
|
}
|
||||||
|
|
||||||
|
export function observe(runObj, { format = 'text' } = {}) {
|
||||||
|
const game = replay(runObj);
|
||||||
|
const last = runObj.steps[runObj.steps.length - 1] || null;
|
||||||
|
const state = {
|
||||||
|
run: runObj.run, level: runObj.level.id, status: runObj.status, step: runObj.steps.length, budget: runObj.budget,
|
||||||
|
orientation: game.orientation, rotationsUsed: game.rotations, remaining: game.tiles.length, total: runObj.level.tiles.length,
|
||||||
|
pieces: pieceRoutes(game),
|
||||||
|
cells: [...game.level.cells].sort((a, b) => a.r - b.r || a.q - b.q).map(c => ({ q: c.q, r: c.r, pairs: pairs(c.tracks) })),
|
||||||
|
last: last ? { action: last.action, result: last.result } : null,
|
||||||
|
};
|
||||||
|
if (format === 'json') return state;
|
||||||
|
const lines = [];
|
||||||
|
lines.push(`LEVEL ${state.level} · ${state.cells.length} 格 · 剩余 ${state.remaining}/${state.total} 枚 · 朝向 ${state.orientation} · 已用 ${state.step}/${state.budget} 步 · 状态 ${state.status}`);
|
||||||
|
if (state.last) lines.push(`上一步：${describeAction(state.last.action)} → ${describeResult(state.last.result)}`);
|
||||||
|
lines.push('每枚棋子当前的路线（箭头出发 → 依次经过的格 → 出界/闭环；[tX] = 那格上压着棋子 tX）：');
|
||||||
|
for (const p of state.pieces) lines.push(' ' + routeLine(p));
|
||||||
|
if (format === 'full') {
|
||||||
|
lines.push('格子轨道（q,r: 端口配对 a-b；端口 0=↑ 1=↗ 2=↘ 3=↓ 4=↙ 5=↖）：');
|
||||||
|
let row = [];
|
||||||
|
for (const c of state.cells) {
|
||||||
|
row.push(`${c.q},${c.r}:${c.pairs.replace(/ /g, ',')}`);
|
||||||
|
if (row.length === 4) { lines.push(' ' + row.join(' ')); row = []; }
|
||||||
|
}
|
||||||
|
if (row.length) lines.push(' ' + row.join(' '));
|
||||||
|
}
|
||||||
|
return lines.join('\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
export function describeAction(action) {
|
||||||
|
if (action.type === 'tap') return `点击 ${action.id}`;
|
||||||
|
if (action.type === 'rotate') return action.direction === 1 ? '右转' : '左转';
|
||||||
|
if (action.type === 'peek') return `查看 ${action.id}`;
|
||||||
|
if (action.type === 'undo') return '撤销';
|
||||||
|
if (action.type === 'giveup') return '放弃';
|
||||||
|
return action.type;
|
||||||
|
}
|
||||||
|
export function describeResult(result) {
|
||||||
|
if (!result) return '';
|
||||||
|
if (result.status === 'exit') return `离场（走了 ${result.cells} 格、${result.bends} 个弯）${result.opened ? `，另有 ${result.opened} 枚棋子的路通了` : ''}`;
|
||||||
|
if (result.status === 'blocked') return `被 ${result.blocker} 挡住，退回原位`;
|
||||||
|
if (result.status === 'loop') return '路线绕回原处，没有出口';
|
||||||
|
if (result.status === 'rotated') return `底盘转到朝向 ${result.orientation}${result.opened ? `，${result.opened} 枚棋子的出口打开了` : '，这个朝向没有新的出口'}`;
|
||||||
|
if (result.status === 'undone') return '已撤销上一步';
|
||||||
|
if (result.status === 'peek') return result.route === 'exit' ? `它现在能离场：走 ${result.cells} 格、${result.bends} 个弯，经过 ${result.path}` : result.route === 'blocked' ? `它现在会在 ${result.path} 被 ${result.blocker} 挡住` : '它的路线绕回原处';
|
||||||
|
if (result.status === 'cleared') return '全部离场，通关';
|
||||||
|
if (result.status === 'rejected') return `无效：${result.reason}`;
|
||||||
|
return result.status;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function act(runObj, action) {
|
||||||
|
if (runObj.status !== 'playing') return reject(runObj, action, `本局已结束（${runObj.status}）`);
|
||||||
|
if (runObj.steps.length >= runObj.budget) { runObj.status = 'budget'; return reject(runObj, action, '步数预算用完'); }
|
||||||
|
if (!ACTIONS.includes(action?.type)) return reject(runObj, action, `未知动作 ${action?.type}，可用：${ACTIONS.join(' / ')}`);
|
||||||
|
if (action.type === 'peek' && !runObj.peek) return reject(runObj, action, '本局未开启 peek');
|
||||||
|
const game = replay(runObj);
|
||||||
|
let result;
|
||||||
|
if (action.type === 'giveup') {
|
||||||
|
runObj.status = 'gave-up'; runObj.reason = action.reason || '';
|
||||||
|
result = { status: 'gave-up', reason: runObj.reason };
|
||||||
|
} else if (action.type === 'undo') {
|
||||||
|
result = game.undo() ? { status: 'undone' } : { status: 'rejected', reason: '没有可撤销的步骤' };
|
||||||
|
if (result.status === 'rejected') return reject(runObj, action, result.reason);
|
||||||
|
} else if (action.type === 'rotate') {
|
||||||
|
if (![1, -1].includes(action.direction)) return reject(runObj, action, 'rotate 需要 direction 1（右）或 -1（左）');
|
||||||
|
const before = new Set(available(game.level, game.tiles).map(t => t.id));
|
||||||
|
const turned = game.rotate(action.direction);
|
||||||
|
if (turned.status !== 'rotated') return reject(runObj, action, turned.status);
|
||||||
|
const opened = available(game.level, game.tiles).filter(t => !before.has(t.id)).length;
|
||||||
|
result = { status: 'rotated', orientation: game.orientation, opened };
|
||||||
|
} else {
|
||||||
|
const tile = game.tiles.find(t => t.id === action.id);
|
||||||
|
if (!tile) return reject(runObj, action, `没有棋子 ${action.id}`);
|
||||||
|
if (action.type === 'peek') {
|
||||||
|
const route = trace(game.level, tile, game.tiles);
|
||||||
|
const path = route.steps.map(s => `${s.q},${s.r}`).join('→');
|
||||||
|
result = { status: 'peek', route: route.status, cells: route.steps.length, bends: countBends(route), blocker: route.blocker || null, path };
|
||||||
|
} else {
|
||||||
|
const before = new Set(available(game.level, game.tiles).map(t => t.id));
|
||||||
|
const route = game.tap(tile.id);
|
||||||
|
if (route.status === 'exit') {
|
||||||
|
const opened = available(game.level, game.tiles).filter(t => !before.has(t.id)).length;
|
||||||
|
result = { status: 'exit', cells: route.steps.length, bends: countBends(route), opened };
|
||||||
|
} else result = { status: route.status, blocker: route.blocker || null };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Wall-clock stamp per step. Steps in one batch land microseconds apart, so
|
||||||
|
// the gap between batches is the agent's real decision latency.
|
||||||
|
const step = { n: runObj.steps.length + 1, action, note: action.note || '', result, remaining: game.tiles.length, orientation: game.orientation, batch: null, at: Date.now() };
|
||||||
|
runObj.steps.push(step);
|
||||||
|
if (game.won) { runObj.status = 'cleared'; step.result = { ...result, cleared: true }; }
|
||||||
|
else if (runObj.steps.length >= runObj.budget && runObj.status === 'playing') runObj.status = 'budget';
|
||||||
|
return { step, feedback: describeResult(step.result) + (game.won ? '。全部离场，通关。' : '') };
|
||||||
|
}
|
||||||
|
function reject(runObj, action, reason) {
|
||||||
|
return { step: null, feedback: `无效：${reason}`, rejected: { action, reason } };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Several actions in one decision, the way a person chains the obvious taps.
|
||||||
|
// The chain stops at the first surprise (a tap that does not leave) and after
|
||||||
|
// any turn — the board has to be re-read after either.
|
||||||
|
export function actMany(runObj, actions, { note = '' } = {}) {
|
||||||
|
const last = runObj.steps[runObj.steps.length - 1];
|
||||||
|
const batch = (last?.batch || 0) + 1;
|
||||||
|
const outcomes = [];
|
||||||
|
let stopped = null;
|
||||||
|
for (let i = 0; i < actions.length; i++) {
|
||||||
|
const action = { ...actions[i], note: actions[i].note || note };
|
||||||
|
const out = act(runObj, action);
|
||||||
|
if (out.step) out.step.batch = batch;
|
||||||
|
outcomes.push(out);
|
||||||
|
if (!out.step || runObj.status !== 'playing') break;
|
||||||
|
if (action.type === 'tap' && out.step.result.status !== 'exit' && i < actions.length - 1) { stopped = '这一步没成功，后面的动作没有执行'; break; }
|
||||||
|
if (action.type === 'rotate' && i < actions.length - 1) { stopped = '旋转后盘面已变，后面的动作没有执行'; break; }
|
||||||
|
}
|
||||||
|
return { outcomes, stopped, batch };
|
||||||
|
}
|
||||||
|
|
||||||
|
// One decision = one look at the board: a lone step, or a whole chained batch.
|
||||||
|
// Its latency is the wall-clock gap since the previous decision was committed,
|
||||||
|
// which for a shell agent includes its own thinking and process startup.
|
||||||
|
export function decisionTimeline(runObj) {
|
||||||
|
const groups = [];
|
||||||
|
for (const step of runObj.steps) {
|
||||||
|
const previous = groups[groups.length - 1];
|
||||||
|
if (previous && step.batch != null && previous.batch === step.batch) { previous.steps.push(step); previous.at = step.at ?? previous.at; }
|
||||||
|
else groups.push({ batch: step.batch, steps: [step], at: step.at ?? null, from: step.n });
|
||||||
|
}
|
||||||
|
let cursor = runObj.startedAt ? Date.parse(runObj.startedAt) : null;
|
||||||
|
return groups.map(group => {
|
||||||
|
const ms = group.at != null && cursor != null ? group.at - cursor : null;
|
||||||
|
if (group.at != null) cursor = group.at;
|
||||||
|
return { from: group.from, actions: group.steps.length, at: group.at, ms };
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export const RATING_QUESTIONS = [
|
||||||
|
{ key: 'clarity', text: '路线看得清吗（1 完全看不清 – 5 一眼看清）' },
|
||||||
|
{ key: 'rotation', text: '旋转有意义吗（1 随便转都行/没用 – 5 每次转都是关键决定）' },
|
||||||
|
{ key: 'stuck', text: '有没有觉得无解的时刻（1 从未 – 5 长时间以为无解）' },
|
||||||
|
{ key: 'fun', text: '好玩吗（1 – 5）' },
|
||||||
|
{ key: 'difficulty', text: '难度（1 太简单 – 5 太难）' },
|
||||||
|
{ key: 'worst', text: '最困惑的一步是第几步，为什么（文字）' },
|
||||||
|
];
|
||||||
|
export function rate(runObj, answers) {
|
||||||
|
const rating = {};
|
||||||
|
for (const q of RATING_QUESTIONS) rating[q.key] = answers?.[q.key] ?? null;
|
||||||
|
runObj.rating = rating;
|
||||||
|
return rating;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Post-hoc metrics. The solver is allowed here — the run is over.
|
||||||
|
export function metrics(runObj) {
|
||||||
|
const game = new Game(runObj.level);
|
||||||
|
const solution = solveRotations(runObj.level);
|
||||||
|
const m = { run: runObj.run, level: runObj.level.id, persona: runObj.persona, model: runObj.model, agent: runObj.agent || '', peek: runObj.peek, status: runObj.status,
|
||||||
|
steps: runObj.steps.length, taps: 0, exits: 0, deadTaps: 0, grazeDeadTaps: 0, longestDeadStreak: 0, rotations: 0, effectiveRotations: 0, blindRotations: 0,
|
||||||
|
wrongWayRotations: 0, longestBlindStreak: 0, undos: 0, peeks: 0, stuckEntries: 0, stepsToFirstExit: null,
|
||||||
|
minimumRotations: solution.solvable ? solution.rotations : null, cleared: runObj.status === 'cleared', gaveUp: runObj.status === 'gave-up', rating: runObj.rating };
|
||||||
|
let dead = 0, blind = 0;
|
||||||
|
for (const step of runObj.steps) {
|
||||||
|
const a = step.action;
|
||||||
|
if (a.type === 'tap') {
|
||||||
|
m.taps++;
|
||||||
|
const tile = game.tiles.find(t => t.id === a.id);
|
||||||
|
const geoCells = tile ? new Set(trace(game.level, tile, [], true).steps.map(s => `${s.q},${s.r}`)) : new Set();
|
||||||
|
const route = game.tap(a.id);
|
||||||
|
if (route.status === 'exit') { m.exits++; dead = 0; blind = 0; if (m.stepsToFirstExit === null) m.stepsToFirstExit = step.n; }
|
||||||
|
else {
|
||||||
|
m.deadTaps++; dead++; m.longestDeadStreak = Math.max(m.longestDeadStreak, dead);
|
||||||
|
const blocker = route.blocker && game.tiles.find(t => t.id === route.blocker);
|
||||||
|
if (blocker && !geoCells.has(`${blocker.q},${blocker.r}`)) m.grazeDeadTaps++;
|
||||||
|
}
|
||||||
|
} else if (a.type === 'rotate') {
|
||||||
|
m.rotations++;
|
||||||
|
const other = available(rotateBoard(game.level, -a.direction), game.tiles).length;
|
||||||
|
const before = new Set(available(game.level, game.tiles).map(t => t.id));
|
||||||
|
game.rotate(a.direction);
|
||||||
|
const opened = available(game.level, game.tiles).filter(t => !before.has(t.id)).length;
|
||||||
|
if (opened) { m.effectiveRotations++; blind = 0; }
|
||||||
|
else { m.blindRotations++; blind++; m.longestBlindStreak = Math.max(m.longestBlindStreak, blind); if (other) m.wrongWayRotations++; }
|
||||||
|
} else if (a.type === 'undo') { m.undos++; game.undo(); }
|
||||||
|
else if (a.type === 'peek') m.peeks++;
|
||||||
|
if (!game.won && a.type !== 'peek' && !available(game.level, game.tiles).length) m.stuckEntries++;
|
||||||
|
}
|
||||||
|
m.rotationEfficiency = m.rotations ? m.effectiveRotations / m.rotations : null;
|
||||||
|
m.actionsPerPiece = runObj.level.tiles.length ? m.steps / runObj.level.tiles.length : null;
|
||||||
|
const timeline = decisionTimeline(runObj);
|
||||||
|
m.decisions = timeline.length;
|
||||||
|
m.actionsPerDecision = timeline.length ? m.steps / timeline.length : null;
|
||||||
|
const latencies = timeline.map(d => d.ms).filter(ms => typeof ms === 'number' && ms >= 0);
|
||||||
|
const sorted = [...latencies].sort((a, b) => a - b);
|
||||||
|
const at = p => sorted.length ? sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * p))] : null;
|
||||||
|
const lastAt = runObj.steps.length ? runObj.steps[runObj.steps.length - 1].at : null;
|
||||||
|
m.timing = {
|
||||||
|
totalMs: lastAt != null && runObj.startedAt ? lastAt - Date.parse(runObj.startedAt) : null,
|
||||||
|
firstDecisionMs: latencies.length ? latencies[0] : null,
|
||||||
|
medianDecisionMs: at(.5), p90DecisionMs: at(.9), maxDecisionMs: sorted.length ? sorted[sorted.length - 1] : null,
|
||||||
|
// Pure model latency, when the direct-API loop recorded it.
|
||||||
|
thinkMs: runObj.timing?.decisions?.length ? runObj.timing.decisions.reduce((s, d) => s + d.thinkMs, 0) : null,
|
||||||
|
};
|
||||||
|
return m;
|
||||||
|
}
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
import test from 'node:test';
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { hexagon } from '../src/core.js';
|
||||||
|
import { createRun, observe, act, actMany, rate, metrics, replay, rulesText, decisionTimeline } from '../src/bench.js';
|
||||||
|
|
||||||
|
// Straight rails on a 19-cell board: t1 sits on t0's route, so t0 is blocked
|
||||||
|
// until t1 leaves. Turning the board swaps which rails the pieces sit on.
|
||||||
|
function fixture() {
|
||||||
|
return {
|
||||||
|
version: 1, id: 'bench-fixture', name: 'fixture', cells: hexagon(2), rotationEnabled: true, moveLimit: 2,
|
||||||
|
tiles: [{ id: 't0', q: 0, r: 0, dir: 0, color: 0 }, { id: 't1', q: 0, r: -1, dir: 0, color: 1 }],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test('observation exposes board and pieces but never which pieces can leave', () => {
|
||||||
|
const run = createRun({ level: fixture(), run: 'obs', budget: 10 });
|
||||||
|
const json = observe(run, { format: 'json' });
|
||||||
|
assert.equal(json.remaining, 2);
|
||||||
|
assert.equal(json.cells.length, 19);
|
||||||
|
assert.deepEqual(Object.keys(json.pieces[0]).sort(), ['arrow', 'color', 'dir', 'id', 'q', 'r', 'route']);
|
||||||
|
// Routes are the drawn line a person sees: geometric, with pieces sitting on it marked.
|
||||||
|
const t0 = json.pieces.find(p => p.id === 't0'), t1 = json.pieces.find(p => p.id === 't1');
|
||||||
|
assert.equal(t0.route.end, 'exit');
|
||||||
|
assert.deepEqual(t0.route.onRoute, ['t1'], 't1 sits on t0\'s line');
|
||||||
|
assert.deepEqual(t1.route.onRoute, []);
|
||||||
|
assert.deepEqual(t0.route.grazes, [], 'a blocker on the line is not a graze');
|
||||||
|
assert.deepEqual(t1.route.grazes, [], 'nothing brushes a clear straight line');
|
||||||
|
assert.ok(!JSON.stringify(json).includes('available'));
|
||||||
|
assert.ok(!JSON.stringify(json).includes('solvable'));
|
||||||
|
const text = observe(run);
|
||||||
|
assert.match(text, /剩余 2\/2 枚/);
|
||||||
|
assert.match(text, /t0 @ 0,0 ↑：0,-1\[t1\] → 0,-2 → 出界（2 格 · 0 弯）· 路上有 t1/);
|
||||||
|
assert.match(text, /t1 @ 0,-1 ↑：0,-2 → 出界（1 格 · 0 弯）· 路上没有棋子/);
|
||||||
|
assert.doesNotMatch(text, /端口配对/, 'the rail table is opt-in');
|
||||||
|
assert.match(observe(run, { format: 'full' }), /端口配对/);
|
||||||
|
assert.match(rulesText(false), /怎么玩/);
|
||||||
|
assert.doesNotMatch(rulesText(false), /peek/);
|
||||||
|
assert.match(rulesText(true), /peek/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('actions go through the real game: blocked, exit with unlock, cleared', () => {
|
||||||
|
const run = createRun({ level: fixture(), run: 'play', budget: 10 });
|
||||||
|
let out = act(run, { type: 'tap', id: 't0', note: '先试最里面的' });
|
||||||
|
assert.equal(out.step.result.status, 'blocked');
|
||||||
|
assert.equal(out.step.result.blocker, 't1');
|
||||||
|
assert.match(out.feedback, /被 t1 挡住/);
|
||||||
|
out = act(run, { type: 'tap', id: 't1' });
|
||||||
|
assert.equal(out.step.result.status, 'exit');
|
||||||
|
assert.equal(out.step.result.opened, 1, 'removing the gate opens t0');
|
||||||
|
out = act(run, { type: 'tap', id: 't0' });
|
||||||
|
assert.equal(out.step.result.status, 'exit');
|
||||||
|
assert.equal(run.status, 'cleared');
|
||||||
|
assert.equal(replay(run).won, true);
|
||||||
|
assert.match(act(run, { type: 'tap', id: 't0' }).feedback, /已结束/);
|
||||||
|
const m = metrics(run);
|
||||||
|
assert.equal(m.taps, 3); assert.equal(m.exits, 2); assert.equal(m.deadTaps, 1);
|
||||||
|
assert.equal(m.longestDeadStreak, 1); assert.equal(m.stepsToFirstExit, 2); assert.equal(m.cleared, true);
|
||||||
|
assert.equal(run.steps[0].note, '先试最里面的');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('rotate reports how many pieces opened, undo restores, peek is gated', () => {
|
||||||
|
const run = createRun({ level: fixture(), run: 'turn', budget: 10 });
|
||||||
|
const turned = act(run, { type: 'rotate', direction: 1 });
|
||||||
|
assert.equal(turned.step.result.status, 'rotated');
|
||||||
|
assert.equal(turned.step.result.orientation, 1);
|
||||||
|
assert.equal(typeof turned.step.result.opened, 'number');
|
||||||
|
assert.equal(observe(run, { format: 'json' }).orientation, 1);
|
||||||
|
assert.equal(act(run, { type: 'undo' }).step.result.status, 'undone');
|
||||||
|
assert.equal(observe(run, { format: 'json' }).orientation, 0);
|
||||||
|
assert.match(act(run, { type: 'peek', id: 't0' }).feedback, /未开启 peek/);
|
||||||
|
assert.match(act(run, { type: 'rotate', direction: 2 }).feedback, /direction/);
|
||||||
|
const peeking = createRun({ level: fixture(), run: 'peek', peek: true, budget: 10 });
|
||||||
|
const seen = act(peeking, { type: 'peek', id: 't0' });
|
||||||
|
assert.equal(seen.step.result.status, 'peek');
|
||||||
|
assert.equal(seen.step.result.route, 'blocked');
|
||||||
|
assert.equal(seen.step.result.blocker, 't1');
|
||||||
|
assert.equal(metrics(peeking).peeks, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a chained batch stops at the first failed tap and after a turn', () => {
|
||||||
|
const run = createRun({ level: fixture(), run: 'chain', budget: 20 });
|
||||||
|
// t0 is blocked by t1: the chain must stop there and not tap t1.
|
||||||
|
let batch = actMany(run, [{ type: 'tap', id: 't0' }, { type: 'tap', id: 't1' }], { note: '连点试试' });
|
||||||
|
assert.equal(batch.outcomes.length, 1);
|
||||||
|
assert.equal(batch.outcomes[0].step.result.status, 'blocked');
|
||||||
|
assert.match(batch.stopped, /没成功/);
|
||||||
|
assert.equal(run.steps.length, 1);
|
||||||
|
assert.equal(run.steps[0].batch, 1);
|
||||||
|
assert.equal(run.steps[0].note, '连点试试');
|
||||||
|
// A turn ends the chain even when it succeeds.
|
||||||
|
batch = actMany(run, [{ type: 'rotate', direction: 1 }, { type: 'tap', id: 't1' }]);
|
||||||
|
assert.equal(batch.outcomes.length, 1);
|
||||||
|
assert.match(batch.stopped, /旋转后/);
|
||||||
|
actMany(run, [{ type: 'undo' }]);
|
||||||
|
// The good order clears in one decision.
|
||||||
|
batch = actMany(run, [{ type: 'tap', id: 't1' }, { type: 'tap', id: 't0' }]);
|
||||||
|
assert.equal(batch.outcomes.length, 2);
|
||||||
|
assert.equal(batch.stopped, null);
|
||||||
|
assert.equal(run.status, 'cleared');
|
||||||
|
const m = metrics(run);
|
||||||
|
assert.equal(m.steps, 5);
|
||||||
|
assert.equal(m.decisions, 4, 'four looks at the board: blocked chain, turn, undo, clearing chain');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the agent self-reports its identity, and it survives into metrics', () => {
|
||||||
|
const run = createRun({ level: fixture(), run: 'who', model: 'claude-opus-5', agent: 'Claude Code · ultracode', budget: 5 });
|
||||||
|
assert.equal(run.model, 'claude-opus-5');
|
||||||
|
assert.equal(run.agent, 'Claude Code · ultracode');
|
||||||
|
assert.equal(metrics(run).agent, 'Claude Code · ultracode');
|
||||||
|
// Missing is fine — the banner falls back to the model alone.
|
||||||
|
assert.equal(createRun({ level: fixture(), run: 'anon' }).agent, '');
|
||||||
|
// A runaway self-description must not become an unbounded banner.
|
||||||
|
assert.equal(createRun({ level: fixture(), run: 'long', agent: 'x'.repeat(500) }).agent.length, 120);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('timing groups a batch into one decision and measures the gap before it', () => {
|
||||||
|
const run = createRun({ level: fixture(), run: 'timing', budget: 20, startedAt: new Date(1000).toISOString() });
|
||||||
|
actMany(run, [{ type: 'tap', id: 't1' }, { type: 'tap', id: 't0' }]);
|
||||||
|
// Stamps come from the clock; rewrite them so the assertions are exact.
|
||||||
|
run.steps[0].at = 3000; run.steps[1].at = 3010;
|
||||||
|
const timeline = decisionTimeline(run);
|
||||||
|
assert.equal(timeline.length, 1, 'a chained batch is one decision');
|
||||||
|
assert.equal(timeline[0].actions, 2);
|
||||||
|
assert.equal(timeline[0].from, 1);
|
||||||
|
assert.equal(timeline[0].ms, 2010, 'latency runs from startedAt to the batch commit');
|
||||||
|
const m = metrics(run);
|
||||||
|
assert.equal(m.decisions, 1);
|
||||||
|
assert.equal(m.actionsPerDecision, 2);
|
||||||
|
assert.equal(m.timing.totalMs, 2010);
|
||||||
|
assert.equal(m.timing.medianDecisionMs, 2010);
|
||||||
|
assert.equal(m.timing.thinkMs, null, 'only the direct-API loop records model latency');
|
||||||
|
// A run recorded before timestamps existed must not crash or invent numbers.
|
||||||
|
const legacy = createRun({ level: fixture(), run: 'legacy', budget: 20 });
|
||||||
|
act(legacy, { type: 'tap', id: 't1' });
|
||||||
|
delete legacy.steps[0].at;
|
||||||
|
assert.equal(metrics(legacy).timing.totalMs, null);
|
||||||
|
assert.equal(decisionTimeline(legacy)[0].ms, null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('budget and give-up end the run; rating is recorded', () => {
|
||||||
|
const run = createRun({ level: fixture(), run: 'budget', budget: 1 });
|
||||||
|
act(run, { type: 'tap', id: 't0' });
|
||||||
|
assert.equal(run.status, 'budget');
|
||||||
|
assert.match(act(run, { type: 'tap', id: 't1' }).feedback, /已结束/);
|
||||||
|
const quit = createRun({ level: fixture(), run: 'quit', budget: 5 });
|
||||||
|
act(quit, { type: 'giveup', reason: '看起来无解' });
|
||||||
|
assert.equal(quit.status, 'gave-up');
|
||||||
|
assert.equal(quit.reason, '看起来无解');
|
||||||
|
assert.equal(metrics(quit).gaveUp, true);
|
||||||
|
const rating = rate(quit, { clarity: 2, stuck: 5, worst: '第 1 步' });
|
||||||
|
assert.equal(rating.clarity, 2); assert.equal(rating.fun, null);
|
||||||
|
assert.equal(metrics(quit).rating.stuck, 5);
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user