fix(bench): rebuild each round's level set from its runs, not from a snapshot
The round comparison was about to be wrong. levels/campaign-50-r2.json was copied from levels/campaign-50.json to preserve "the set round 2 played", but select-50 writes that path in place and several selection experiments ran between the round and the copy. The file kept 28 of the 50 boards round 2 had actually played and 46 of round 3's — so the comparison was scoring one round against another round's levels, and it read as round 3 regressing. Runs store the whole board and its campaign index, so a round is its own record and cannot drift. round-set.mjs rebuilds it, the report uses that, and the snapshot file is gone rather than left lying around looking authoritative. With the rounds correctly matched, round 3 is not a regression and not an improvement: back band 2.17 against round 2's 2.39, on a rating that retests at 0.55, which is not enough to separate seventeen levels. The report says that rather than picking whichever reading flatters the change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -14,7 +14,7 @@ import { layout } from '../src/geometry.js';
|
||||
import { metrics } from '../src/bench.js';
|
||||
|
||||
const args = Object.fromEntries(process.argv.slice(2).map(a => { const [k, ...v] = a.replace(/^--/, '').split('='); return [k, v.length ? v.join('=') : true]; }));
|
||||
const AGENT = args.agent ?? 'R1 · grok xhigh';
|
||||
const AGENT = args.agent ?? 'R3 · grok xhigh';
|
||||
const OUT = resolve(args.out ?? 'docs/review-50.html');
|
||||
|
||||
const raw = JSON.parse(readFileSync(resolve('levels/campaign-50.json'), 'utf8'));
|
||||
|
||||
Reference in New Issue
Block a user