CharacterBench reproduction audit bundle

Everything behind Section 6 of the CharacterBench reproduction report: the scripts as run, their results and logs, and the frozen inputs from which the scoring can be recomputed without access to the platform. Nothing here contains test-set text, dialogue or a model's reply: the test set stays with its authors (github.com/thu-coai/CharacterBench), and replies are referred to only by sample.

Check without running anything

shasum -a 256 -c SHA256SUMS      # in a copy of this directory

Every file's SHA-256 is listed below and in SHA256SUMS.

Recompute the scores from the frozen judge outputs

Each line of frozen/judge_outputs_<run>.jsonl holds a sample's ten judge outputs as returned. Read each with check() from the authors' all_eval.py, average the ten, and per language and test file take mean / max(human scores of the kept samples) × 5; AVG is the mean of the 13 columns. The result must equal the run's result.perLang in frozen/runs.json at two decimals. code/tools/experiments/characterbench_recompute.py does exactly this with only this bundle and the authors' repository (log: logs/11-recompute.log):

python3 -I code/tools/experiments/characterbench_recompute.py repo frozen

Re-run the whole audit

git clone https://github.com/thu-coai/CharacterBench repo && git -C repo checkout 66c113c
curl -L -o eprint.tar.gz https://arxiv.org/e-print/2412.11912v1 && mkdir eprint && tar xzf eprint.tar.gz -C eprint
python3 -I code/tools/experiments/characterbench_spec_check.py eprint repo frozen/val_zh.jsonl frozen/val_en.jsonl
python3 -I code/tools/experiments/characterbench_judge_audit.py out.json frozen/val_zh.jsonl frozen/val_en.jsonl     --repeat frozen/judge_rep0_zh.jsonl frozen/judge_rep0_en.jsonl --seeds frozen/judge_rep{1,2,3}_{zh,en}.jsonl

The prompt and pipeline checks (characterbench_prompt_check.py, characterbench_audit_platform.ts, characterbench_audit.py) also need the platform's samples and the runs' replies; their outputs and logs are in results/ and logs/, and the logs give each command as run. The judge requests themselves can be repeated against any server running the checkpoint with characterbench_judge_validation.py (sampled outputs repeat only on the same deployment).

Runs audited

• 31a478305c1641a6: characterbench-2.0 · 2026-10-27, 6,460 judged samples (paper anchor, not on the leaderboard)
• 47920340ab8e49fe: characterbench-2.0 · 2026-10-27, 6,488 judged samples
• 7ec615d6637844cd: characterbench-2.0 · 2026-10-27, 6,474 judged samples
• a9c1f7d7a75f4fb3: characterbench-2.0 · 2026-10-27, 6,494 judged samples
• e279fdfe13874179: characterbench-2.0 · 2026-10-27, 6,476 judged samples
• e78e8134973f4bd5: characterbench-2.0 · 2026-10-27, 6,494 judged samples
• eebe847ae59b4d41: characterbench-2.0 · 2026-10-27, 6,462 judged samples (paper anchor, not on the leaderboard)
• fb8c23842faa41d1: characterbench-2.0 · 2026-10-27, 6,494 judged samples

Files